MantisBT

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0001420VCMIOtherpublic2013-08-26 02:492017-07-22 14:16
Reporteracme_pjz 
Assigned ToIvan 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusassignedResolutionopen 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0001420: Read data file and map file with other text encodings
DescriptionIs it possible to read data file and map file with other configurable text encodings? In current version the font files can be configured already. But it still doesn't work on Heroes3 Simplified Chinese edition, because the text encoding is CP936. If the text encoding is configurable, then VCMI will support more languages of Heroes3.
TagsNo tags attached.
Attached Files7z file icon H3Bitmap.7z [^] (219,570 bytes) 2013-09-03 06:11
7z file icon HZK.7z [^] (427,903 bytes) 2013-09-03 06:11
png file icon vcmi-chinese.png [^] (693,206 bytes) 2013-09-08 15:06


jpg file icon 1.jpg [^] (54,280 bytes) 2013-09-11 05:30


jpg file icon 1-1.jpg [^] (74,040 bytes) 2013-10-03 04:57


jpg file icon 1-2.jpg [^] (65,890 bytes) 2013-10-03 04:58

- Relationships

-  Notes
(0003895)
Ivan (developer)
2013-08-26 09:33

Hi. This is first time I hear about H3 version that uses multi-byte encoding. Can you upload fonts and text files from your version of H3?

Normally they can be found in file Data/H3bitmap.lod which can be opened using MMArchive, can be downloaded here: http://wogarchive.ru/download.php?id=119 [^]

Unpack all files with *.txt and *.fnt extensions and upload them here.
(0003921)
acme_pjz (reporter)
2013-09-03 06:15

Text files uploaded. They are all in CP936 (a.k.a. GBK) encoding. And I think font files in H3bitmap.lod don't contain Chinese characters, because I found some file named HZK** in the root directory of H3, and I think they are Simplified Chinese font file.
(0003922)
Ivan (developer)
2013-09-03 10:18

Thanks. Yes - these 3 files are indeed Chinese fonts and encoding seems to match to GBK character tables I found.
I'll add support for these fonts soon - looks simple enough.

For reference:

These files consist from sequence of bitmaps, 10x10, 12x12 or 24x24 pixels in size, 1 bit per pixel, width is aligned to full byte (10x10 image actually uses 2x10 bytes per character). No header or anything like that.

Font files contain ONLY Chinese characters, ASCII symbols should be taken from English fonts.
Mapping of 2-byte GBK characters to image index:
index = (first - 0x80) * 0xA0 + (second - 0x40)
May be a bit different - file size indicates 8000+ characters while GBK can fit 20000+ symbols.
(0003935)
acme_pjz (reporter)
2013-09-06 16:00

I’ll test it soon. BTW, it need to convert to UTF8 or UTF16 if using SDL_ttf IMO
(0003936)
Ivan (developer)
2013-09-06 16:28

Proper unicode support is more difficult to implement. Right now I'm leaning towards making vcmi work with GBK encoding so no conversion is needed.

I also can't use SDL_ttf here because those fonts you posted are bitmaps while SDL_ttf works only with true type vector fonts. Not really a problem since original H3 fonts are also bitmaps and are already supported (including different languages like Russian)
(0003961)
Ivan (developer)
2013-09-08 15:08

Looks to be working, will commit my changes soon.

Does everything on this image looks OK?
http://bugs.vcmi.eu/file_download.php?file_id=1465&type=bug [^]
(0003967)
Tow (developer)
2013-09-08 16:16

The text line in the message (the white line under the title text) seems to be clipped, missing several bottom pixel rows.
Same issue with labels next to the Quest Log and Dismiss buttons.
Other text seems fine.
(0003982)
acme_pjz (reporter)
2013-09-09 05:18

Same as Tow said, and maybe too few spaces between characters.

And I want to point out that your mapping formula is incorrect. It looks like H3 Simplified Chinese edition only supports GB2312 encoding, i.e. GBK/1 and GBK/2 in GBK standard. After experiments I found that the correct mapping formula is

index = (first - 0xA1) * 94 + (second - 0xA1)

Be careful, it can map non-GB2312 character into the valid range :| For non-GB2312 characters and other edition of H3 (possibly Traditional Chinese edition which may using other text encodings?) an iconv calling and SDL_ttf must be used (VCMI can be configured to use TTF vector font by changing config/fonts.json, right?)

The second note is while HZK10 looks no problem, characters in HZK12 and HZK24H seems X-Y flipped!
(0003986)
Ivan (developer)
2013-09-09 11:15

Can you make screenshot of Chinese version and post it here for comparison? I compared several characters to what I see in text editor and they were looking fine.

- missing line is already fixed

- GBK vs GB2312: actually I ran into this issue already. After a quick search I found a discussion on Chinese support in Era. Including link to GBK fonts. Switching from GBK to GB2312 is possible but since GBK already includes GB2312 I think supporting GBK would be more than enough.

- unicode: I think I'll try to add some basic unicode support (utf8 to be precise). This will allow using ttf font with any encoding BUT encoding must be selected manually - I haven't found any reliable way to detect encoding. Especially problematic for very similar Win1250...Win1252 which are used in majority of H3 versions. This will solve "what encoding to use" problem but from my experience ttf fonts quality is a bit lower compared to native bitmaps.
(0003995)
acme_pjz (reporter)
2013-09-11 05:31

Hi, I just uploaded a screenshot, taken from Internet, but it represents Chinese version well.
(0004059)
acme_pjz (reporter)
2013-10-02 05:38

I tried VCMI 0.94, but it still doesn't work, either GBK or GB2312. Should I need to put HZK10 and other files to specified directory?
(0004060)
Ivan (developer)
2013-10-02 16:27

Right now you'll need to install this "mod" - it provides some files necessary for Chinese support (fonts & config file)
http://download.vcmi.eu/mods/repository/chinese%20fonts.zip [^]
Download this file and unpack int into Mods/ directory.

This mod can be also installed via launcher.
(0004061)
acme_pjz (reporter)
2013-10-03 05:15

Thanks, it works. But there are a few glitches: mojibake due to incorrect character boundary detection (characters in green circles) and string which is too wide (in cyan circles).

http://bugs.vcmi.eu/file_download.php?file_id=1532&type=bug [^]

http://bugs.vcmi.eu/file_download.php?file_id=1533&type=bug [^]

Maybe you can fix them in next versions?
(0004062)
Ivan (developer)
2013-10-03 11:59
edited on: 2013-10-03 14:29

Sure, will fix. Note that time between our releases is around 3 months so you'll have to wait a bit.

1) Too wide string: will see what I can do - it seems that ASCII characters need special treatment.

2) What's wrong in green circles? Missing characters? Incorrect line breaks? Something else? I don't know Chinese so I have not idea how it should look like.

(0004063)
acme_pjz (reporter)
2013-10-04 07:34
edited on: 2013-10-04 14:20

1) How about fall back to default ASCII font file for ASCII characters (and characters are variable width)? This is the behavior of original H3.

2) The character render is incorrect in green circles, which called "乱码" in Chinese, and (hopefully) Mojibake in English. (http://en.wikipedia.org/wiki/Mojibake [^]) Since GBK is 2-byte encoding and I think the line breaking code is not taken this into account, so it break the line inside character.

By the way, Are there any nightly builds for Windows (or Linux)?

[EDIT] Sorry, but I have already found them in the forum.

(0004064)
Ivan (developer)
2013-10-04 13:19

1) I thought about this at first but Chinese fonts have different height. OK, I'll try to find matching fonts. Shouldn't be hard to implement.

2) Thanks. Yeah - that's probably some bug in line splitting code. Will fix.
(0004065)
Ivan (developer)
2013-10-04 15:14

>> By the way, Are there any nightly builds for Windows (or Linux)?
You can find nighty builds for Ubuntu here: https://launchpad.net/~vcmi/+archive/ppa [^]
For technical reasons (multiple versions of Ubuntu) launcher is disabled but othervice they should work.

"Nighty builds" for Windows are usually done some time before release so there won't be any Windows builds for around 2 months.
(0004086)
Jolly Wing (reporter)
2013-10-19 13:31

Hi, Ivan.

I have made some little changes on the source code 0.9.3 to make vcmi to support the Game Data archives from Simplified Chinese version. I think such a modification can also support Game Data archives from CJK (Chinese, Japanese, Korean and such multi-bytes language) versions through some improvements.

When the engine reads map data and general text data, I convert the GBK encoded string into UTF8 encoded string, then I render the strings with SDL_TTF_RenderUTF8. It works!

The changes I made are list as follow:

1. client/CMessage.cpp
in function CMessage::breakText(), line 153

            // added by [email protected], 2013-08-28
            // If the text[z] is less than 0, it is the first byte of a UTF8 Chinese word.
#ifdef ZH_CN
            else if (text[z] < 0){
                z++;
                z++;
             lineLength += graphics->fonts[font]->getSymbolWidth(text[z]);
            }
#endif

2. client/gui/Fonts.cpp
in function CTrueTypeFont::getStringWidth(), line 255

    // added by [email protected], 2013-08-28
    // If we are handling simplified chinese, it is a UTF8 string
#ifdef ZH_CN
    TTF_SizeUTF8(font.get(), data.c_str(), &width, NULL);
#else
    TTF_SizeText(font.get(), data.c_str(), &width, NULL);
#endif

3. client/gui/Fonts.cpp
in function CTrueTypeFont::renderText(), line 279

     if (blended)
            // added by [email protected], 2013-09-28 Sat
            // If we are handling simplified chinese game data, it is a UTF8 string
#ifdef ZH_CN
         rendered = TTF_RenderUTF8_Blended(font.get(), data.c_str(), color);
#else
         rendered = TTF_RenderText_Blended(font.get(), data.c_str(), color);
#endif
     else
            // added by [email protected], 2013-09-28 Sat
            // If we are handling simplified chinese game data, it is a UTF8 string
#ifdef ZH_CN
         rendered = TTF_RenderUTF8_Solid(font.get(), data.c_str(), color);
#else
         rendered = TTF_RenderText_Solid(font.get(), data.c_str(), color);
#endif

4. add lib/ConvertEncoding.cpp and lib/ConvertEncoding.h

The content of ConvertEncoding.h is:

char * convert_enc(char *src_enc, char *dest_enc, const char * src_string);

The content of ConvertEncoding.cpp is:

/*
 * ConvertEncoding.cpp, for vcmi using CJK(China/Japan/Korea) data.
 *
 * Authors: Wu Jiqing ([email protected])
 *
 * License: GNU General Public License v2.0 or later
 *
 */
// added by jiqingwu([email protected])
// 2013-09-27 Fri
#include <stdio.h>
#include <iconv.h>
#include <string.h>

// added by [email protected], 2013-09-27 Fri
char * convert_enc(char *src_enc, char *dest_enc, const char * src_string)
{
#define UTF8_STR_LEN 5000

    static char out_string[UTF8_STR_LEN], *sin, *sout;
    int in_len, out_len, ret;
    iconv_t c_pt;

    if ((c_pt = iconv_open(dest_enc, src_enc)) == (iconv_t)-1)
    {
        printf("iconv open failed!\n");
        return NULL;
    }
    // iconv(c_pt, NULL, NULL, NULL, NULL);
    in_len = strlen(src_string) + 1;
    out_len = UTF8_STR_LEN;
    sin = (char *)src_string;
    sout = out_string;
    ret = iconv(c_pt, &sin, (size_t *)&in_len, &sout, (size_t *)&out_len);
    if (ret == -1)
    {
        return NULL;
    }
    iconv_close(c_pt);
    return out_string;
}

to link ConvertEncoding.o into library, add two lines into lib/CMakeLists.txt:

set(lib_SRCS
        ...
        ConvertEncoding.cpp
)

set(lib_HEADERS
        ...
        ConvertEncoding.h
)

5. lib/CGeneralTextHandler.cpp,
To include "ConvertEncoding.h"

// added by jiqingwu([email protected])
// 2013-09-27 Fri
#include "ConvertEncoding.h"

in function CLegacyConfigParser::readString(), line 112

    // added by [email protected], 2013-09-27 Fri
    // convert gbk string to utf-8 string.
    // (For simplified Chinese game data, the string is GBK encoded)
#ifdef ZH_CN
    char * utf8_str = convert_enc("GBK", "UTF8", ret.c_str());
    return std::string((const char*)utf8_str);
#else
    return ret;
#endif

6. lib/filesystem/CBinaryReader.cpp,
to include "ConvertEncoding.h":

// added by <[email protected]>, 2013-09-28 Sat
#include "../ConvertEncoding.h"

in function CBinaryReader::readString(), line 95

    // added by [email protected], 2013-08-22
    // If we are handling chinese data, convert gbk string to utf-8 string.
#ifdef ZH_CN
    char * utf8_str = convert_enc("GBK", "UTF8", ret.c_str());
    return std::string((const char*)utf8_str);
#else
    return ret;
#endif

7. add such a line into ./CMakeLists.txt to enable supporting Simplifed Chinese Game Data

add_definitions(-DZH_CN)

8. cmake, make, make install

To Play

Use the Data from the chinese version of Death of Shadow. Link the 'Data', 'Maps', 'Mp3' directories under /usr/local/share/vcmi like this (You need have root privilege):
cd /usr/local/share/vcmi
ln -s /Data/Dir/of/ChineseGame Data
ln -s /Maps/Dir/of/ChineseGame Maps
ln -s /Mp3/Dir/of/ChineseGame Mp3

To show chinese characters in this game, you need put a true type font which supports Chinese into /usr/local/share/vcmi/Data.

cp /chinese/font/path /usr/local/share/vcmi/Data

In addition, you need edit the /usr/local/share/vcmi/config/fonts.json, modify the truetype font section like this:

"trueType":
{
    "BIGFONT" : { "file" : "ChineseFont.ttf", "size" : 22, "blend" : true},
    "CALLI10R" : { "file" : "ChineseFont.ttf", "size" : 10, "blend" : true},
    "CREDITS" : { "file" : "ChineseFont.ttf", "size" : 28, "blend" : true},
    "HISCORE" : { "file" : "ChineseFont.ttf", "size" : 13, "blend" : true},
    "MEDFONT" : { "file" : "ChineseFont.ttf", "size" : 16, "blend" : true},
    "SMALFONT" : { "file" : "ChineseFont.ttf", "size" : 13, "blend" : true},
    "TIMES08R" : { "file" : "ChineseFont.ttf", "size" : 11, "blend" : true},
    "TINY" : { "file" : "ChineseFont.ttf", "size" : 11, "blend" : true},
    "VERD10B" : { "file" : "ChineseFont.ttf", "size" : 13, "blend" : true}
}
Where ChineseFont.ttf is your true type font.

Then, we can Play Game
$ vcmiclient
(0004091)
Ivan (developer)
2013-10-20 10:09

Thanks. I'll take a look on this. I don't like idea of using ifdef's to enable some functionality but you've tracked every place that needs changes - that will help.
(0007163)
SXX (administrator)
2017-07-22 14:16

Interesting how this works now. Last year VCMI got support for UTF in file paths:
https://github.com/vcmi/vcmi/pull/156 [^]

- Issue History
Date Modified Username Field Change
2013-08-26 02:49 acme_pjz New Issue
2013-08-26 09:33 Ivan Note Added: 0003895
2013-08-26 10:06 Ivan Assigned To => Ivan
2013-08-26 10:06 Ivan Status new => feedback
2013-09-03 06:11 acme_pjz File Added: H3Bitmap.7z
2013-09-03 06:11 acme_pjz File Added: HZK.7z
2013-09-03 06:15 acme_pjz Note Added: 0003921
2013-09-03 06:15 acme_pjz Status feedback => assigned
2013-09-03 10:18 Ivan Note Added: 0003922
2013-09-06 16:00 acme_pjz Note Added: 0003935
2013-09-06 16:28 Ivan Note Added: 0003936
2013-09-08 15:06 Ivan File Added: vcmi-chinese.png
2013-09-08 15:08 Ivan Note Added: 0003961
2013-09-08 16:16 Tow Note Added: 0003967
2013-09-09 05:18 acme_pjz Note Added: 0003982
2013-09-09 11:15 Ivan Note Added: 0003986
2013-09-11 05:30 acme_pjz File Added: 1.jpg
2013-09-11 05:31 acme_pjz Note Added: 0003995
2013-10-02 05:38 acme_pjz Note Added: 0004059
2013-10-02 16:27 Ivan Note Added: 0004060
2013-10-03 04:57 acme_pjz File Added: 1-1.jpg
2013-10-03 04:58 acme_pjz File Added: 1-2.jpg
2013-10-03 05:15 acme_pjz Note Added: 0004061
2013-10-03 11:59 Ivan Note Added: 0004062
2013-10-03 14:29 Ivan Note Edited: 0004062 View Revisions
2013-10-04 07:34 acme_pjz Note Added: 0004063
2013-10-04 13:19 Ivan Note Added: 0004064
2013-10-04 13:48 acme_pjz Note Edited: 0004063 View Revisions
2013-10-04 14:20 acme_pjz Note Edited: 0004063 View Revisions
2013-10-04 15:14 Ivan Note Added: 0004065
2013-10-19 13:31 Jolly Wing Note Added: 0004086
2013-10-20 10:09 Ivan Note Added: 0004091
2017-07-22 14:16 SXX Note Added: 0007163

Site | Forums | Wiki | Slack | GitHub


Copyright © 2000 - 2019 MantisBT Team
Hosting provided by DigitalOcean