missing vietnamese codec TCVN 5712:1993 in Python

[progfou]

BPO	21081
Nosy	@malemburg, @jwilk, @ezio-melotti, @serhiy-storchaka, @ztane, @matorban, @son
Files	TCVN5712-1.TXT: codec table for TCVN 5712:1993 VN1 TCVN5712-2.TXT: codec table for TCVN 5712:1993 VN2 TCVN5712-3.TXT: codec table for TCVN 5712:1993 VN3 issue21081.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-03-28.01:38:41.596>
labels = ['type-feature', '3.7', 'expert-unicode']
title = 'missing vietnamese codec TCVN 5712:1993 in Python'
updated_at = <Date 2020-04-28.12:48:12.224>
user = 'https://bugs.python.org/progfou'

bugs.python.org fields:

activity = <Date 2020-04-28.12:48:12.224>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2014-03-28.01:38:41.596>
creator = 'progfou'
dependencies = []
files = ['37054', '37055', '37056', '45080']
hgrepos = []
issue_num = 21081
keywords = ['patch']
message_count = 19.0
messages = ['215012', '215032', '215033', '215041', '215043', '215047', '229736', '230192', '230193', '230232', '230235', '278581', '279137', '279149', '279151', '320615', '367518', '367519', '367520']
nosy_count = 10.0
nosy_names = ['lemburg', 'jwilk', 'ezio.melotti', 'progfou', 'hieu.nguyen', 'serhiy.storchaka', 'ztane', 'matorban', 'sondecorpaint', 'son_lotus']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21081'
versions = ['Python 3.7']

[progfou]

In Python version 2.x and at least 3.2 there no Vietnamese encoding support for TCVN 5712:1993.

This encoding is currently largely used in Vietnam and I think it would be usefull to add it to the python core encodings.

I already wrote some codec code, based on the codecs already available, that I successfully used in real life situation.

I would like to give it as a contribution to Python.

[malemburg]

Some comments:

• Please provide some background information how widely the encoding is used. I get less than 1000 hits in Google when looking for "TCVN 5712:1993". Now, the encoding was a standard in Vietnam, but it has been updated in 1999 to TCVN 5712:1999. There's also an encoding called VSCII.

• In the file you write "kind of TCVN 5712:1993 VN3 with CP1252 additions". This won't work, since we can only accept codecs which are based on set standards. It would be better to provide a link to an official Unicode character set mapping table and then use the gencodec.py script on this table.

• For Vietnamese, Python already provides cp1258 - how much is this encoding used in comparison to e.g. TCVN 5712:1993 ?

Resources:

• Vietnamese encodings: http://www.panl10n.net/english/outputs/Survey/Vietnamese.pdf

• East Asian encodings: http://www.unicode.org/iuc/iuc15/tb1/slides.pdf

[malemburg]

Retargeting to 3.5, since all other releases don't allow addition of new features.

[progfou]

• Please provide some background information how widely the encoding is used. I get less than 1000 hits in Google when looking for "TCVN 5712:1993".

Here is the background for the need for this encoding.

The recent laws[0] in Vietnam have set TCVN 6909:2001 (Unicode based) as the standard encoding everybody should use. Still, there was more than 30 old Vietnamese encodings that were used for tenths of years before that, with some of them being still used (it takes times for people to accept the change and for technicians to do what's required to change technology). Among them, TCVN 5712:1993 was (is) mostly used in the North of Vietnam and VNI (a private company encoding) in the South of Vietnam.

Worse than that, these old encodings use the C0 bank to store some Vietnamese letters (especially the 'ư', one of the most used in this language), which has the very unpleasant consequence to let some software (like OpenOffice/LibreOffice) being unable to render the texts correctly, even when using the correct fonts. Since this was a showstopper for Free Software adoption in Vietnam, I decided at that time to create a tool[1][2] to help in converting from these old encodings to Unicode. The project was then endorsed by the Ministry of Sciences and Technology of Vietnam, which asked me to make further developments[3].

Even if these old encodings are, hopefully, not the widest used in Vietnam now, there are still tons/plenty of old documents (sorry, I can't be more precise on the volume of administrative or private documents) that need to be read/modified or, best, converted to Unicode; and here is where the encodings are needed. Now every time some Vietnamese people (and Laotian people, I'll come back on this in another bug report) want to use OpenOffice/LibreOffice and still be able to open their old documents, they have to install this Python extension for this.

I foresee there will be not only plain documents to convert but also databases and other kind of data storage. And here is where Python has a great occasion to become the tool of choice.

[0] http://thuvienphapluat.vn/archive/Quyet-dinh-72-2002-QD-TTg-thong-nhat-dung-bo-ma-ky-tu-chu-Viet-TCVN-6909-2001-trao-doi-thong-tin-dien-tu-giua-to-chuc-dang-nha-nuoc-vb49528.aspx
[1] http://wiki.hanoilug.org/projects:ovniconv
[2] http://extensions.services.openoffice.org/project/ovniconv
[3] http://extensions.services.openoffice.org/en/project/b2uconverter

Now, the encoding was a standard in Vietnam, but it has been updated in 1999 to TCVN 5712:1999.

I have to admit I missed this one. It may explain the differences I saw when I reversed engineered the TCVN encoding through the study the documents Vietnamese users provided to me. I will check this one and come back with more details.

There's also an encoding called VSCII.

VSCII is the same as TCVN 5712:1993.

This page contains interesting information about these encodings: http://www.informatik.uni-leipzig.de/~duc/software/misc/tcvn.txt

• In the file you write "kind of TCVN 5712:1993 VN3 with CP1252 additions". This won't work, since we can only accept codecs which are based on set standards.

I can understand that and I'll do my best to check if it's really based on one of the TCVN standards, be it 5712:1993 or 5712:1999. Still, after years of usage, I know perfectly that it's exactly the encoding we need (for the North part of Vietnam at least).

It would be better to provide a link to an official Unicode character set mapping table and then use the gencodec.py script on this table.

I saw a reference to this processing tool in the Python provided encodings and tried to find a Unicode mapping table at the Unicode website but failed up to now. I'll try harder.

• For Vietnamese, Python already provides cp1258 - how much is this encoding used in comparison to e.g. TCVN 5712:1993 ?

To be efficient at typing Vietnamese, you need a keyboard input software (Vietkey and Unikey being the most used). Microsoft tried to create dedicated Vietnamese encoding (cp1258) and keyboard, but I never saw or heard about its adoption at any place. Knowing the way Vietnamese users use their computer, I would say it probably has never been in real use.

• Vietnamese encodings: http://www.panl10n.net/english/outputs/Survey/Vietnamese.pdf

In this sentence you can see the most used old encodings in Vietnam: “On the Linux platform, fonts based on Unicode [6], TCVN, VNI and VPS [7] encodings can be adequately used to input Vietnamese text.”

This is not only the most used on Linux (in fact, on Linux we have to use Unicode, mostly because of the problem I explained before) but also on Windows. I don't know the situation for Mac OS or other OS though.

My goal is to add these encodings into Python, to help Vietnam make its steps into Unicode.

• East Asian encodings: http://www.unicode.org/iuc/iuc15/tb1/slides.pdf

This document tells: “Context is critical—Unicode is considered the “newer” character set in the context of this talk.” It was written in the goal to put Unicode as a replacement for all already covered charsets, which then shall become obsolete. So, of course, in this point of view, every 8 bits Vietnamese charsets are obsolete. But it doesn't mean there are not of use anymore, not at all!

[malemburg]

Thanks for your answers. I think the best way forward would be to some
up with an official encoding map of the TCVN 5712:1999 encoding,
translate that into a format that gencodec.py can use and then
add the generated codec to Python 3.5.

We can then add the reference to the original encoding map
to the generated file.

This is how we've added a couple of other encodings for which there
were no official Unicode mapping files as well.

Please also provide a patch for the documentation and sign the
Python contrib form:

https://www.python.org/psf/contrib/contrib-form/

Thanks,

Marc-Andre Lemburg
eGenix.com

[progfou]

I will prepare the official encoding map(s) based on the standard(s).

I'll also have to check which encoding correspond to my current encoding map, since this is the one useful in real life.

Please also provide a patch for the documentation

I currently have no idea how to do this. Could you point me to a documentation sample or template please?

and sign the Python contrib form:
https://www.python.org/psf/contrib/contrib-form/

I did it yesterday. The form tells it can take days to be integrated, but I did receive the signed document as a confirmation.

Thanks for your concern, J.C.

[progfou]

A note to inform about my progress. (I had a long period without free time at hand)

While seeking (again) official documents on the topic, I mainly found a lot of non-official ones, but some are notorious enough to use them as references.

I am now in the process of creating the requested patch. I am currently studying the proper way to do it. I expect to get it ready this weekend, in the hope to have it accepted for Python 3.5.

[progfou]

I failed to find anything about TCVN 5712:1999 except the official announcement of it superseding TCVN 5712:1993 on TCVN's website. I also was not able to find any material using TCVN 5712:1999. My guess is that TCVN 6909:2001 having been released only 2 years after, TCVN 5712:1999 probably had no time to get in real use.

Anyway, TCVN 5712:1993 is the real one, the one having been in used for almost 2 decades. So this is why I provided codec tables for this one.

There is 3 flavors of it. The most used one for documents is the third one (TCVN 5712:1993 VN3). It is used with the so called “ABC fonts” which are of common knowledge in Vietnam. But the first one may be of use in databases. I never got access to real (large) Vietnamese databases so I can't confirm it for sure. I still provided the 3 flavors, just in case.

Still, since VN3 is a subset of VN2, which itself is a subset of VN1, you may choose to only include the first one, TCVN 5712:1993 VN1, I leave this up to you. FYI, GNU Recode and Glibc Iconv currently implement "tcvn" as VN1. (but the Epson printer company implement VN3…)

[progfou]

Marc-Andre, about “Please also provide a patch for the documentation”, could you please guide me on this?

I can write some documentation, but I simply don't know in what form you expect it. Could you point me to some examples please?

[malemburg]

Jean Christophe: Please have a look at the patch for ticket http://bugs.python.org/issue22681 as example of the doc patch.

Thanks.

[serhiy-storchaka]

Or bpo-22682.

Needed:

• The codec itself (in Lib/encodings/ directory).
• Entries in aliases table (Lib/encodings/aliases.py).
• A row in encodings table (Doc/library/codecs.rst).
• An entry in What's New (Doc/whatsnew/3.5.rst).
• May be addition in encodings table in Lib/locale.py. May be regenerate aliases table.
• An entry in all encodings list (Lib/test/test_codecs.py) and mentions in some other tests (Lib/test/test_unicode.py, Lib/test/test_xml_etree.py).

[matorban]

Here this is a patch to added vietnamese codec tcvn. I am not sure about the name of the codecs...tcvn5712, tcvn5712_3 ?
test_xml_etree, test_codesc, test_unicode is running. Is it enough for the doc?

[serhiy-storchaka]

Since no Unicode mapping table is found at the Unicode website, we need at least the link to public official document that specifies the encoding.

If VN3 is a subset of VN2, which itself is a subset of VN1, VN1 definitely looks the most preferable choice for including in Python distribution. Especially if it was chosen by other popular software.