Universal Unicode Codec for POSIX iconv

[hyeshik]

BPO	670715
Nosy	@malemburg, @loewis, @doerwalter, @hyeshik
Files	iconvcodec-1.txt: Patch iconvcodec-2.txt: Revised patch (r2) iconvcodec-3.txt diff-char.txt: Use '0' as test char. diff-debug.txt: Add debugging output to see what iconv() generated.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = <Date 2003-04-03.05:04:51.000>
created_at = <Date 2003-01-19.16:51:35.000>
labels = ['library']
title = 'Universal Unicode Codec for POSIX iconv'
updated_at = <Date 2003-04-03.05:04:51.000>
user = 'https://github.com/hyeshik'

bugs.python.org fields:

activity = <Date 2003-04-03.05:04:51.000>
actor = 'loewis'
assignee = 'loewis'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-01-19.16:51:35.000>
creator = 'hyeshik.chang'
dependencies = []
files = ['4938', '4939', '4940', '4941', '4942']
hgrepos = []
issue_num = 670715
keywords = ['patch']
message_count = 14.0
messages = ['42479', '42480', '42481', '42482', '42483', '42484', '42485', '42486', '42487', '42488', '42489', '42490', '42491', '42492']
nosy_count = 5.0
nosy_names = ['lemburg', 'loewis', 'doerwalter', 'tzot', 'hyeshik.chang']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue670715'
versions = ['Python 2.3']

[hyeshik]

Here's the unicode codec using POSIX iconv(3) library.

Tested on these platforms and seems to work:
FreeBSD/i386, FreeBSD/alpha, FreeBSD/ia64,
FreeBSD/sparc64, MacOS X/ppc, HP-UX/pa-risc2

This codec implementation supports PEP-293, also.

[malemburg]

Logged In: YES
user_id=38388

The patch looks good, but you'll need to add some form
of testing to underline the "seems to work" :-)

Some docs on how to use the codec would also be needed.

Martin von Loewis has written a similar codec some months ago.
Perhaps you two could get in touch and sort out the details ?!

[hyeshik]

Logged In: YES
user_id=55188

Thank you for comments. :->
I uploaded a new revised patch with unittest and some code
style fixes.

I saw Martin v. Loewis's iconvcodecs about a years ago.
His implementation is very neat, but it had a limit on error
handling due to recursive call.

[loewis]

Logged In: YES
user_id=21627

I'm quite happy with this patch, and will apply it shortly.
However, I am concerned that it is specific for GNU iconv.
IMO, there should be machinery to find out the "internal"
encoding, in case native the native iconv implementation is
used instead of GNU iconv.

[hyeshik]

Logged In: YES
user_id=55188

iconv implementations on commercial UNIXes are very scary.

Solaris implementation:
no support for CJK <-> UCS conversion.
They support UCS[24] only for iso-8859 and UTFs.

HP-UX implementation:
They have useless iconv. HP-UX iconv has no unicode support.

BSD implementation (Konstantin Chuguev's):
compatible with this patch (provides ucs-[24]-internal)

GLIBC implementation:
provides ucs-[24] and they are same with GNU iconv's
ucs-[24]-internal.
Because ucs-[24] of GNU/BSD implentation is big endian
always. We can't use ucs-[24] for every platform.

In conclusion, we must use 3rd party iconv on Solaris or HP-UX.
And, we need to detect whether the linked iconv is GNU/BSD
iconv or GLIBC iconv. (how?)

I'll investigate how to detect them, but ... :)

[loewis]

Logged In: YES
user_id=21627

Hmm. I see that Solaris does support conversion of CJK to
UTF-8. So even though we cannot convert into the internal
encoding, we could still convert through UTF-8.

Looking at /usr/lib/nls/iconv/config.iconv of HP-UX 11.11, I
see conversions from and to ucs2, for iso-8859, eucJP, sjis,
eucTW, big5, roc15, kore5, hp15CN, and many IBM code pages.

So I think the iconv codec should convert into the Python
internal representation if possible. If no encoding name for
that is known, it should convert to ucs2 (be) if possible,
or else to UTF-8; in all cases, it will then construct a
Unicode object from the resulting string.

[loewis]

Logged In: YES
user_id=21627

I have committed it now with minimal changes so that it
works on Linux, as

setup.py 1.138
__init__.py 1.15
iconv_codec.py 1.1
regrtest.py 1.117
NEWS 1.627

I will make further changes; please watch the CVS. If you
would like to make further changes, please submit patches
against the CVS.

[hyeshik]

Logged In: YES
user_id=55188

Thank you very much for your works.
I'm working on UCS endian detection and UCS transparent
wrapper for UTF-{8,16}. I'll submit new patch in a week.
Please feel free to change my codes because I'm not familiar
with python code convention and culture. :-)

[doerwalter]

Logged In: YES
user_id=89016

iconvcodec-3.txt does byteswapping under the following
conditions:
    #ifndef WORDS_BIGENDIAN
    #ifdef __GNU_LIBRARY__
Byteswapping is done before encoding to the whole input and
to every piece returned from iconv() for decoding.

Detecting whether byteswapping is neccessary might not work
reliably with the above tests. If this is the case, a test
call to iconv() should probably be done in
Modules/_conv_codec.c::init_iconv_codec() to determine
whether to byte swap or not.

Another possibility might be to use utf-16/utf-32 instead of
ucs-2/ucs-4.

One test still fails: test_sane(), because it uses the
internal encoding in Python, where the real endianness is of
course unknown. The test also assumes a narrow Python build.