iconv_codec NG

[hyeshik]

BPO	713820
Nosy	@loewis, @hyeshik
Files	python-iconvcodec-ng.diff.gz: patch (rev. 2)

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2003-04-03.05:07:58.000>
created_at = <Date 2003-04-02.11:10:16.000>
labels = ['library']
title = 'iconv_codec NG'
updated_at = <Date 2003-04-03.05:07:58.000>
user = 'https://github.com/hyeshik'

bugs.python.org fields:

activity = <Date 2003-04-03.05:07:58.000>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-04-02.11:10:16.000>
creator = 'hyeshik.chang'
dependencies = []
files = ['5152']
hgrepos = []
issue_num = 713820
keywords = ['patch']
message_count = 2.0
messages = ['43268', '43269']
nosy_count = 2.0
nosy_names = ['loewis', 'hyeshik.chang']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue713820'
versions = ['Python 2.3']

[hyeshik]

This new implementation of iconv_codec resolves
problems of current
implementations below:

• Having a reentrant context vulnerable point:
  encoder and/or decoder
  can be called multiple level in a same time when PEP-293
  codec error
  callback can call another iconv encoder session, too.
  So, all encode/
  decode session must open their own iconv session but
  the current
  implementation shares the iconv session in the whole
  codec life time.

• StreamReader can't work correctly: Because iconv
  keeps their context
  private, StreamReader can't work smart only with
  encode/decode function.
  Also, handling EINVAL and giving pending characters
  from previous data
  to error callback is very weak in the current
  implementation.

• Putting a replacement character as just '?' is not
  safe for many
  encodings: On stateful encodings and non-byte stream
  encodings, we need
  to encode with iconv even for the replacement character.

• Can't use encoding names including - and
  uppercases: Because codec
  subsystem changes - to _ and uppercases to lowercases,
  we can't pass
  them to the iconv_codec module without loss. For
  example, we need the
  next aliases to use CJK encodings on Sun iconv:

# simplified chinese
"euc_cn": "zh_CN.euc",
"iso_2022_zh": "zh_CN.iso2022-CN",
"gbk": "zh_CN.gbk",
"cp935": "zh_CN-cp935",

# traditional chinese
"euc_tw": "zh_TW.euc",
"iso_2022_tw": "zh_TW.iso2022-7",
"big5": "zh_TW.big5",
"cp937": "zh_TW.cp937",

# japanese
"iso_2022_jp": "ISO-2022-JP",
"euc_jp": "eucJP",
"shift_jis": "PCK",

# korean
"euc_kr": "ko_KR.euc",
"iso_2022_kr": "ISO-2022-KR",
"johab": "ko_KR.johap",
"cp932": "ko_KR.cp932",
"cp949": "ko_KR.cp949",

• Can't try multiple unicode encodings or methods: On
  some iconv
  implementations like of HP-UX or Solaris, UCS2 ->
  ISO-8859-1 is
  available but UCS2 -> euc-kr isn't avaiable and only
  UTF-8 -> euc-kr is.

And, many multibyte codecs such as CJK or iconv might
have duplicated
code for processing error callbacks and handling
Streams. So, I splitted
them out to another source. CJK and iconv codecs can
share them just in
source level by putting multibytecodec.c to Modules/
and linking the
file to each of the codecs. Alternatively, if
multibytecodec.c goes
to Python/ and is linked to main python library, the
codecs can be
compiled and loaded by themselves.
multibytecodec.c, the common multibyte codec framework
can be used by
any usual multibyte encodings. By using it, some codec
writer can create
a codec for his/her multibyte encodings without any
care for handling
error callbacks or implementing StreamReader structure.
I wrote CJK codecs using it. and
will submit a patch in an individual patch report.

[loewis]

Logged In: YES
user_id=21627

Because of complaints about its brokenness, I had to revert
the current iconv codec. It is extremely unlikely that such
code will be added for Python 2.3. So please update this
submission for the current CVS. There is no need to hurry,
though.