php’s dumb xml parsing behavior
steve minutillo, author of feed on feeds, runs headlong into the execrable character encoding behavior of php’s xml parsing functions. hey, i was complaining about that just last year... (via phil ringnalda.)
and a related link, this article from the w3c explains how to deal with encoding issues in forms and has a nice regex that verifies whether a string is valid utf-8.
here’s some links culled from an i18n discussion on the twiki site:
Now that I've looked a bit more, there are many algorithms out there for charset detection, but most are aimed at HTML page auto-detection, and may well not work well for URLs:
- Frank Tang's charset detection links - includes simple Perl UTF-8 detector based on legal codings
- Excellent paper on Mozilla's 3-part algorithm using coding legality, character frequencies and two-character frequencies - detects the language as well as the encoding. Too complex for use on URLs, but looks very good.
- Discussion on IRC auto-detection of charsets
- Simple UTF-8 detector in C
- CPAN:Unicode::Japanese - includes auto-detection for various Japanese charsets
- CPAN:Encode::Guess - auto-detection from suitably dissimilar charsets (needs Perl 5.8)
- Browser detection for forms input datatypes including useful undocumented JavaScript to check IE's current charset (try this out now if you are using IE - see Sandbox.TestCharset).
- TextCat, tool for language detection - in Perl, OpenSource
i really need to write the slides for my talk at oscon, which will cover exactly this sort of thing.
