SPUG: ajax character encoding issue solved, but WHY?

[Johnston, Mark]

Phil,
 
    This is a shot in the dark, but is the encoding on the page that
interacts with the web service via AJAX by chance ISO-8859-1 (either set
explicitly or by default on the server)?  You didn't mention whether
"utf8" or "UTF-8" is the character encoding you are now specifying when
calling Encode::decode on the JSON text stream, so I'm not sure how that
might have affected the resulting text stream as it enters the wire
between the two processes.
 
    The world of character encoding is fraught with all kinds of
interesting situations.  Once upon a time a "byte" and "character" were
interchangeable, and all was well with the world ... 8^)) ...  In the
world of processes operating under different character encoding schemes,
however, sometimes the waters get muddled.  If the sender and receiver
are not in absolute agreement as to the character encoding of a stream
of bytes (_and_ capable of translating between character encodings when
necessary), the interpretation of what characters are meant by a
particular byte stream can be different, which leads to the type of
problem you describe.
 
     --Mark
 

________________________________

From: spug-list-bounces+mark.johnston=pnl.gov at pm.org
[mailto:spug-list-bounces+mark.johnston=pnl.gov at pm.org] On Behalf Of
Phil Mitchell
Sent: Wednesday, June 10, 2009 12:41 PM
To: spug-list at mail.pm.org
Subject: SPUG: ajax character encoding issue solved, but WHY?


I solved the character encoding issue that I wrote about earlier, but I
don't understand why the fix works. I'm hoping someone can explain this
to me!

The issue was that non-ascii chars were appearing as junk BUT only when
retrieved via ajax calls. Otherwise, they displayed fine. The junk
display was due to them being interpreted as ISO-8859-1, but I could not
figure out why the browser was interpreting that way. All my data is
handled as UTF-8.

The problem was fixed by calling Encode::decode on the data prior to
sending back via ajax. BUT WHY?

I am using the JSON view to render ajax responses, and it sets the
charset header correctly to UTF-8. Of course, even when you decode, perl
still represents as "internal" utf8. But why should this be necessary?

-- 
==========================
http://www.bikewise.org

2People citizen's network for climate action: http://www.2people.org

Greater Seattle Climate Dialogues: http://www.climatedialogues.org
==========================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/spug-list/attachments/20090616/de80bdfd/attachment.html>