<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [Kc] cleaning up unicode?</TITLE>
</HEAD>
<BODY>
<BR>
<P><FONT SIZE=2>John Reinke wrote:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I have a text file which contains a character I am guessing as being</FONT>
<BR><FONT SIZE=2>> considered a unicode character. It is the letter 'u' with the accent</FONT>
<BR><FONT SIZE=2>> mark over it that looks like an apostrophe, which appears in some text</FONT>
<BR><FONT SIZE=2>> editors as <FA>.</FONT>
</P>
<P><FONT SIZE=2>This need not be unicode, and likely isn't. Many ASCII extended character sets contain the accented vowels. It is most likely that you're using one of those.</FONT></P>
<P><FONT SIZE=2> </FONT>
<BR><FONT SIZE=2>> I'd like to convert any of those characters to be regular ASCII</FONT>
<BR><FONT SIZE=2>> characters, most likely with a tr command, but I haven't been able to</FONT>
<BR><FONT SIZE=2>> find a way to match that character. Any suggestions?</FONT>
</P>
<P><FONT SIZE=2>Are you saying that:</FONT>
</P>
<P><FONT SIZE=2> s/üûùú/u/g;</FONT>
<BR><FONT SIZE=2> s/Ü/U/g;</FONT>
</P>
<P><FONT SIZE=2>Doesn't work?</FONT>
</P>
<P><FONT SIZE=2> </FONT>
<BR><FONT SIZE=2>> Once I can match those characters, is there an easy way to convert all</FONT>
<BR><FONT SIZE=2>> accented characters to their non-accented counterparts, such that the</FONT>
<BR><FONT SIZE=2>> accent will disappear but the same letter will remain?</FONT>
</P>
<P><FONT SIZE=2>I'm not sure I'd recommend this. The accents after all do represent different vowel sounds. And the simple conversion you're suggesting doesn't correctly convert them to their non-accented counterparts. For instance, Germans typing on keyboards without umlauts (the 2 dots above ö) would type schön as schoen... There's another word schon which has a different meaning. </FONT></P>
<P><FONT SIZE=2>Then again my sum knowledge of locale and internationalization issues is pretty meager. I don't know if it'd even be possible to do what you suggest unless you could identify the language and locale for each section of text you wished to transform. Even then I'm not aware of any modules to perform such conversions. Though that isn't to say they don't exist.</FONT></P>
<P><FONT SIZE=2>It'd be nice to know what you're actually trying to accomplish. If for instance you're wanting to munge this text file so it can be displayed in HTML... you might be interested in HTML::Entities:</FONT></P>
<P><FONT SIZE=2> <A HREF="http://search.cpan.org/author/GAAS/HTML-Parser-3.27/lib/HTML/Entities.pm" TARGET="_blank">http://search.cpan.org/author/GAAS/HTML-Parser-3.27/lib/HTML/Entities.pm</A></FONT>
</P>
<P><FONT SIZE=2>Which for example allows you to do this:</FONT>
</P>
<P><FONT SIZE=2> $input = "vis-à-vis Beyoncé's naïve papier-mâché résumé";</FONT>
<BR><FONT SIZE=2> print encode_entities($in), "\n"</FONT>
</P>
<P><FONT SIZE=2>which would result in:</FONT>
</P>
<P><FONT SIZE=2> vis-&agrave;-vis Beyonc&eacute;'s na&iuml;ve papier-m&acirc;ch&eacute;</FONT>
<BR><FONT SIZE=2> r&eacute;sum&eacute;</FONT>
</P>
<P><FONT SIZE=2>--</FONT>
<BR><FONT SIZE=2>Garrett Goebel</FONT>
<BR><FONT SIZE=2>IS Development Specialist</FONT>
</P>
<P><FONT SIZE=2>ScriptPro Direct: 913.403.5261</FONT>
<BR><FONT SIZE=2>5828 Reeds Road Main: 913.384.1008</FONT>
<BR><FONT SIZE=2>Mission, KS 66202 Fax: 913.384.2180</FONT>
<BR><FONT SIZE=2>www.scriptpro.com garrett@scriptpro.com </FONT>
</P>
</BODY>
</HTML>