From aesop at fables.co.za  Wed Apr  9 07:38:43 2008
From: aesop at fables.co.za (aesop at fables.co.za)
Date: Wed, 09 Apr 2008 16:38:43 +0200
Subject: [Za-pm] introduction
Message-ID: <47FCF113.25311.130CF5F@aesop.fables.co.za>

Hello,

Just introducing myself with a small problem. Not a very perlish one 
although that is likely to change.

I have a flat file database (Clue) running on DOS still! 20 years use 
but am forced to migrate now.

Over the years I added some new fields into the database, and then 
later I moved one of them to put it in a better place. The new fields 
only get added into new records, there is no way to add the field 
into old records which can only be updated by pulling out on an edit 
and then resaving. So I have some records with the same field in 
different places, and older records without the new fields at all. 
Doesn't faze it.

Perl to the rescue. Although new to Perl I have got a number of 
routines up and running to fix these sort of inconsistencies in the 
database. I'll isolate this to two routines.

The whole database comprises 161 sequential files at present. NO CR 
or LF. Clue handles all formatting by an Alt-127 (little house) 
character followed by a letter/s. Open the file in the Midnite 
Commander editor and you have a one liner wandering 30+k off to the 
right.

One routine moves a field and its data in the record, another adds 
another field in if it doesn't exist already. I do this under Linux. 
The problem is after running the routines, reported successfully by 
various Perl test routines (like 6699 records, 6699 of each field, 
all fields in the correct order), we get problems when the database 
is imported back into Clue - like will not count the records 
correctly under certain circumstances. Strictly speaking the move 
routine causes no problem, it is the add-in one that does.

I have spent hours checking and refining the Perl routines, trying 
variants in methodology, more hours spent methodically checking by 
hand the modified Clue files, all to no avail. I can see nothing 
wrong but of course there must be. I run them under DOS, I modify 
them in Linux under Perl 5.8, I copy them back onto the DOS machine, 
and then it goes wrong.  

So I don't think the question is necessarily a Perl one at present, 
but it is this. Can anything happen to the files (that don't have CRs 
or LFs in them for Perl to handle in any way and we slurp in a file 
at a time as one string to process  ~= m/regex/replace/gx) in the 
transition from one system to another? 

I need to have that clear in my mind, that it is not an operating 
system thing before I stare at my Perl scripts yet again. It can only 
be something in the files that has been changed but yet ....

A couple of good answers can give direction to my search for the 
problem, and thanks in advance.

Thanks
Anne----
Anne Wainwright


From tvilliers at lastminute.com  Wed Apr  9 07:43:06 2008
From: tvilliers at lastminute.com (Tielman de Villiers)
Date: Wed, 09 Apr 2008 15:43:06 +0100
Subject: [Za-pm] introduction
In-Reply-To: <47FCF113.25311.130CF5F@aesop.fables.co.za>
References: <47FCF113.25311.130CF5F@aesop.fables.co.za>
Message-ID: <1207752186.28122.23.camel@lmn12961.lastminute.com>


On Wed, 2008-04-09 at 16:38 +0200, aesop at fables.co.za wrote:
> So I don't think the question is necessarily a Perl one at present, 
> but it is this. Can anything happen to the files (that don't have CRs 
> or LFs in them for Perl to handle in any way and we slurp in a file 
> at a time as one string to process  ~= m/regex/replace/gx) in the 
> transition from one system to another? 
> 

Make the smallest possible sample of the problem; then take an octal
dump of the file before and after conversion: od -tc <filename>


--tielman







From jonathan at hst.org.za  Thu Apr 10 02:25:37 2008
From: jonathan at hst.org.za (Jonathan McKeown)
Date: Thu, 10 Apr 2008 11:25:37 +0200
Subject: [Za-pm] introduction
In-Reply-To: <47FCF113.25311.130CF5F@aesop.fables.co.za>
References: <47FCF113.25311.130CF5F@aesop.fables.co.za>
Message-ID: <200804101125.38372.jonathan@hst.org.za>

On Wednesday 09 April 2008 16:38, aesop at fables.co.za wrote:

> I have a flat file database (Clue) running on DOS still! 20 years use
> but am forced to migrate now.
[snip]
> The whole database comprises 161 sequential files at present. NO CR 
> or LF. Clue handles all formatting by an Alt-127 (little house)
> character followed by a letter/s. Open the file in the Midnite
> Commander editor and you have a one liner wandering 30+k off to the
> right.
>
> One routine moves a field and its data in the record, another adds
> another field in if it doesn't exist already. I do this under Linux.
> The problem is after running the routines, reported successfully by
> various Perl test routines (like 6699 records, 6699 of each field,
> all fields in the correct order), we get problems when the database
> is imported back into Clue - like will not count the records
> correctly under certain circumstances. Strictly speaking the move
> routine causes no problem, it is the add-in one that does.

Pure guesswork at this stage, but my money would be on some sort of indexing 
issue, either in a separate file or embedded, perhaps in the letter(s) after 
the chr(127), which is causing Clue to be confused about where records start 
and end.

Is Clue producing correct data for (at least some of) the records when 
queried? How far off is the record count? What if anything is the 
significance of the letter(s) after the DEL (the ASCII 127 character)? Is 
there any documentation on the Web about Clue and its file formats?

> I have spent hours checking and refining the Perl routines, trying
> variants in methodology, more hours spent methodically checking by
> hand the modified Clue files, all to no avail. I can see nothing
> wrong but of course there must be. I run them under DOS, I modify
> them in Linux under Perl 5.8, I copy them back onto the DOS machine,
> and then it goes wrong.
>
> So I don't think the question is necessarily a Perl one at present,
> but it is this. Can anything happen to the files (that don't have CRs
> or LFs in them for Perl to handle in any way and we slurp in a file
> at a time as one string to process  ~= m/regex/replace/gx) in the
> transition from one system to another?

I don't think it should fiddle about with anything other than line-endings - 
which as you say don't occur in the file. Checking the filesizes should let 
you know if there are any unexpected differences in size (from mystery bytes 
being added in somehow), and if there are changes you can use diff or cmp on 
the Linux box to find out what they are.

> I need to have that clear in my mind, that it is not an operating
> system thing before I stare at my Perl scripts yet again. It can only
> be something in the files that has been changed but yet ....

How big are the data files and your Perl routines? Provided it's not too huge 
(and of course that your data isn't confidential) you can email a sample of 
the data and a copy of your code to me direct and I'd be prepared to take a 
look and see if anything jumps out at me.

Sorry, lots of questions but no real answers...

Jonathan

From aesop at fables.co.za  Sat Apr 12 05:10:28 2008
From: aesop at fables.co.za (aesop at fables.co.za)
Date: Sat, 12 Apr 2008 14:10:28 +0200
Subject: [Za-pm] introduction
In-Reply-To: <1207752186.28122.23.camel@lmn12961.lastminute.com>
References: <47FCF113.25311.130CF5F@aesop.fables.co.za>,
	<1207752186.28122.23.camel@lmn12961.lastminute.com>
Message-ID: <4800C2D4.22160.FBA65D@aesop.fables.co.za>

Tielman, hello,

Thank you for the idea, sounds good although I have never used that. 
I'll try this out this weekend.

Regards
Anne

Your Wednesday, April 09, 2008 email from Tielman de Villiers  said 
thus::

> Make the smallest possible sample of the problem; then take an octal
> dump of the file before and after conversion: od -tc <filename>
----
Anne Wainwright


From aesop at fables.co.za  Sat Apr 12 05:29:59 2008
From: aesop at fables.co.za (aesop at fables.co.za)
Date: Sat, 12 Apr 2008 14:29:59 +0200
Subject: [Za-pm] introduction
In-Reply-To: <200804101125.38372.jonathan@hst.org.za>
References: <47FCF113.25311.130CF5F@aesop.fables.co.za>,
	<200804101125.38372.jonathan@hst.org.za>
Message-ID: <4800C767.11440.10D864A@aesop.fables.co.za>

Jonathan, hello,

Thank you for your comments.

Your Thursday, April 10, 2008 email from Jonathan McKeown  said 
thus::

> Pure guesswork at this stage, but my money would be on some sort of
> indexing issue, either in a separate file or embedded, perhaps in the
> letter(s) after the chr(127), which is causing Clue to be confused
> about where records start and end.

There is no separate indexing file that I have ever found. Each of 
the 161 files has a small field at the beginning that echoes the 
search field of the first record in that file. The search field is 
generated automatically from Author & Title fields on saving a new 
record.  It is that simple - more or less.

I don't think we should move too far out of the ambit of the perl 
subject matter for discussion on this list so will get back to you 
direct with some answers later if the od thingy suggested by Tielman 
is not suggestive of where the problem may lie.

My perl scripts are small, one screen with comments which I now find 
with -p -i switches can be substantially reduced in size as I can 
jettison all the file creation/saving/renaming baggage. Probably 
Jeffrey Friedl would cast them all as simple one-liners!

The Clue files of which there are 161 are typically < 30k each.

Regards
Anne



----
Anne Wainwright


From aesop at fables.co.za  Thu May  8 09:20:11 2008
From: aesop at fables.co.za (Anne Wainwright)
Date: Thu, 08 May 2008 18:20:11 +0200
Subject: [Za-pm] ascii high order character conversion
Message-ID: <4823445B.13018.33FC3A@aesop.fables.co.za>

Hi.

I have data prepared on a dos programme that involves high order 
characters, like european letters with umlauts, cedillas, acute and grave 
accents etc.

I have a dos utility that I wrote that converts all of these to plain 
unaccented characters, a simple replacement operation. The reason being 
that in moving the data to Windows it does not show them correctly and this 
was the easiest way to go at the time. Now I am away from that route and 
want to build this into my perl database conversion routine (convert from 
proprietary to delimited).

Now I am wondering if there is an easier way in perl than doing a s/// for 
each of the characters used. I looked in the Perl Cookbook, and had a 
wander through the CPAN modules, but nothing struck me as specific for the 
task in hand.

Not that lines of s/// wouldn't do the job, but I wondered if there was a 
more concise way of programming this to convert either to the plain 
unaccented character or to the correct windows character.

[maybe I must study the "perlebcdic Considerations for running Perl on 
EBCDIC platforms" found on CPAN which looks like it might be a guide. 
suggests tr///   , will absorb this evening]

Had hoped for a ready module from CPAN, but see nothing.

Any ideas gratefully received on what must have been a common problem some 
years back?


Regards
Anne
----
Anne Wainwright


From tvilliers at Lastminute.com  Thu May  8 09:29:40 2008
From: tvilliers at Lastminute.com (Tielman De Villiers)
Date: Thu, 8 May 2008 17:29:40 +0100
Subject: [Za-pm] ascii high order character conversion
In-Reply-To: <4823445B.13018.33FC3A@aesop.fables.co.za>
References: <4823445B.13018.33FC3A@aesop.fables.co.za>
Message-ID: <016F0AD55E0CD143A2032456E6DA2C0A0F480CE8@LMNUKMXV01.corp.lastminute.com>

It depends how your input data is formatted.

If it is UTF8 encoded, in double bytes, and if you have perl 5.6.1 and
above, use Unicode::Normalize:

use Unicode::Normalize qw/:all/;
...
$string =~ s/([\x80-\xFF])/substr(decompose($1),0,1)/eg;

If it's formatted by one or other Windows app, with special characters
such as the Euro symbol single byte encoded, then your input is probably
CP1252. I'm not sure about the available conversion modules.

Alternatively, just run your whole input file through the libiconv C
library:

iconv --from-code=ISO-8859-1 --to-code=UTF-8


--tielman











-----Original Message-----
From: za-pm-bounces+tvilliers=lastminute.com at pm.org
[mailto:za-pm-bounces+tvilliers=lastminute.com at pm.org] On Behalf Of Anne
Wainwright
Sent: 08 May 2008 17:20
To: za-pm at pm.org
Subject: [Za-pm] ascii high order character conversion

Hi.

I have data prepared on a dos programme that involves high order 
characters, like european letters with umlauts, cedillas, acute and
grave 
accents etc.

I have a dos utility that I wrote that converts all of these to plain 
unaccented characters, a simple replacement operation. The reason being 
that in moving the data to Windows it does not show them correctly and
this 
was the easiest way to go at the time. Now I am away from that route and

want to build this into my perl database conversion routine (convert
from 
proprietary to delimited).

Now I am wondering if there is an easier way in perl than doing a s///
for 
each of the characters used. I looked in the Perl Cookbook, and had a 
wander through the CPAN modules, but nothing struck me as specific for
the 
task in hand.

Not that lines of s/// wouldn't do the job, but I wondered if there was
a 
more concise way of programming this to convert either to the plain 
unaccented character or to the correct windows character.

[maybe I must study the "perlebcdic Considerations for running Perl on 
EBCDIC platforms" found on CPAN which looks like it might be a guide. 
suggests tr///   , will absorb this evening]

Had hoped for a ready module from CPAN, but see nothing.

Any ideas gratefully received on what must have been a common problem
some 
years back?


Regards
Anne
----
Anne Wainwright

_______________________________________________
Za-pm mailing list
Za-pm at pm.org
http://mail.pm.org/mailman/listinfo/za-pm

From jonathan at hst.org.za  Fri May  9 01:49:42 2008
From: jonathan at hst.org.za (Jonathan McKeown)
Date: Fri, 9 May 2008 10:49:42 +0200
Subject: [Za-pm] ascii high order character conversion
In-Reply-To: <4823445B.13018.33FC3A@aesop.fables.co.za>
References: <4823445B.13018.33FC3A@aesop.fables.co.za>
Message-ID: <200805091049.42218.jonathan@hst.org.za>

On Thursday 08 May 2008 18:20, Anne Wainwright wrote:
> Hi.
>
> I have data prepared on a dos programme that involves high order
> characters, like european letters with umlauts, cedillas, acute and grave
> accents etc.

This brings you into the messy realms of character sets and encodings. I would 
guess if the original data was prepared in DOS the encoding would be either 
cp850 (European) or cp437 (extended American). I would further guess cp437 - 
computers sold here tend to be configured as US rather than European.

> Not that lines of s/// wouldn't do the job, but I wondered if there was a
> more concise way of programming this to convert either to the plain
> unaccented character or to the correct windows character.

The Windows character set would be cp1252. This is similar, but not identical, 
to latin-1 (iso-8859-1). Depending on the destination for your data I'd 
suggest going for either latin-1 - likely to be supported - or utf-8 - 
support is patchy in some tools, although Perl handles it fine.

> [maybe I must study the "perlebcdic Considerations for running Perl on
> EBCDIC platforms" found on CPAN which looks like it might be a guide.
> suggests tr///   , will absorb this evening]

For changing fixed characters into other fixed characters tr/// is much faster 
than regexes. Bear in mind you've got a huge amount of documentation 
(including perlebcdic) installed with your perl - try perldoc perltoc for a 
table of contents.

> Had hoped for a ready module from CPAN, but see nothing.

Have a look at Encoding and family, which have been in the core since perl 
5.7.3 (dev version of 5.8); also perlIO layers which will do the conversion 
on the fly (I think that may be a 5.8 thing too).

Jonathan

From aesop at fables.co.za  Tue May 13 03:20:41 2008
From: aesop at fables.co.za (Anne Wainwright)
Date: Tue, 13 May 2008 12:20:41 +0200
Subject: [Za-pm] ascii high order character conversion
In-Reply-To: <016F0AD55E0CD143A2032456E6DA2C0A0F480CE8@LMNUKMXV01.corp.lastminute.com>
References: <4823445B.13018.33FC3A@aesop.fables.co.za>,
	<016F0AD55E0CD143A2032456E6DA2C0A0F480CE8@LMNUKMXV01.corp.lastminute.com>
Message-ID: <48298799.26413.9CA987@aesop.fables.co.za>

Jonathan & Tielman,

Thanks a lot, I think that I can make something out of that. The input is a 
 dos file (untainted by Windows) and I am sure that you are right about 
cp437.

I didn't have perldoc on until very recently, somehow it never got on. So I 
have only just realised what a great resource it is.

I do note that when using the editor in linux, either gedit when working on 
my pearls, or the inbuilt one in mc when having a squizz in the directory, 
that many of the characters don't show up. Like my Alt-127 (DEL or little 
house) shows up as a block (an outline square character). And things like C-
cedilla also show as a block. I suppose I should look at the editor 
settings. I know that gedit does not detect it as cp437 so a bit of 
investigation needed there.

This of course fouls up tr/// because it only trs discrete characters and I 
don't think it very smart to paste in blocks which you can't read even if 
they would would work (which I think they do, at least in regexes which I 
tried out before I found \x) and don't think you can put in \x7F for 
instance into tr///

Written in dos, modified on linux perl (5.8), sent over to Windoze! 
Hopefully one day to some or other sql database but that is too big a step 
today.

Thanks
Anne

> It depends how your input data is formatted.
> 
> If it is UTF8 encoded, in double bytes, and if you have perl 5.6.1 and
> above, use Unicode::Normalize:
> 
> use Unicode::Normalize qw/:all/;
> ...
> $string =~ s/([\x80-\xFF])/substr(decompose($1),0,1)/eg;
> 
> If it's formatted by one or other Windows app, with special characters
> such as the Euro symbol single byte encoded, then your input is probably
> CP1252. I'm not sure about the available conversion modules.
> 
> Alternatively, just run your whole input file through the libiconv C
> library:
> 
> iconv --from-code=ISO-8859-1 --to-code=UTF-8
> 
> 
> --tielman
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: za-pm-bounces+tvilliers=lastminute.com at pm.org
> [mailto:za-pm-bounces+tvilliers=lastminute.com at pm.org] On Behalf Of Anne
> Wainwright
> Sent: 08 May 2008 17:20
> To: za-pm at pm.org
> Subject: [Za-pm] ascii high order character conversion
> 
> Hi.
> 
> I have data prepared on a dos programme that involves high order 
> characters, like european letters with umlauts, cedillas, acute and
> grave 
> accents etc.
> 
> I have a dos utility that I wrote that converts all of these to plain 
> unaccented characters, a simple replacement operation. The reason being 
> that in moving the data to Windows it does not show them correctly and
> this 
> was the easiest way to go at the time. Now I am away from that route and
> 
> want to build this into my perl database conversion routine (convert
> from 
> proprietary to delimited).
> 
> Now I am wondering if there is an easier way in perl than doing a s///
> for 
> each of the characters used. I looked in the Perl Cookbook, and had a 
> wander through the CPAN modules, but nothing struck me as specific for
> the 
> task in hand.
> 
> Not that lines of s/// wouldn't do the job, but I wondered if there was
> a 
> more concise way of programming this to convert either to the plain 
> unaccented character or to the correct windows character.
> 
> [maybe I must study the "perlebcdic Considerations for running Perl on 
> EBCDIC platforms" found on CPAN which looks like it might be a guide. 
> suggests tr///   , will absorb this evening]
> 
> Had hoped for a ready module from CPAN, but see nothing.
> 
> Any ideas gratefully received on what must have been a common problem
> some 
> years back?
> 
> 
> Regards
> Anne
> ----
> Anne Wainwright
> 
> _______________________________________________
> Za-pm mailing list
> Za-pm at pm.org
> http://mail.pm.org/mailman/listinfo/za-pm

----
Anne Wainwright


From aesop at fables.co.za  Tue May 13 03:38:07 2008
From: aesop at fables.co.za (Anne Wainwright)
Date: Tue, 13 May 2008 12:38:07 +0200
Subject: [Za-pm] Databases & Perl
Message-ID: <48298BAF.929.AC9E4E@aesop.fables.co.za>

Hi,

A ramble of a know nothing, for comment and suggestions.

Bear in mind that apart from my dos database I am fairly competant with 
Paradox. For the uninitiated, this Windows app was originally from Borland, 
later from Corel, and provides a database with integrated design 
environment where you can add your buttons and tables etc to a form and 
then add code to those objects to make it go the way you want. Far more 
powerful than Excel was. Design of forms and reports is a snap.

Now realising that even Paradox's time has come with it on the back burner 
at Corel and perhaps not likely to survive many more M$ operating system 
upgrades, the thought to migrate to something like mySQL has been in my 
mind as the ultimate way out to freedom from M$.

I see and note that Perl has many modules and features designed to enable 
it to work with an sql type database. The questions in my mind revolve 
around some complex applications that I have created in Paradox. Am I going 
to have to create them with a much lower level language than I am used to? 
Paradox's ObjectPal provides a fully integrated development environment 
where you drag and drop and add some code to the objects already dragged, 
sized, coloured, and dropped.

Further confusion because so much of what I read (well, scan) on sql 
databases refers to their use for internet applications and how to view the 
data through your browser. Has Perl passed its sell-by date in this field? 
Other languages such as php and many others appear to hog the limelight in 
this field. Is this now the de facto way to interface with a database - via 
a browser that perhaps dispenses with the whole procedure to create a full 
application interface? Is PHP and your browser the quick and easy way even 
if you are only using the application on your own intranet (mine has 4 
machines at the most) and not over the internet?

I am wondering which path to slowly follow. Maybe I should investigate 
things like Firebird database which seems to offer an already established 
graphical interface and forget working from the ground up with Perl.

As you can see I am lost in the desert here. Any comments on how the ground 
is laid out these days and which path I should take are most welcome. 
Rather post to the list as my email client will handle these automagically 
for me.

Thanks
Anne
----
Anne Wainwright


From jonathan at hst.org.za  Tue May 13 05:48:05 2008
From: jonathan at hst.org.za (Jonathan McKeown)
Date: Tue, 13 May 2008 14:48:05 +0200
Subject: [Za-pm] Databases & Perl
In-Reply-To: <48298BAF.929.AC9E4E@aesop.fables.co.za>
References: <48298BAF.929.AC9E4E@aesop.fables.co.za>
Message-ID: <200805131448.06214.jonathan@hst.org.za>

On Tuesday 13 May 2008 12:38, Anne Wainwright wrote:
> Hi,
>
> A ramble of a know nothing, for comment and suggestions.
>
> Bear in mind that apart from my dos database I am fairly competant with
> Paradox. For the uninitiated, this Windows app was originally from Borland,
> later from Corel, and provides a database with integrated design
> environment where you can add your buttons and tables etc to a form and
> then add code to those objects to make it go the way you want. Far more
> powerful than Excel was. Design of forms and reports is a snap.
[snip]
> I see and note that Perl has many modules and features designed to enable
> it to work with an sql type database. The questions in my mind revolve
> around some complex applications that I have created in Paradox. Am I going
> to have to create them with a much lower level language than I am used to?
> Paradox's ObjectPal provides a fully integrated development environment
> where you drag and drop and add some code to the objects already dragged,
> sized, coloured, and dropped.

The problem you have with developing GUI applications is that producing a 
graphical interface is complicated. It's easier if you have a development 
environment that lets you drag and drop your components in the way that 
Access or Paradox does.

> Further confusion because so much of what I read (well, scan) on sql
> databases refers to their use for internet applications and how to view the
> data through your browser.

Another way to develop an application which looks good and is easy to use with 
a mouse is to make it a Web application. This is very popular because the 
application now only needs a Web browser installed on the client rather than 
the whole application, and using HTML as your output format offloads a lot of 
the nonsense involved in managing the gory details of screen layout onto the 
browser.

> Has Perl passed its sell-by date in this field?

Now that's almost a religious question. Perl has been around for 20 years and 
for much of that time, it was King of the Web. It is still good at what it 
does, and the libraries that are available from CPAN (including many web 
tools) are still actively developed (apparently 25% of the 13,000+ modules on 
CPAN have a most recent version upload date in the last 4 months, and half of 
them have been updated in the last 17 months).

On the other hand, it's harder to learn than PHP (I would argue that once 
you've learned Perl, it's easier to program well with than PHP - others would 
disagree), and there seem to be fewer Perl programmers around, especially in 
South Africa for some reason, to help you out.

> As you can see I am lost in the desert here. Any comments on how the ground
> is laid out these days and which path I should take are most welcome.
> Rather post to the list as my email client will handle these automagically
> for me.

If I were in your shoes, I would probably take the web approach: it gives you 
much more flexibility in the long run (can easily move from Windows to Linux 
etc). Having decided that, I would probably look at web frameworks, such as 
Catalyst for Perl (there are similar frameworks for other languages), which 
make the whole process less painful (the buzzword to look out for is CRUD, 
Create-Retrieve-Update-Delete, describing the four basic operations you need 
on a database. Googling for CRUD database will get you some more 
information).

Bear in mind another of Perl's strengths is the DBI database interaction 
module. In PHP, if you write your application for MySQL and then change your 
mind and decide to use a different database, you have to find and replace 
every single mysql_**** function with the equivalents for the new database. 
(This is starting to change - there is at least one database access layer 
available now). In Perl, provided you've used DBI.pm, all you need to do is 
change your connect statement. DBI provides a standard interface to your data 
and automatically loads the right DBD (database-dependent) module for the 
database you're connecting to.

Jonathan

From tvilliers at lastminute.com  Tue May 13 10:30:13 2008
From: tvilliers at lastminute.com (De Villiers, Tielman)
Date: Tue, 13 May 2008 18:30:13 +0100
Subject: [Za-pm] Databases & Perl
In-Reply-To: <200805131448.06214.jonathan@hst.org.za>
References: <48298BAF.929.AC9E4E@aesop.fables.co.za>
	<200805131448.06214.jonathan@hst.org.za>
Message-ID: <2BAF54A88C2F734C9911D3D4638850440101D091@sglonms02.Global.ad.sabre.com>

 

> -----Original Message-----
> On Behalf Of Jonathan McKeown
> Sent: 13 May 2008 13:48

> On Tuesday 13 May 2008 12:38, Anne Wainwright wrote:

> > Bear in mind that apart from my dos database I am fairly competant 
> > with Paradox. For the uninitiated, this Windows app was originally 
> > from Borland, later from Corel, and provides a database with 
> > integrated design environment where you can add your buttons and 
> > tables etc to a form and then add code to those objects to make it
go 
> > the way you want. Far more powerful than Excel was. Design 
> > of forms and reports is a snap.

Sound like MS Access to me.

> > Further confusion because so much of what I read (well, scan) on sql

> > databases refers to their use for internet applications and how to 
> > view the data through your browser.
> 
> Another way to develop an application which looks good and is 
> easy to use with a mouse is to make it a Web application. 
> This is very popular because the application now only needs a 
> Web browser installed on the client rather than the whole 
> application, and using HTML as your output format offloads a 
> lot of the nonsense involved in managing the gory details of 
> screen layout onto the browser.

Yep, this sounds like a good candidate for a webapp.

> > As you can see I am lost in the desert here. Any comments on how the

> > ground is laid out these days and which path I should take 
> > are most welcome.
> 
> If I were in your shoes, I would probably take the web 
> approach: it gives you much more flexibility in the long run 
> (can easily move from Windows to Linux etc). Having decided 
> that, I would probably look at web frameworks, such as 
> Catalyst for Perl (there are similar frameworks for other 
> languages), which make the whole process less painful.

You might also want to look at Mason, a Perl-based templating engine.

> 
> Bear in mind another of Perl's strengths is the DBI database 
> interaction module.

Definitely.


I'd recommend to spend a lot of time thinking about your data and 
how to convert to a relational database (PostgreSQL, MySQL or 
even Oracle XE) and making the *database* your application. Then 
you can play with CGI scripts/web frameworks/GUIs, knowing
all of these are just interfaces to your *real* application.

--tielman



From francois at busii.com  Fri May 16 07:15:58 2008
From: francois at busii.com (Francois Marais)
Date: Fri, 16 May 2008 16:15:58 +0200
Subject: [Za-pm] Perl programmer wanted
Message-ID: <f8a7b1960805160715s7063d7afw9743f5b1f755e0ce@mail.gmail.com>

Hi,

We are looking for an experienced Perl programmer to do data staging
development
and assist in skilling up a team of developers in Perl.

The successful candidate will have experience in:
 - SQL
 - Perl with DBI on Linux/Unix
 - data integration/batch processing IT environments
 - version control using CVS or Subversion

This is an opportunity to use your Perl and SQL expertise to gain experience
in data warehousing
in a large IT environment using leading-edge database technology.

Remuneration will be market-related.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.pm.org/pipermail/za-pm/attachments/20080516/bf64ce8b/attachment.html 

From jkeen at verizon.net  Sat May 17 09:24:02 2008
From: jkeen at verizon.net (James E Keenan)
Date: Sat, 17 May 2008 12:24:02 -0400
Subject: [Za-pm] Please help with the Perl Monger Group Census
Message-ID: <18C99714-7922-43B7-BA48-BC5249B92932@verizon.net>

za.pm has apparently not yet responded to the Perl Monger group  
census conducted biannually by Dave Cross (census at pm.org):

http://mag-sol.com/census2008/missing.cgi

Can whoever is 'in charge' take a minute to respond?  Thanks.

From aesop at fables.co.za  Fri May 30 08:38:21 2008
From: aesop at fables.co.za (Ian Balchin)
Date: Fri, 30 May 2008 17:38:21 +0200
Subject: [Za-pm] Databases & Perl
In-Reply-To: <200805131448.06214.jonathan@hst.org.za>
References: <48298BAF.929.AC9E4E@aesop.fables.co.za>,
	<200805131448.06214.jonathan@hst.org.za>
Message-ID: <48403B8D.5485.17EE666@aesop.fables.co.za>

Jonathan & Tielman, hi,

As I progress with my interim port to Paradox I note a small problem. Not 
that I might not find a way around it, but it justifies a pause.

The problem is that the Pdox normal field length max is 256 characters. 
There are 'memo' fields that will take as much as you like, but export from 
Memo fields and import into Memo fields (say from/as a delimited format) 
takes place at the 256 character level, or is even not supported - at least not 
without special code. ie you can write up several k of data but you are likely 
to get only 256 chars in or out.

If I was looking at one of the SQL databases, is this a likely scenario for 
them as well?

If you ask where does this extra data come from, then my Clue database 
has no restrictions whatseover on the field length (this was a major selling 
point 20+ years back for book dealers making catalogues as 256 chars or 
whatever is very restrictive in terms of book descriptions - titles can even be 
longer than that). Thus I have fields with perhaps a thousand characters that 
are likely to be severly truncated.

Sorry if this is a bit off topic, but it all boils down to perl in the end!

regards
Anne




On 13 May 2008 at 14:48, Jonathan McKeown wrote:

> On Tuesday 13 May 2008 12:38, Anne Wainwright wrote:
> > Hi,
> >
> > A ramble of a know nothing, for comment and suggestions.
> >
> > Bear in mind that apart from my dos database I am fairly competant with
> > Paradox. For the uninitiated, this Windows app was originally from Borland,
> > later from Corel, and provides a database with integrated design
> > environment where you can add your buttons and tables etc to a form and
> > then add code to those objects to make it go the way you want. Far more
> > powerful than Excel was. Design of forms and reports is a snap.
> [snip]
> > I see and note that Perl has many modules and features designed to enable
> > it to work with an sql type database. The questions in my mind revolve
> > around some complex applications that I have created in Paradox. Am I going
> > to have to create them with a much lower level language than I am used to?
> > Paradox's ObjectPal provides a fully integrated development environment
> > where you drag and drop and add some code to the objects already dragged,
> > sized, coloured, and dropped.
> 
> The problem you have with developing GUI applications is that producing a 
> graphical interface is complicated. It's easier if you have a development 
> environment that lets you drag and drop your components in the way that 
> Access or Paradox does.
> 
> > Further confusion because so much of what I read (well, scan) on sql
> > databases refers to their use for internet applications and how to view the
> > data through your browser.
> 
> Another way to develop an application which looks good and is easy to use with 
> a mouse is to make it a Web application. This is very popular because the 
> application now only needs a Web browser installed on the client rather than 
> the whole application, and using HTML as your output format offloads a lot of 
> the nonsense involved in managing the gory details of screen layout onto the 
> browser.
> 
> > Has Perl passed its sell-by date in this field?
> 
> Now that's almost a religious question. Perl has been around for 20 years and 
> for much of that time, it was King of the Web. It is still good at what it 
> does, and the libraries that are available from CPAN (including many web 
> tools) are still actively developed (apparently 25% of the 13,000+ modules on 
> CPAN have a most recent version upload date in the last 4 months, and half of 
> them have been updated in the last 17 months).
> 
> On the other hand, it's harder to learn than PHP (I would argue that once 
> you've learned Perl, it's easier to program well with than PHP - others would 
> disagree), and there seem to be fewer Perl programmers around, especially in 
> South Africa for some reason, to help you out.
> 
> > As you can see I am lost in the desert here. Any comments on how the ground
> > is laid out these days and which path I should take are most welcome.
> > Rather post to the list as my email client will handle these automagically
> > for me.
> 
> If I were in your shoes, I would probably take the web approach: it gives you 
> much more flexibility in the long run (can easily move from Windows to Linux 
> etc). Having decided that, I would probably look at web frameworks, such as 
> Catalyst for Perl (there are similar frameworks for other languages), which 
> make the whole process less painful (the buzzword to look out for is CRUD, 
> Create-Retrieve-Update-Delete, describing the four basic operations you need 
> on a database. Googling for CRUD database will get you some more 
> information).
> 
> Bear in mind another of Perl's strengths is the DBI database interaction 
> module. In PHP, if you write your application for MySQL and then change your 
> mind and decide to use a different database, you have to find and replace 
> every single mysql_**** function with the equivalents for the new database. 
> (This is starting to change - there is at least one database access layer 
> available now). In Perl, provided you've used DBI.pm, all you need to do is 
> change your connect statement. DBI provides a standard interface to your data 
> and automatically loads the right DBD (database-dependent) module for the 
> database you're connecting to.
> 
> Jonathan
> _______________________________________________
> Za-pm mailing list
> Za-pm at pm.org
> http://mail.pm.org/mailman/listinfo/za-pm

----
Anne Wainwright


From aesop at fables.co.za  Fri May 30 08:49:22 2008
From: aesop at fables.co.za (Ian Balchin)
Date: Fri, 30 May 2008 17:49:22 +0200
Subject: [Za-pm] Require and <>
Message-ID: <48403E22.12490.188FCDB@aesop.fables.co.za>

Hi,

Having got a number of routines to massage my data I thought that I should 
get them all together into one.

First thought, that I would sew them all up together in one giant script. Since 
I have several divergent paths and would have several of 'mega-scripts', I 
could end up with a maintenance problem if I decided to change one of the 
component scripts which I would have to do within each mega-script. Not a 
big deal, but ...

Second thought, 'call' all of them from within the mega-script. I look at the 
camel book and decide that 'require' is the way to go. Well, yes and no 
because the scripts rely on command line file input which goes to <> and it 
looks like this does not retain the input for all the scripts called one after the 
other. Any way, my simple effort generated error messages which I can't 
repeat  at this moment.

I may be wrong. However, my thought was then to save <> to a $myfiles at 
the beginning and then reset prior to calling each succeeding script. I haven't 
actually tried this yet. So, question.

Is there a more elegant way of handling this small problem?

with thanks for any suggestion
Anne--
Fables Bookshop (Proprietor: Ian Balchin) Est. 1990
119 High Street, Grahamstown, 6139, South Africa.
Tel/Fax: Shop  +27-(0)46-636-1525
Tel: Home office:  +27-(0)46-622-2474
cell: 
Skype: fablesbooks    (subject to my being at the computer 
station).
http://www.fables.co.za/
subscribe to our email africana catalogue:
http://lists.imaginet.co.za/mailman/listinfo/fables-list
for secure credit card transmission:
http://www.sabda.co.za/fables.htm
email: aesop at fables.co.za
ici on parle francais
Founder member Southern African Book Dealers Association


From tvilliers at lastminute.com  Fri May 30 12:03:02 2008
From: tvilliers at lastminute.com (Tielman de Villiers)
Date: Fri, 30 May 2008 20:03:02 +0100
Subject: [Za-pm] Databases & Perl
In-Reply-To: <48403B8D.5485.17EE666@aesop.fables.co.za>
Message-ID: <C4660DF6.1CCD%tvilliers@lastminute.com>

On 2008-05-30 16:38, "Ian Balchin" <aesop at fables.co.za> wrote:

> The problem is that the Pdox normal field length max is 256 characters.
> There are 'memo' fields that will take as much as you like, but export from
> Memo fields and import into Memo fields (say from/as a delimited format)
> takes place at the 256 character level, or is even not supported - at least
> not 
> without special code. ie you can write up several k of data but you are likely
> to get only 256 chars in or out.
> 
> If I was looking at one of the SQL databases, is this a likely scenario for
> them as well?

No. You can use VARCHAR or TEXT fields to store more than 255 characters.
See:

http://dev.mysql.com/doc/refman/5.0/en/char.html
http://www.postgresql.org/docs/8.3/static/datatype-character.html
http://www.idevelopment.info/data/Oracle/DBA_tips/Database_Administration/DB
A_25.shtml
http://mc-computing.com/Databases/FieldTypes.html


--tielman





From tvilliers at lastminute.com  Fri May 30 12:06:55 2008
From: tvilliers at lastminute.com (Tielman de Villiers)
Date: Fri, 30 May 2008 20:06:55 +0100
Subject: [Za-pm] Require and <>
In-Reply-To: <48403E22.12490.188FCDB@aesop.fables.co.za>
Message-ID: <C4660EDF.1CD3%tvilliers@lastminute.com>

On 2008-05-30 16:49, "Ian Balchin" <aesop at fables.co.za> wrote:

> Having got a number of routines to massage my data I thought that I should
> get them all together into one.
> 
> First thought, that I would sew them all up together in one giant script.
> Since 
> I have several divergent paths and would have several of 'mega-scripts', I
> could end up with a maintenance problem if I decided to change one of the
> component scripts which I would have to do within each mega-script. Not a
> big deal, but ...
> 
> Second thought, 'call' all of them from within the mega-script. I look at the
> camel book and decide that 'require' is the way to go. Well, yes and no
> because the scripts rely on command line file input which goes to <> and it
> looks like this does not retain the input for all the scripts called one after
> the 
> other. Any way, my simple effort generated error messages which I can't
> repeat  at this moment.
> 
> I may be wrong. However, my thought was then to save <> to a $myfiles at
> the beginning and then reset prior to calling each succeeding script. I
> haven't 
> actually tried this yet. So, question.
> 
> Is there a more elegant way of handling this small problem?
> 

Create modules and use "use" ?

http://perldoc.perl.org/perlmod.html#Perl-Modules

--tielman