From linux at codehelp.co.uk Thu Feb 15 07:50:14 2007 From: linux at codehelp.co.uk (Neil Williams) Date: Thu, 15 Feb 2007 15:50:14 +0000 Subject: [DCPM] shorthand required Message-ID: <20070215155014.49692f31.linux@codehelp.co.uk> http://search.cpan.org/dist/XML-QOFQSF/lib/XML/QOFQSF.pm This module is getting quite laborious. I'm wondering if there is a shorthand for handling structs of unknown composition. Current code for parsing just one struct from XML. if ($g->{type} eq 'pilot_address') { my $c = new Contact; $c->entryCity($g->{'string'}->[0]->{content}); $c->entryCustom4($g->{'string'}->[1]->{content}); $c->entryPhone1($g->{'string'}->[2]->{content}); $c->entryZip($g->{'string'}->[3]->{content}); $c->entryLastname($g->{'string'}->[4]->{content}); $c->entryPhone2($g->{'string'}->[5]->{content}); $c->entryNote($g->{'string'}->[6]->{content}); $c->category($g->{'string'}->[7]->{content}); $c->entryFirstname($g->{'string'}->[8]->{content}); $c->entryPhone3($g->{'string'}->[9]->{content}); $c->entryTitle($g->{'string'}->[10]->{content}); $c->entryPhone4($g->{'string'}->[11]->{content}); $c->entryCompany($g->{'string'}->[12]->{content}); $c->entryPhone5($g->{'string'}->[13]->{content}); $c->entryState($g->{'string'}->[14]->{content}); $c->entryCustom1($g->{'string'}->[15]->{content}); $c->entryAddress($g->{'string'}->[16]->{content}); $c->entryCustom2($g->{'string'}->[17]->{content}); $c->entryCountry($g->{'string'}->[18]->{content}); $c->entryCustom3($g->{'string'}->[19]->{content}); $c->guid($g->{'guid'}->[0]->{content}); push @contacts, $c; } to parse the XML, now: foreach my $a (@_) { $count++; $writer->startTag("object", 'type' => 'pilot_address', 'count' => "$count"); $writer->startTag('string', 'type' => 'entryCity'); $writer->characters($a->entryCity) if ($a->entryCity); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryCustom4'); $writer->characters($a->entryCustom4) if ($a->entryCustom4); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryPhone1'); $writer->characters($a->entryPhone1) if ($a->entryPhone1); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryZip'); $writer->characters($a->entryZip) if ($a->entryZip); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryLastname'); $writer->characters($a->entryLastname) if ($a->entryLastname); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryPhone2'); $writer->characters($a->entryPhone2) if ($a->entryPhone2); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryNote'); $writer->characters($a->entryNote) if ($a->entryNote); $writer->endTag('string'); $writer->startTag('string', 'type' => 'category'); $writer->characters($a->category) if ($a->category); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryFirstname'); $writer->characters($a->entryFirstname) if ($a->entryFirstname); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryPhone3'); $writer->characters($a->entryPhone3) if ($a->entryPhone3); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryTitle'); $writer->characters($a->entryPhone3) if ($a->entryTitle); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryPhone4'); $writer->characters($a->entryPhone4) if ($a->entryPhone4); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryCompany'); $writer->characters($a->entryCompany) if ($a->entryCompany); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryPhone5'); $writer->characters($a->entryPhone5) if ($a->entryPhone5); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryState'); $writer->characters($a->entryState) if ($a->entryState); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryCustom1'); $writer->characters($a->entryCustom1) if ($a->entryCustom1); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryAddress'); $writer->characters($a->entryPhone3) if ($a->entryAddress); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryCustom2'); $writer->characters($a->entryCustom2) if ($a->entryCustom2); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryCountry'); $writer->characters($a->entryCountry) if ($a->entryCountry); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryCustom3'); $writer->characters($a->entryCustom3) if ($a->entryCustom3); $writer->endTag('string'); $writer->startTag('guid', 'type' => 'guid'); $writer->characters($a->guid) if ($a->guid); $writer->endTag('guid'); $writer->endTag('object'); } to write new XML. (New routine, planned for v0.03 which is not in CPAN yet.) Now this is one of the simpler structs that contains nearly all string values. Others include booleans, integers and key-value pairs, amongst others. There are currently over a dozen structs to be supported by this module with lots more in the future. I also need to convert ALL these structs to and from other formats - like SQLite. Is there some way of iterating through: struct (Contact => { "entryCity" => '$', "entryCustom4" => '$', "entryPhone1" => '$', "entryZip" => '$', "entryLastname" => '$', "entryPhone2" => '$', "entryNote" => '$', "category" => '$', "entryFirstname" => '$', "entryPhone3" => '$', "entryTitle" => '$', "entryPhone4" => '$', "entryCompany" => '$', "entryPhone5" => '$', "entryState" => '$', "entryCustom1" => '$', "entryAddress" => '$', "entryCustom2" => '$', "entryCountry" => '$', "entryCustom3" => '$', "guid" => '$', }); (along with some form of relation between the fields in the struct and the TYPE of data contained within?) Some structs also contain references to instances of other structs. Or some way of encoding the above struct, WITH the 'type' data, so that the structs, the parser and the writer can be generated on-the-fly? This module could get very large if I can't find a way to shorten the setup. Any ideas? -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20070215/4376daf0/attachment.bin From steve at devon-it.co.uk Thu Feb 15 09:55:28 2007 From: steve at devon-it.co.uk (Steve Marvell) Date: Thu, 15 Feb 2007 17:55:28 +0000 Subject: [DCPM] shorthand required In-Reply-To: <20070215155014.49692f31.linux@codehelp.co.uk> References: <20070215155014.49692f31.linux@codehelp.co.uk> Message-ID: <20070215175528.GA29476@devon-it.co.uk> I'm not sure what the question is, since the example is so complex. Do you have a reduced version of the question/questions? Steve From linux at codehelp.co.uk Thu Feb 15 10:39:54 2007 From: linux at codehelp.co.uk (Neil Williams) Date: Thu, 15 Feb 2007 18:39:54 +0000 Subject: [DCPM] shorthand required In-Reply-To: <20070215175528.GA29476@devon-it.co.uk> References: <20070215155014.49692f31.linux@codehelp.co.uk> <20070215175528.GA29476@devon-it.co.uk> Message-ID: <20070215183954.30f4eabb.linux@codehelp.co.uk> On Thu, 15 Feb 2007 17:55:28 +0000 Steve Marvell wrote: > I'm not sure what the question is, since the example is so complex. Do > you have a reduced version of the question/questions? > > Steve Take a single parameter struct: if ($g->{type} eq 'pilot_address') { my $c = new Contact; $c->entryCity($g->{'string'}->[0]->{content}); } foreach my $a (@contacts) { $writer->startTag('string', 'type' => 'entryCity'); $writer->characters($a->entryCity) if ($a->entryCity); $writer->endTag('string'); } struct (Contact => { "entryCity" => '$', } The struct is fixed (it's based in C code). What I want to do is compress this code because each element is intrinsically predictable from the struct itself - provided certain meta data is preserved. I have at least 12 objects, each with 10-20 parameters. So far, that has led to a module exceeding 1,000 lines and I haven't added the SQLite code yet. e.g. is there any way of setting "entryCity" just once? Can a hash or array be used to specify "entryCity" everywhere except the struct? That would mean using a variable to declare a method. Instead of $a->entryCity, some form of $a->$b. I want to stop having to repeat each parameter handler: $writer->startTag('string', 'type' => 'entryCity'); $writer->characters($a->entryCity) if ($a->entryCity); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryCustom4'); $writer->characters($a->entryCustom4) if ($a->entryCustom4); $writer->endTag('string'); $writer->startTag('string', 'type' => 'entryPhone1'); $writer->characters($a->entryPhone1) if ($a->entryPhone1); $writer->endTag('string'); and have some form of: foreach $foo (keys %bar) { $writer->startTag($bar->type, 'type' => $foo) $writer->characters($foo) if ($foo); $writer->endTag($bar->type); } then a similar loop for setting the value in the struct: $c->$foo($g->{$bar->type}->[0]->{content}); That would reduce the module size by maybe 75%. The full code, including a test routine, is available in CPAN. -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20070215/e0a7f0af/attachment.bin From steve at devon-it.co.uk Thu Feb 15 10:48:30 2007 From: steve at devon-it.co.uk (Steve Marvell) Date: Thu, 15 Feb 2007 18:48:30 +0000 Subject: [DCPM] shorthand required In-Reply-To: <20070215183954.30f4eabb.linux@codehelp.co.uk> References: <20070215155014.49692f31.linux@codehelp.co.uk> <20070215175528.GA29476@devon-it.co.uk> <20070215183954.30f4eabb.linux@codehelp.co.uk> Message-ID: <20070215184830.GA31701@devon-it.co.uk> Neil Williams wrote: > if ($g->{type} eq 'pilot_address') > { > my $c = new Contact; > $c->entryCity($g->{'string'}->[0]->{content}); > } Where does the entryCity subroutine come from? Is there a good reason you're using a subroutine over simply setting it directly? Steve From linux at codehelp.co.uk Thu Feb 15 11:17:47 2007 From: linux at codehelp.co.uk (Neil Williams) Date: Thu, 15 Feb 2007 19:17:47 +0000 Subject: [DCPM] shorthand required In-Reply-To: <20070215184830.GA31701@devon-it.co.uk> References: <20070215155014.49692f31.linux@codehelp.co.uk> <20070215175528.GA29476@devon-it.co.uk> <20070215183954.30f4eabb.linux@codehelp.co.uk> <20070215184830.GA31701@devon-it.co.uk> Message-ID: <20070215191747.7a40ff79.linux@codehelp.co.uk> On Thu, 15 Feb 2007 18:48:30 +0000 Steve Marvell wrote: > Neil Williams wrote: > > > if ($g->{type} eq 'pilot_address') > > { > > my $c = new Contact; > > $c->entryCity($g->{'string'}->[0]->{content}); > > } > > Where does the entryCity subroutine come from? >From the original C source code of the C object - in this case, used by pilot-qof. The perl scripts are not handling their own data, this is data from third-party sources (generally C applications) that perl is converting into the formats required by other third-party sources. Neither the import format nor the export format are designed with perl in mind. Neither format can be modified. The old middleware problem. > Is there a good reason you're using a subroutine over simply setting > it directly? That would still need the variable "entryCity" to be translated into a variable name. What's the syntax for setting a variable directly when the variable name is itself a variable? -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20070215/290d8d26/attachment.bin From aaron.trevena at gmail.com Fri Feb 16 04:55:16 2007 From: aaron.trevena at gmail.com (Aaron Trevena) Date: Fri, 16 Feb 2007 12:55:16 +0000 Subject: [DCPM] shorthand required In-Reply-To: <20070215191747.7a40ff79.linux@codehelp.co.uk> References: <20070215155014.49692f31.linux@codehelp.co.uk> <20070215175528.GA29476@devon-it.co.uk> <20070215183954.30f4eabb.linux@codehelp.co.uk> <20070215184830.GA31701@devon-it.co.uk> <20070215191747.7a40ff79.linux@codehelp.co.uk> Message-ID: Hi Neil, Your XML creation seems incredibly verbose. I would suggest two things : 1) try using XML::DOM::BagOfTricks (http://search.cpan.org/~teejay/XML-DOM-BagOfTricks-0.05/lib/XML/DOM/BagOfTricks.pm) which allows you to do create complex XML tags in a single statement. 2) Create a dictionary holding source and destination fieldnames and loop through that - a simple array of hashes (to preserve any necessary ordering) should be fairly simple then you could do stuff like: $destination_object->$destination_fieldname($source_object->$source_fieldname) if ($source_object->$source_fieldname); Combining these two things should make your code a lot shorter, and therefore easier to document. cheers, A. -- http://www.aarontrevena.co.uk LAMP System Integration, Development and Hosting From linux at codehelp.co.uk Sat Feb 17 13:39:59 2007 From: linux at codehelp.co.uk (Neil Williams) Date: Sat, 17 Feb 2007 21:39:59 +0000 Subject: [DCPM] shorthand required In-Reply-To: References: <20070215155014.49692f31.linux@codehelp.co.uk> <20070215175528.GA29476@devon-it.co.uk> <20070215183954.30f4eabb.linux@codehelp.co.uk> <20070215184830.GA31701@devon-it.co.uk> <20070215191747.7a40ff79.linux@codehelp.co.uk> Message-ID: <20070217213959.b595aee7.linux@codehelp.co.uk> On Fri, 16 Feb 2007 12:55:16 +0000 "Aaron Trevena" wrote: > Hi Neil, > > Your XML creation seems incredibly verbose. I know. It wouldn't matter if I could collapse it into 3 lines for all parameters of all objects, instead of 3 lines per parameter, per object. Note that this isn't just XML creation, it is XML writing, SQLite queries and other, as yet unwritten, methods too. > I would suggest two things : > 1) try using XML::DOM::BagOfTricks > (http://search.cpan.org/~teejay/XML-DOM-BagOfTricks-0.05/lib/XML/DOM/BagOfTricks.pm) > which allows you to do create complex XML tags in a single statement. Just now, I'd rather stick with XML::Simple and XML::Writer. > 2) Create a dictionary holding source and destination fieldnames and > loop through that - a simple array of hashes (to preserve any > necessary ordering) should be fairly simple then you could do stuff > like: > > $destination_object->$destination_fieldname($source_object->$source_fieldname) > if ($source_object->$source_fieldname); I'm not sure if I follow you here. There is no 'destination' object and I don't see how to convert a struct into this dictionary. Take a small struct: struct (ToDo => { "todo_note" => '$', #string (perl: string) "todo_description" => '$', #string (perl: string) "category" => '$', #string (perl: string) "guid" => '$', #guid (perl: string of hexadecimal characters only) "date_due" => '$', #time (perl: string passed to time2str) "todo_priority" => '$', #gint32 (perl: integer) "todo_complete" => '$', #gint32 (perl: integer) "todo_length" => '$', #gint32 (perl: integer) }); I get as far as: my @todos = ('todo_note' => 'string', 'todo_description' => 'string', 'category' => 'string', 'guid' => 'guid', 'date_due' => 'time', 'todo_priority' => 'gint32', 'todo_complete' => 'gint32', 'todo_length' => 'gint32'); But I'm not at all sure that I follow how to get to: > $destination_object->$destination_fieldname($source_object->$source_fieldname) > if ($source_object->$source_fieldname); How does that cope with objects that have different numbers of different types of parameters, all with different names? How is the 'dictionary' created and populated? I've already got a hash of array references where the arrays contain instances of the structs containing the data. -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20070217/1d323c5f/attachment.bin From linux at codehelp.co.uk Sat Feb 17 13:56:36 2007 From: linux at codehelp.co.uk (Neil Williams) Date: Sat, 17 Feb 2007 21:56:36 +0000 Subject: [DCPM] shorthand required In-Reply-To: <20070217213959.b595aee7.linux@codehelp.co.uk> References: <20070215155014.49692f31.linux@codehelp.co.uk> <20070215175528.GA29476@devon-it.co.uk> <20070215183954.30f4eabb.linux@codehelp.co.uk> <20070215184830.GA31701@devon-it.co.uk> <20070215191747.7a40ff79.linux@codehelp.co.uk> <20070217213959.b595aee7.linux@codehelp.co.uk> Message-ID: <20070217215636.3fa6ccca.linux@codehelp.co.uk> On Sat, 17 Feb 2007 21:39:59 +0000 Neil Williams wrote: > > $destination_object->$destination_fieldname($source_object->$source_fieldname) > > if ($source_object->$source_fieldname); > > I'm not sure if I follow you here. There is no 'destination' object and > I don't see how to convert a struct into this dictionary. > > How does that cope with objects that have different numbers of > different types of parameters, all with different names? How is > the 'dictionary' created and populated? Another problem: Dictionary keys are not guaranteed unique across all objects *unless* the key is actually a combination of the object name AND the parameter name. This cannot be changed - the objects already exist in third-party code with which this module needs to be compliant. -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20070217/b41152e5/attachment.bin From linux at codehelp.co.uk Sat Feb 17 14:37:15 2007 From: linux at codehelp.co.uk (Neil Williams) Date: Sat, 17 Feb 2007 22:37:15 +0000 Subject: [DCPM] shorthand required In-Reply-To: <20070217215636.3fa6ccca.linux@codehelp.co.uk> References: <20070215155014.49692f31.linux@codehelp.co.uk> <20070215175528.GA29476@devon-it.co.uk> <20070215183954.30f4eabb.linux@codehelp.co.uk> <20070215184830.GA31701@devon-it.co.uk> <20070215191747.7a40ff79.linux@codehelp.co.uk> <20070217213959.b595aee7.linux@codehelp.co.uk> <20070217215636.3fa6ccca.linux@codehelp.co.uk> Message-ID: <20070217223715.5ff74955.linux@codehelp.co.uk> On Sat, 17 Feb 2007 21:56:36 +0000 Neil Williams wrote: Actually, I think I've got most of it - following the model of the upstream C source code which separates the DATA from the OBJECT: my %objects = (); my %todos = ('todo_note' => 'string', 'todo_description' => 'string', 'category' => 'string', 'guid' => 'guid', 'date_due' => 'time', 'todo_priority' => 'gint32', 'todo_complete' => 'gint32', 'todo_length' => 'gint32'); $objects{'pilot_todo'} = \%todos; $field = 'pilot_todo'; $param = 'date_due'; print $objects{$field}->{$param}; (this prints 'time'). Therefore, I can have a struct for the instances that contain the user data and the definition for the meta-data, one def per backend. I can use the above to create the XML snippet: and also call something like: $t->date_due as $foo->$param to retrieve the data to insert into the