From aaron.trevena at gmail.com Tue Dec 5 08:39:40 2006 From: aaron.trevena at gmail.com (Aaron Trevena) Date: Tue, 5 Dec 2006 16:39:40 +0000 Subject: [DCPM] Technical Meeting after new year? Message-ID: Hi All, I was wondering if anybody else would be interested in coming to and/or talking at a Technical Talks meeting after the new year. I'm pre-occupied sorting out the nursery, and working 7 days a week on freelance work until mid-january, and the baby is due in march, so I probably won't be doing much (or any) open source stuff for a while after that. I was hoping to organise something before now, but moving house has put paid to that. cheers, A. -- http://www.aarontrevena.co.uk LAMP System Integration, Development and Hosting From matthewbrowning at gmail.com Tue Dec 5 09:47:26 2006 From: matthewbrowning at gmail.com (Matthew Browning) Date: Tue, 5 Dec 2006 17:47:26 +0000 Subject: [DCPM] Technical Meeting after new year? In-Reply-To: References: Message-ID: <2bfa7e30612050947u5e9efec5n53c25c9049961b25@mail.gmail.com> On 12/5/06, Aaron Trevena wrote: > > I was wondering if anybody else would be interested in coming to > and/or talking at a Technical Talks meeting after the new year. > I'll go if it's in a town with a train station and I know about it in advance. I'll talk if anyone wants me to. I've done a couple of talks here recently (with slides!) so I can just recycle one of them. MB From aaron.trevena at gmail.com Wed Dec 6 06:28:11 2006 From: aaron.trevena at gmail.com (Aaron Trevena) Date: Wed, 6 Dec 2006 14:28:11 +0000 Subject: [DCPM] Technical Meeting after new year? In-Reply-To: <2bfa7e30612050947u5e9efec5n53c25c9049961b25@mail.gmail.com> References: <2bfa7e30612050947u5e9efec5n53c25c9049961b25@mail.gmail.com> Message-ID: On 05/12/06, Matthew Browning wrote: > On 12/5/06, Aaron Trevena wrote: > > > > I was wondering if anybody else would be interested in coming to > > and/or talking at a Technical Talks meeting after the new year. > > > > I'll go if it's in a town with a train station and I know about it in advance. Plymouth seems most likely - the waterfront made a good venue last time I think, although scheduling it when samba classes aren't happening in the next bit of the bar might be wise. > I'll talk if anyone wants me to. I've done a couple of talks here > recently (with slides!) so I can just recycle one of them. Cool, any interesting subjects? I had some fun with HOP::Lexer, picking up where you left off parsing aviation data, found it can be a bit less than helpful when you have a series of identical strings that can only told apart by position, context and 'knowing it when I see it'. Cheers, A. -- http://www.aarontrevena.co.uk LAMP System Integration, Development and Hosting From matthewbrowning at gmail.com Wed Dec 6 07:39:35 2006 From: matthewbrowning at gmail.com (Matthew Browning) Date: Wed, 6 Dec 2006 15:39:35 +0000 Subject: [DCPM] Technical Meeting after new year? In-Reply-To: References: <2bfa7e30612050947u5e9efec5n53c25c9049961b25@mail.gmail.com> Message-ID: <2bfa7e30612060739n7456db22uc61c9f30151d5cd1@mail.gmail.com> On 12/6/06, Aaron Trevena wrote: > Cool, any interesting subjects? > 'Template::Toolkit is better than HTML::Template' (as if it needs saying) 'Declarative Interfaces are teh Nice Interfaces' 'How to do Testing' > I had some fun with HOP::Lexer, picking up where you left off parsing > aviation data, found it can be a bit less than helpful when you have a > series of identical strings that can only told apart by position, > context and 'knowing it when I see it'. > I know what you mean. It's still a tidy module and I like the way it makes it trivial to whack business logic into callbacks. Some problemos are always going to be tricky. I've used it again to successfully (where others have failed) produce yummy summaries out of DB log files with half a million statements in them. Easy-peasy and you can pass it on for maintenance without it coming back. From linux at codehelp.co.uk Fri Dec 15 14:17:12 2006 From: linux at codehelp.co.uk (Neil Williams) Date: Fri, 15 Dec 2006 22:17:12 +0000 Subject: [DCPM] hierarchies Message-ID: <20061215221712.735bf68d.linux@codehelp.co.uk> We're all familiar with dependencies outside perl, I need to work out a new way of representing them inside perl - using a hierarchy or tree. The basic question is: Given an existing set of packages A, which packages need to be built *FIRST* in order to eventually build a system to support package set B. The first packages to built will be the ones with no missing dependencies on a system that *only* has packages from set A installed, the next series to build will be those that would have no missing dependencies if the first set is built and installed correctly. At each stage, the dependencies of the built packages may change because the work involves trimming these dependency trees to the bare minimum. The tree needs to go all the way from glibc right up to Gnome with as many levels as may be necessary. Any ideas? (or should I do this in C?) Is something similar already available? -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061215/7bbb21ad/attachment.bin From simon at technocool.net Fri Dec 15 15:06:34 2006 From: simon at technocool.net (Simon Waters) Date: Fri, 15 Dec 2006 23:06:34 +0000 Subject: [DCPM] hierarchies In-Reply-To: <20061215221712.735bf68d.linux@codehelp.co.uk> References: <20061215221712.735bf68d.linux@codehelp.co.uk> Message-ID: <45832A7A.8010603@technocool.net> Neil Williams wrote: > > Given an existing set of packages A, which packages need to be built > *FIRST* in order to eventually build a system to support package set B. Can you define what problem you are trying to solve, rather than what you think you need to do to solve it. > Any ideas? (or should I do this in C?) Perl surely ;) > Is something similar already available? CPAN already does some of this. The guy who wrote some of the CPAN tools, also produced tools for creating Debian packages from CPAN, the archives of the Debian Perl group discuss a lot of the issues of packaging Perl modules for Debian, and as a side effect some work was done on dependencies. There are at least two major sets of tools for doing the CPAN -> Deb transition.... As regards external dependencies, I don't think CPAN, or the modules, contain enough explicit information to say what is needed to reliably build a module that requires to compile C code. Although I dare say a lot of cases can be automated. Similarly some the CPAN dependencies are optional, and CPAN (or a descendant) will prompt which way you want to do something. So CPAN dependencies can get pretty complex. There are some tools to build Perl executables, which will pull all the Perl modules to achieve some task. But then Perl being Perl there are also on demand tools, that get those modules when they are needed. Dare I say there is more than one way to do it! A scary amount of new Perl modules appear in Etch. We are doing Catalyst work at ZyNet, and these days "apt-get install libcatalyst-perl" gets you a working system. Although I wouldn't like to comment how up to date it is, we have too much CPAN stuff, and I don't think we have the version management as strict as we should. But then most of our Perl code before the Catalyst app had only shallow dependencies, probably a tie to a Postgres database is the deepest we've gone previously. Indeed quite a lot of our Perl code pre-empted what is now in CPAN, as it was written be people who long predated me, at least one of whom was both brighter and better educated that I. So module version management is likely to become a big issue for me as system administrator. Especially with tools like "Version", which make the task potentially harder (well harder for the system admin, if easier for the programmer). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 288 bytes Desc: OpenPGP digital signature Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061215/296a436c/attachment.bin From linux at codehelp.co.uk Fri Dec 15 15:40:07 2006 From: linux at codehelp.co.uk (Neil Williams) Date: Fri, 15 Dec 2006 23:40:07 +0000 Subject: [DCPM] hierarchies In-Reply-To: <45832A7A.8010603@technocool.net> References: <20061215221712.735bf68d.linux@codehelp.co.uk> <45832A7A.8010603@technocool.net> Message-ID: <20061215234007.c7739bda.linux@codehelp.co.uk> On Fri, 15 Dec 2006 23:06:34 +0000 Simon Waters wrote: > Neil Williams wrote: > > > > Given an existing set of packages A, which packages need to be built > > *FIRST* in order to eventually build a system to support package set B. > > Can you define what problem you are trying to solve, rather than what > you think you need to do to solve it. A 'system' (which may be not much more than an ISO image or chroot jail) has a (small) set of packages A and needs a (large) set of packages B. Packages are available elsewhere to go from A to B but these are incompatible with A so each package must be rebuilt using only packages already available in set A. Once successfully built, new packages get included into set A so that you incrementally move closer to a full set B. By following the known dependencies of packages within B, identify the sequence that can build all packages in set B starting from set A. i.e. this is Build-Depends, rather than the binary depends. The initial set of packages, A, includes a compiler for that system and nothing else - not even a kernel - because it is a cross-build. The script needs to lay out a path from a bare compiler system to a full Gnome environment without requiring any packages that are either not already installed or cannot be successfully built using the packages already installed. So start with packages that depend only on libc6 and tzdata, then move on to packages that need these packages but nothing else, etc. Additionally, building any package will involve modifying the packages such as to remove unnecessary or unwanted modules thereby modifying the dependency tree on-the-fly. The target is to install the absolute minimum number of packages and/or substitute large packages for smaller ones such as to require the smallest total amount of installation space on the target system whilst still including as much as possible of set B. -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061215/904bda65/attachment.bin From linux at codehelp.co.uk Fri Dec 15 15:54:59 2006 From: linux at codehelp.co.uk (Neil Williams) Date: Fri, 15 Dec 2006 23:54:59 +0000 Subject: [DCPM] hierarchies In-Reply-To: <45832A7A.8010603@technocool.net> References: <20061215221712.735bf68d.linux@codehelp.co.uk> <45832A7A.8010603@technocool.net> Message-ID: <20061215235459.7b053251.linux@codehelp.co.uk> On Fri, 15 Dec 2006 23:06:34 +0000 Simon Waters wrote: > Neil Williams wrote: > > > > Given an existing set of packages A, which packages need to be built > > *FIRST* in order to eventually build a system to support package set B. > > Can you define what problem you are trying to solve, rather than what > you think you need to do to solve it. Put another way: package foo exists, package bar needs to be built. bar depends on foo and baz. baz depends only on foo. It's obvious that baz must be built first, then bar. Trying to build bar first will fail due to the lack of baz. How to calculate this on a larger scale? -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061215/2602e2b3/attachment.bin From simon at technocool.net Fri Dec 15 17:10:39 2006 From: simon at technocool.net (Simon Waters) Date: Sat, 16 Dec 2006 01:10:39 +0000 Subject: [DCPM] hierarchies In-Reply-To: <20061215234007.c7739bda.linux@codehelp.co.uk> References: <20061215221712.735bf68d.linux@codehelp.co.uk> <45832A7A.8010603@technocool.net> <20061215234007.c7739bda.linux@codehelp.co.uk> Message-ID: <4583478F.3070002@technocool.net> Neil Williams wrote: > >> Can you define what problem you are trying to solve, rather than >> what you think you need to do to solve it. > > A 'system' (which may be not much more than an ISO image or chroot > jail) has a (small) set of packages A and needs a (large) set of > packages B. Packages are available elsewhere to go from A to B but > these are incompatible with A so each package must be rebuilt using > only packages already available in set A. Once successfully built, > new packages get included into set A so that you incrementally move > closer to a full set B. By following the known dependencies of > packages within B, identify the sequence that can build all packages > in set B starting from set A. i.e. this is Build-Depends, rather than > the binary depends. If the problem is "how to build B" given A, you don't need to solve this elaborate problem. Grow the dependency tree from B till it is complete, which is a trivial problem to handle with recursion. Terminate the recursion when you hit a package which is in A, or one you already included. Now loop over the list building any unbuilt package in the list that already has its dependencies. Loop over the list until everything is built, or until an iteration over the loop doesn't improve anything (at which point you know the problem has no solution). Presumably you can do this second step "virtually" to check a solution exists, before doing anything expensive. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 288 bytes Desc: OpenPGP digital signature Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061216/397dfa84/attachment.bin From linux at codehelp.co.uk Sat Dec 16 03:21:04 2006 From: linux at codehelp.co.uk (Neil Williams) Date: Sat, 16 Dec 2006 11:21:04 +0000 Subject: [DCPM] hierarchies In-Reply-To: <4583478F.3070002@technocool.net> References: <20061215221712.735bf68d.linux@codehelp.co.uk> <45832A7A.8010603@technocool.net> <20061215234007.c7739bda.linux@codehelp.co.uk> <4583478F.3070002@technocool.net> Message-ID: <20061216112104.c6fbe6ae.linux@codehelp.co.uk> On Sat, 16 Dec 2006 01:10:39 +0000 Simon Waters wrote: > Neil Williams wrote: > > > >> Can you define what problem you are trying to solve, rather than > >> what you think you need to do to solve it. > > > > A 'system' (which may be not much more than an ISO image or chroot > > jail) has a (small) set of packages A and needs a (large) set of > > packages B. Packages are available elsewhere to go from A to B but > > these are incompatible with A so each package must be rebuilt using > > only packages already available in set A. Once successfully built, > > new packages get included into set A so that you incrementally move > > closer to a full set B. By following the known dependencies of > > packages within B, identify the sequence that can build all packages > > in set B starting from set A. i.e. this is Build-Depends, rather than > > the binary depends. > > If the problem is "how to build B" given A, you don't need to solve this > elaborate problem. Unfortunately, I will, because B is a moving target. :-) The process of building B from A changes the dependency tree and thereby changes the process itself. This doesn't normally happen in Debian but the process of cross-building causes changes in the dependencies of each package. Extra packages can be generated, stages omitted and whole branches bypassed. e.g. sqlite pre-depends on debconf in Debian but debconf depends on perl and perl will not actually be available in set A (emdebian does not class perl as essential due to it's size). So Emdebian may predefine what debconf would configure and omit debconf from the dependencies of sqlite, allowing sqlite to be built at a different stage of the process. sqlite doesn't need perl itself, it is only used by debconf to configure sqlite. Therefore, the output of the script needs to say: "These packages have no missing dependencies or no dependencies at all: ...." (Build these). "These packages have no missing dependencies except dependencies that are unsupported: ...." (Patch these or drop the package and patch packages that depend on it). "These packages have no missing dependencies except dependencies that cross-building should make unnecessary: ...." (Implement alternatives to replace these, e.g. predefine what debconf would configure.) It may prove to be impossible to predict whether some of the initial build list are eventually unnecessary, that's a price that might just have to be paid. It just means emdebian carrying a few packages that never get installed. They should be easy enough to identify afterwards. > Grow the dependency tree from B till it is complete, which is a trivial > problem to handle with recursion. It's trivial to get a *list* of total dependencies. It isn't trivial (to me) to create a dependency tree, it is the hierarchy that is difficult. What structures can implement a usable dependency tree, rather than just a list? Package | ---- dependency1 | | | ---------- dependency1.1 | ----- dependency2 I'd find this a lot easier in C if only the original data was as easily accessible as it is through Perl. :-( The main problem is that the depth of the tree cannot be predetermined. Using references to represent each dependency just gets incredibly messy when trying to locate dependency12.4.2.7. Now try changing dependency8.3.1.8.4.7 such that it no longer needs dependency13.9.2.6 ! The most useful effects all happen at the bottom end of the tree, from the perspective of starting the tree with package set B. I've got code that can compare the dependencies of a package against the available packages - it includes handling for versioned-depends and cross-building artifacts - but it cannot provide a *path* to build the missing dependencies and their dependencies. It just says "foo missing" when what it needs to say is "baz, bar, foo missing" in build order. > Terminate the recursion when you hit a > package which is in A, or one you already included. If B is "Gnome" or "Gtk", I can't stop recursion at that point, I need to go back to a different branch. Following one line may only be two levels but each dependency of B needs to be tracked all the way back to A. Recursion stops when the dependency itself has no dependencies (in the case of A, it generally ends at tzdata). > Now loop over the list building any unbuilt package in the list that > already has its dependencies. Building each package changes the dependency tree because certain components can be omitted e.g. language bindings, to save size. > Loop over the list until everything is > built, or until an iteration over the loop doesn't improve anything (at > which point you know the problem has no solution). It is the storage of the "list" that is troubling me - the ability to remove or add dependencies and then recalculate the tree. e.g. to replace debconf with dash (which will simply feed the emdebian defaults to the package instead of allowing configuration) and work out that the package has no further unmet dependencies. (dash replaces bash). > Presumably you can do this second step "virtually" to check a solution > exists, before doing anything expensive. Unreliable - this may be necessary but with the proviso that the tree is regenerated before builds of packages from the next level are attempted. -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061216/2b6a5ed6/attachment-0001.bin From linux at codehelp.co.uk Wed Dec 27 03:13:10 2006 From: linux at codehelp.co.uk (Neil Williams) Date: Wed, 27 Dec 2006 11:13:10 +0000 Subject: [DCPM] libapt-pkg-perl Message-ID: <20061227111310.f228d150.linux@codehelp.co.uk> More inadequately documented perl: http://fly.isti.cnr.it/doc/libapt-pkg-perl/examples/apt-cache http://www.penguin-soft.com/penguin/man/3/AptPkg::Cache.html How do I retrieve the "LongDesc" string from that lot? I can't even tell where to start. -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061227/64b05c16/attachment.bin From simon at technocool.net Wed Dec 27 07:58:14 2006 From: simon at technocool.net (Simon Waters) Date: Wed, 27 Dec 2006 15:58:14 +0000 Subject: [DCPM] libapt-pkg-perl In-Reply-To: <20061227111310.f228d150.linux@codehelp.co.uk> References: <20061227111310.f228d150.linux@codehelp.co.uk> Message-ID: <45929816.30303@technocool.net> Neil Williams wrote: > More inadequately documented perl: > > http://fly.isti.cnr.it/doc/libapt-pkg-perl/examples/apt-cache > > http://www.penguin-soft.com/penguin/man/3/AptPkg::Cache.html > > How do I retrieve the "LongDesc" string from that lot? I can't even > tell where to start. I gave up the second time poking AptPkg::Cache caused my system to hang, and become unresponsive to anything but "ping". Weird, this sort of thing just shouldn't happen running scripts as an ordinary user. I'm sure Data::Dumper is doing something horribly recursive, or snaffling memory too quickly. Never seen anything like that running Perl before? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 288 bytes Desc: OpenPGP digital signature Url : http://mail.pm.org/pipermail/devoncornwall-pm/attachments/20061227/892b0ab2/attachment.bin From aaron.trevena at gmail.com Sun Dec 31 13:23:23 2006 From: aaron.trevena at gmail.com (Aaron Trevena) Date: Sun, 31 Dec 2006 21:23:23 +0000 Subject: [DCPM] happy new year Message-ID: have a cool yule, hogmanny, etc. can't be arsed with it myself, so I'm off to bed with a good book. A. -- http://www.aarontrevena.co.uk LAMP System Integration, Development and Hosting