[VPM] Summary of Muldis DB updates since June 21st

[Darren Duncan]

To the Victoria.pm and RCSS groups,

The only last time I made an official Muldis DB release announcement 
to Victoria.pm and RCSS was on 2007 June 21st, for the first CPAN 
release of Muldis DB for Perl 5, which was a wide announcement to a 
dozen-plus forums.  Since then, I made 16 additional CPAN releases, 
but only announced them to a select handful of lists (the 3 Muldis-DB 
lists, the TTM list, and the DBIx-Class list), while others may just 
notice the releases in the CPAN feeds.

Since it is probably imminent that I will start giving talks to 
Victoria.pm and RCSS about this (initially secondary/time-filling 
talks; RCSS first, on October 2nd, probably), I've decided to now 
send you a summary of what changed with Muldis DB since the release 
zero, so you are more up to date and can know better what questions 
to ask or whatever at the meetings (or on the list before that).

----------

1.  The language specification for Muldis D has been separated into 
its own version control and its own line of CPAN releases, under the 
package name Language::MuldisD.  The twin Perl 5 and Perl 6 
implementations of this language, Muldis DB, are maintained and 
distributed separately, the first under the package name Muldis::DB, 
and the latter is bundled with Perl6::Pugs (not yet released on CPAN).

See http://search.cpan.org/dist/Language-MuldisD/ for the language spec.

See http://search.cpan.org/dist/Muldis-DB/ for the implementation in Perl 5.

The Language-MuldisD distro started with what was initially the 
Language.pod file of release zero, plus half of SeeAlso.pod; the 
post-split Muldis-DB distro had all the other lib/ files and 
remaining half of SeeAlso.

This separation was done to encourage treatment of the above 2 things 
as distinct entities, with the emphasis placed on having the Muldis D 
langauge get adopted and implemented for database work, regardless of 
whether the implementation that ends up the most popular is the one I 
make.  Over time I hope that Muldis D will have implementations over 
many programming languages or environments.  I'm making mine (Muldis 
DB) in Perl because that's what I know and have found it very 
suitable for the task, prototyping in particular.  (As an analogy, 
SQL is its own separate language spec, with multiple implementations 
by various projects.)

----------

2.  The license for Muldis DB itself has been changed to the LGPLv3+ 
from the GPLv3+, both the whole core, plus most extensions that I 
expect to start.  All other things being equal, hopefully this change 
will encourage more update of and contributions to the project. 
Unchanged from before, any contributions I make to existing projects, 
such as DBIx-Class, to help them work with Muldis DB, will be under 
the existing license of that project being enabled.

The Muldis D language spec remains GPLv3+, but that shouldn't cause 
any issues since it is all documentation and no one would be making a 
combined work of the documentation, save to extend the spec, and 
aggregation is allowed anyway.

----------

3.  The Muldis D language spec, which was a single file 
(Language.pod) in June, has been split into multiple files, currently 
7 (MuldisD.pm, Basics.pod, Core.pod, Grammar.pod, PerlHosted.pod, 
Temporal.pod, Spatial.pod), and has been greatly fleshed out and/or 
modified.

(Basics.pod)  The DBMS entity namespace has been reorganized.

(Core.pod)  More system-defined data types have been added and/or 
made core, including the ordered role, and rational numbers.

(Core.pod)  A full complement of system-defined operators/routines 
have been added, with signatures and descriptions.  There are over a 
hundred I think.

(Basics.pod, Grammar.pod, PerlHosted.pod)  Muldis D now has 2 
distinct formats, called Concrete and Abstract.  Concrete is a 
string-form that you use if treating Muldis D like an ordinary 
self-hosted language (such as if you make a Parrot compiler for it), 
and Abstract is like an AST that you use either when doing runtime 
data-definition in Muldis D, or you are using Muldis D hosted over 
another language like Perl.

Note that the MuldisD.pm file is only a .pm (containing an empty Perl 
package) rather than a .pod in order to help the CPAN indexer treat 
the all-documentation distro somewhat properly.

----------

4.  The AST-like Perl Hosted (Abstract) flavor of Muldis D, described 
in PerlHosted.pod, is being put forward as my recommendation for a 
defacto standard Perl AST for use by database-centric Perl modules in 
general for their use in specifying database queries, 
data-manipulation, data-definition, etc.

There is already a stated demand for a better AST format to replace 
ones like SQL::Abstract's input format, that the creators of 
DBIx-Class and other object persistence or database abstraction tools 
are using to generate SQL and such behind the scenes, and sometimes 
expose to through their own APIs.  Existing things like SQL::Abstract 
were very limited when it came to wanting to represent many kinds of 
common, or less common, SQL constructs, and often users had to either 
do without or resort to cludges or raw SQL.  The desired replacement 
would do away with such problems.

I have adopted the basic principles of what the CPAN authors were 
discussing for the form of an AST, such as being composed of just 
trees of Perl array-refs, hash-refs, scalars etc, no piles of objects 
etc, and made a representation of Muldis D with these qualifications. 
Since Muldis D in general can represent any reasonable SQL construct 
elegantly, hence so can this AST based on it.

Also, because Perl Hosted Muldis D is composed purely of plain Perl 
data structures, no common Perl module is needed to compose it from, 
and no related bottleneck or possible license hangups due to 
dependency on said module.  Moreover, if this AST is a common 
documented standard, then presumably any of its users could be 
interchangeable for each other, and that reduces possible matters of 
whether works are dependent on or derived works of others, as 
legal/license matters are concerned with.

----------

5.  The Muldis DB API (Interface.pm) has been rewritten, and other 
files (DB.pm, Validator.pm, Example.pm and its support files) updated 
to conform.  The replaced interface definition is also fully 
documented.

While the original Muldis DB Interface classes created a buffer 
between applications and Engines, doing some input checking for the 
latter and some output checking for the latter, the new Muldis DB 
Interface is simply a thin role definition, which each Engine 
implements, and so applications now talk to Engines directly; the 
Interface just says that the Engine does a particular role/interface. 
The removal of the intermediary should improve performance, and also 
give the Engine more flexibility as to how it does input checking. 
Unchanged is that exceptions are always thrown to indicate errors, 
with no exceptions indicating success.

The Muldis DB API now takes Perl-Hosted Muldis D (perl array-refs 
etc) as input and returns such as output, both for representing data 
and code; it does this instead of using trees of AST.pm/Literal.pm 
objects, and AST.pm went away.  This change both greatly increases 
brevity of user Perl code, and presumably performance.

The Muldis DB API is longer based around distinct prepare and execute 
of anonymous statements like DBI does.  Rather, the API is based on 
calls to named routines, system-defined and user-defined.  So DBI's 
prepare() is replaced with invoking call_proc() on a system-defined 
data-definition procedure, giving it PHMD structures (analagous to 
SQL statements) as input, from which it makes a named procedure. 
Then DBI's execute() is replaced with invoking call_proc() again this 
time to the new user-defined routine.  Bind vars on the latter 
invocation either carry input or carry the query results.  Invoking 
database stored procedures is doable with just the latter step alone.

Muldis DB is designed essentially to make it look like you have an 
ordinary application whose components may be written in several 
languages, one happening to be good at database stuff.  For example, 
like a Perl program for which some components are written in C.  In 
this case, replace C with Muldis D.  So its API is fundamentally 
around invoking routines, sort of like they were Perl routines but 
not actually.  I anticipate that somebody's wrapper for Muldis DB 
will make it so you actually are using the syntax to invoke Perl 
routines, perhaps because it manipulated the Perl symbol table to 
make this work (Muldis DB itself doesn't).

----------

6.  There is now some example use code of Muldis DB, both in the 
SYNOPSIS docs of Interface.pm, and in the source code of 
Validator.pm.  The latter is a more realistic example, "find 
suppliers of orange-colored foods".

----------

7.  All this said, the current Muldis DB Example Engine isn't capable 
of doing any work yet, so you can't actually execute queries.  Also, 
the meta-model def in Muldis D needs more fleshing out.  Updates of 
those 2 together should result in something that executes and does 
work, hopefully in my next release, hopefully before next week.  But 
if not, I still know what the the queries look like if they were 
executing.

----------

That's about it for the summary.  Go ahead and look at the current 
docs, and each distro's Changes file details the changes, or you can 
ignore them to just look at the current result.

Feedback or questions welcome in advance of the meetings, in case 
that may help me make project improvements or give a better talk.

Thank you. -- Darren Duncan