[pm-h] Workflow scheduler solution

todd.e.rinaldo at jpmorgan.com todd.e.rinaldo at jpmorgan.com
Fri Apr 11 09:42:07 PDT 2008


All,

As promised from this month's meeting, I am providing the solution to my problem we.

Background: I have a need to run jobs, but some of the jobs have a need to depend on multiple things completing. At present, we have to run them in a linear fashion or leverage ugly sleep commands to assure (hopefully) everything has run before the next command can be run.

Issue: Provide a SQL approach to create a workflow process to deal with multiple parent dependencies. This way you can just query the DB to determine which jobs that have not run yet have met their dependencies and are ready to run. 


SQL to create tables: (This is all sybase by the way. Apologies!)

CREATE TABLE dbo.Scheduler
(
    workflow_id   numeric(20,0) NOT NULL,
    workflow_step smallint      NOT NULL,
    start_time    datetime      NULL,
    end_time      datetime      NULL,
    hostname      varchar(4096) NOT NULL,
    command       varchar(4096) NOT NULL,
    status        smallint      DEFAULT -1 NOT NULL,
    pid           int           DEFAULT 0 NOT NULL,
    log           image         DEFAULT "empty" NULL,
    CONSTRAINT Scheduler_workflow_step
    UNIQUE NONCLUSTERED (workflow_id,workflow_step)
)
LOCK ALLPAGES
go
IF OBJECT_ID('dbo.Scheduler') IS NOT NULL
    PRINT '<<< CREATED TABLE dbo.Scheduler >>>'
ELSE
    PRINT '<<< FAILED CREATING TABLE dbo.Scheduler >>>'
go

CREATE TABLE dbo.SchedulerDepends
(
    workflow_id     numeric(20,0) NOT NULL,
    workflow_step   smallint      NOT NULL,
    depends_on      smallint      NOT NULL,
    status_required smallint      NOT NULL,
    CONSTRAINT SchedulerD_10261517202
    UNIQUE NONCLUSTERED (workflow_id,workflow_step,depends_on)
)
LOCK ALLPAGES
go
IF OBJECT_ID('dbo.SchedulerDepends') IS NOT NULL
    PRINT '<<< CREATED TABLE dbo.SchedulerDepends >>>'
ELSE
    PRINT '<<< FAILED CREATING TABLE dbo.SchedulerDepends >>>'
go


------


-- SQL to find out what's ready to run.
select s1.hostname, s1.command
    from dbo.Scheduler s1
    where s1.start_time is null -- figure out what hasn't started yet
    and not exists( -- figure out what dependencies s1.status has. no rows means it's ready to go.
        select 1
            from dbo.Scheduler s2,  -- used to check the state of jobs s1 is depending on.
                 dbo.SchedulerDepends d -- used to determine which jobs s1 depends on
            where s2.status   != d.status_required -- Is the job in the right state that's required?
              and s2.workflow_step  = d.depends_on  -- find all 
              and d.workflow_step   = s1.workflow_step -- Find all the dependencies for s1 row.
              and s1.workflow_id = s2.workflow_id -- Assure everything's using the same workflow
              and s1.workflow_id = d.workflow_id -- Assure everything's using the same workflow
    )


-- SQL to find out what is currently running.
select * from dbo.Scheduler s1
    where start_time is not null
    and end_time is null




Generally, this communication is for informational purposes only
and it is not intended as an offer or solicitation for the purchase
or sale of any financial instrument or as an official confirmation
of any transaction. In the event you are receiving the offering
materials attached below related to your interest in hedge funds or
private equity, this communication may be intended as an offer or
solicitation for the purchase or sale of such fund(s).  All market
prices, data and other information are not warranted as to
completeness or accuracy and are subject to change without notice.
Any comments or statements made herein do not necessarily reflect
those of JPMorgan Chase & Co., its subsidiaries and affiliates.

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by JPMorgan Chase &
Co., its subsidiaries and affiliates, as applicable, for any loss
or damage arising in any way from its use. If you received this
transmission in error, please immediately contact the sender and
destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.
Please refer to http://www.jpmorgan.com/pages/disclosures for
disclosures relating to UK legal entities.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.pm.org/mailman/private/houston/attachments/20080411/608f962b/attachment-0001.html 


More information about the Houston mailing list