[Chicago-talk] Script design question

Imran Javaid ijavaid at usa.net
Tue Dec 13 19:20:39 PST 2005


Suggestion 1: 
First analyze the A-Z list and see which letters have more users and possibly
also which letters have users who use more disk space. Then divide those into
4 groups so that total time to backup each of those 4 groups takes roughly the
same amount of time. Fork for each of those 4 groups.

Suggestion 2: 
Use threads. Have a control array. Create a thread for an alphabet letter and
add to the control array if the array's length is <= 4. If equal to 4, wait
for a thread to finish and remove that entry from the control array. Create a
new thread, and so forth till you have gone through the alphabet.

-imran

------ Original Message ------
Received: Tue, 13 Dec 2005 06:21:50 PM CST
From: "Young, Darren" <Darren.Young at ChicagoGSB.edu>
To: "Chicago.pm chatter" <chicago-talk at pm.org>
Subject: [Chicago-talk] Script design question

> 
> Looking for some input on a script before I sit down and try to create
> it. The basic need is to launch 4 system commands (at the same time) and
> watch each one for completion and when one is done to start another.
> Basically, it needs to keep 4 child processes running up to a defined
> end.
> 
> If that doesn't make sense, here's what it's for. In order to run
> backups on our mail system we have to use a Sun supplied binary
> (imsbackup). As an argument this binary takes what are called "groups"
> to perform a given backup. Those "groups" are defined in another file
> (backup-groups.conf) and contains lines such as:
>    groupA=a*
>    groupB=b*
> 
> And so on and so forth until the letter Z. groupA are users that start
> with the letter a, groupB are the b's, and so on. Now, since these
> backups take so long to run we're going to run 4 of them in parallel,
> each one against a different group. So, from the command line I would:
>    imsbackup -i -f- /gsbims/groupA > /export/backups/groupA.bkp &
>    imsbackup -i -f- /gsbims/groupB > /export/backups/groupB.bkp &
> 
> Do that for groups A-D and let them run. Then, the first one that
> completed would fall off the "to-do" list and the next letter group will
> be started (E in this case). Then on to F, G, etc. Testing so far says
> that an individual group can take up to 2-3 hours to complete so I have
> to be able to deal with processes that sit and run for a while and
> produce no output. Not sure what exit codes the imsbackup program gives
> back, but I'm betting it's just a 0 or 1.
> 
> Now, if one of them fails all I need to do is log the fact that it did
> and move on. Additionally, I'd like to keep track of the time it took to
> perform each group "thread" and log that as well. I'd prefer not to have
> a script call other scripts as opposed to calling the commands
> themselves, I've found that I lose tidbits here and there from
> sub-scripts. While that's a preference, it's certainly not required.
> 
> The part that I need help with is how to actually spawn each child and
> watch for their outcome (reliably). The waitpid() call seems to wait for
> one particular process to complete or am I reading the docs incorrectly?
> Am I talking about using threads in this case or am I over-engineering
> what I want done? I've done plenty of whil(<PIPE>) stuff previously but
> only on a single process.
> 
> Any thoughts on this would be very appreciated.
> 
> -------------------------------------------------------------
> | Darren Young              | http://www.chicagogsb.edu     |
> | Senior UNIX Administrator | darren.young at chicagogsb.edu   |
> | University of Chicago GSB | darren.young at gsb.uchicago.edu |
> -------------------------------------------------------------
> _______________________________________________
> Chicago-talk mailing list
> Chicago-talk at pm.org
> http://mail.pm.org/mailman/listinfo/chicago-talk
> 





More information about the Chicago-talk mailing list