[Pdx-pm] corruption puzzlement - followup

[Michael Rasmussen]

Kyle,

Thank you very much for the information on race conditions,
SCP issues and directory entry creation latency.  

I'd asked my question as a post mortem for the problem we 
observed on Tuesday.  The script was originally created so
that users could do a Samba mount of the directory and pull
the file when they received an email stating it was available.

Now we've moved into a scenario where I move the files to
other servers for the users and other processes to pick up.
As a result I've killed the process that monitored the directory
and moved the EOL conversion into the script that moves the files
to the NT severs.

code flow is now:
opendir(DIR, $incoming_dir);
@files = grep { /spec of interest/ } readdir(DIR);
foreach $file (@files) {
	# non portable test for open file condition, you're warned
	@openfiles = `/usr/sbin/lsof +d $incoming_dir 2>/dev/null`;
	if ( ! ( grep (/$file/) @openfiles ) ) {
		 proceed with munging and copying and backing up and all;
	} else {
		twiddle bits for a bit and try again
	}
}

Since the files are dropped into $incoming_dir by the tranfer process by 
the business partner and aren't touched by anything/one but my process
I'm confident (let me know if I'm wrong here) that If I catch it on the 
readdir and if it's not open when I test for it then it has arrived on
site completely and I'm free to do my stuff.   The only race condition I 
see is that the file may actually close between the time I test for it 
being open and when I want to act on it, but that's OK since I'll return
later.

Lesson:  Don't throw together a quickie while users are in testing mode
and leave it there.  




On Thu, Aug 07, 2003 at 05:08:40PM -0700, Kyle Hayes wrote:
> Change the the program so that you wait until the file is at least 60 seconds 
> old (if the longest file takes 40 seconds, give yourself some fudge factor).  

Then on a day of network congestion slowing things down or when the market grows
crazy and the file size doubles I'm back where I started.  Hence the defer action
as long as possible approach I eventually took.

> Also note that if you can stat the file, it is probably there, so the -f may 
> be redundant.  

That was polite. :) Yes, if I can stat it, it will pass the existance test.

> I generally process files into different directories.  The raw files land in 
> one directory, and I move them to another directory after processing.  This 
> means that only files that need processing are in the input directory.  

Unstated in my original post, but that is exactly what we do.  Files arrive in 
one directory and then are forwarded on to another machine and saved in an
archive directory locally.
 
> finish and show up in step 3 above.  I've seen up to five second delays on 
> heavily loaded Linux systems running ext2 filesystems.  Ext3 and Reiser 
> running in journalling mode could actually have this problem worse than ext2.  
> The WinNT filesystem can get really weird this way.  On a heavily loaded 
> system, I timed a file taking more than 30 seconds to show up in a directory 
> after a copy operation said it was complete.

Amazing.  I had no idea.  That's very important.
 
> Is there some sort of sentinel that you can look for at the end of the file?  
> If the file is pretty big, just having a fudge factor delay isn't really a 
> solution.  It might alleviate the problem, but it won't solve it.
 
> > ################## start of 2dos ####################
> > #!/usr/bin/perl -i
> >  [snip]
> > ############# end of 2dos #############################
> 
> Erm, where does the output go?  Are these programs sanitized to protect the 
> innocent?

You missed the -i on the #! line, an in place edit with no backup.  

-- 
      Michael Rasmussen  aka  mikeraz
    Be appropriate && Follow your curiosity
 http://www.patch.com/ http://wiki.patch.com/
 http://wiki.patch.com/index.php/BicycleCommuting

   The fortune cookie says:
Never reveal your best argument.