UKC

Hi,

I am using the Linux system call 'inotify' to watch a directory and kick off a process (another script) whenever a csv file is uploaded into the directory. The csv file is then processed and loaded into an Oracle database. This works fine.

The problem I am having is when another csv file is uploaded into the watched directory whilst a previous csv file is still being processed and loaded into the database.

How can I get inotify to 'know' there is another csv file waiting and then process that one after the current csv file has finished being processed?

Just some general pointers on the concept(s) of how to do this would be great. I can work out the details if I know the overall idea.

Probably a bit of a long shot, but thanks in advance.

In reply to Richard White:
Simplest solution is probably to modify your script to exit immediately if there is more than one csv file in the repository.

In reply to RomTheBear:

Thanks for the suggestion but it can't do that. The process / capability needs to be running 24/7 in order to process any csv files that are uploaded.

The csv files come from a remote automated monitoring system and uploads are made automatically when certain criteria are met.

In reply to Richard White:

Are you using one script?

It might be better to keep the inotify as one process and the processing of the CSV as something spawned. That way the inotify script can spawn successive processing jobs without waiting for them to end. You'll need to make sure you database can handle multiple processing jobs or manage the jobs.

You also need to a handle event overflow with inotify, there is some basic info in the man page.

In reply to Richard White:

Don't you get a notification of the other file when you go read again from the file descriptor that came back from the notify_init ?

The documentation suggests that events get queued until you read them.
(I've not used notify I have to admit!)

In reply to Philip:




Indeed it seems a better choice and that way you can simply make the spawned process stay in a wait loop if there is another instance running, if you don't want two imports running at the same time.

In reply to Philip:

That's sounds like a good idea, I'll try to work out how to do that. I am new to using inotify so the man page is, as usual, not easy to understand on first acquaintance.

In reply to MeMeMe:

It is probably the way I am using inotify that is causing the problem.

I'll try and work out how the next part of the script can deal with the queued events.

Thanks.