Calling python worker script from master script as multiple instances

I have a python script that is watching a web service request workspace. Each time a client submits a job to my web service a unique job folder is created in a well known location. I have a script polling this well known location for folder with out a "flag" (a blank text file with a specific name that indicates processing is complete on this job).

Right now my script can call a worker script to process the contents of the new folder but has to wait until the worker script is finished before it can proceed handing folders out.

My question is what are options to have new instances of the worker script instatiated and return control to the manager. Would creating a python executable that takes parameters of the worker script and have the manager script call it via the command line work? Or would creating the worker script into a class that can have numerous instations processing work?

Once the worker script is done it does not need to message back to the manager script job complete. It will does this via dropping a text file into the directory. Though now that I think about it I will have to hold somewhere that the each job directory has been handed out because it will take 1.5 minutes for the worker script to process.

Any advice/links would be much appreciated.

-------------Problems Reply------------

First of all, I agree that you need to put a flag in your directories indicating that a directory is being processed. The master script should be the only one to set the flag, or you will risk race conditions (two worker scripts taking the same directory at the same time). You can use the same file; the master script creates it empty (meaning "in progress") and the worker script writes 1B in it (meaning "done"). That way, the master script only has to check the existence of the flag.

Back to your question:

  • you can indeed make your worker script into a standalone program, and call it via the subprocess module;
  • you can make it a thread (with the threading module [2]), which is somewhat easier to code; this may be inefficient because of the GIL, but if your worker script is highly IO-bound, it should not be too much of a problem;
  • if you are using Python 3, you may want to look at the multiprocessing module [3]
    which I never used but seems to mix the usability of threading without being vulnerable to the GIL; it seems that it is not completely portable though.

Hope this helps

  • [1]
  • [2]
  • [3]
Category:python Views:0 Time:2012-04-10

Related post

  • Can a single perl script running in background hold multiple instances of Log4Perl? 2011-11-28

    I have a script "" which is running in background and which is self-logging using Log4Perl. This script continuously reads in a directory and detects new files created in it with Linux::Inotify2 module. Each detected file is a Storable objec

  • best way in producing a master script for SQL 2010-02-03

    i want to extract specific database tables & stored procedures into one master script. Do you know any software that can help me do this faster? I've tried using the SQL Database publishing tool, but it's not that efficient since its gathering ta

  • put All jquery Scripts in master page 2010-10-26

    i am using jquery and jquery ui plugins in my web application. and i have some server controls that they register their scripts on the page. Since my application should be work Local could i put all scripts in master page ? all scripts i mean

  • python install changes script interpreter 2010-12-23

    I have a Python package that includes a few scripts in a scripts/ folder. My file includes:: #!/usr/bin/env python from distutils.core import setup scripts = ['script1', 'script2', 'script3'] setup(name='Test', version='0.1.0', packages=['te

  • Perl - Master script calling sub-scripts and return status 2011-01-27

    Here's the design I want to accomplish in Perl: A master script calls multiple sub-scripts. The master script controls the calling of each sub-script in a particular sequence and records output from each sub-script in order to decide whether on not t

  • Passing Data to a Python Web Crawler from PHP Script 2011-03-31

    I've got a python crawler crawling a few webpages every few minutes. I'm now trying to implement a user interface to be accessed over the web and to display the data obtained by the crawler. I'm going to use php/html for the interface. Anyway, the us

  • Handling an interactive script from another script , using python 2011-05-17

    I have a shell script and this one is interactive and I am creating some automation to provide inputs to this shell scripts. The automation is done using Python. example, the shell script waits for an input like, "what is the domain name?" now the py

  • Is it better to send XMLHttpRequests to different scripts or one master script that delegates tasks? 2011-06-09

    I am building a rather larger web application with javascript and PHP. The app has several different types of XMLHttpRequests, and my question is about best practice: is it better to send each of those requests to a different PHP script or to one mas

  • Python for a beginner Shell scripting learner? 2011-09-29

    I'm learning Linux administration and after that i'm going to learn programming on Linux using Python. While i'm reading in a Linux Administration book, i came into Shell scripting chapter. I saw Bash language and i got dizzy. Let's say it's not one

  • script loader master pages and path errors 2011-11-28

    I am using [script.js][1] as async script loader along with my master page. The pages on the root are working fine using the master page file as it's master page. When it comes to web pages that are inside folders like below then the path does not wo

  • Calling a Python function from a shell script 2011-12-19

    I am trying to figure out how to call a Python function from a shell script. I have a Python file with multiple functions and I need to use the values returned by them in my shell script. Is there a way to do it. I am doing this in order to read a co

  • How may one determine which version of Python is suitable for a script? 2013-04-01

    I want to determine which version of Python would be most appropriate for a Python script. Is there an automated way to do this? The specific problem I have in mind is to determine which version of Python to use with the script (called ea800_download

  • Looking for a script which will script out all database level persmissions( including xp_cmdshell and sql proxy account) 2009-12-01

    I need a stored procedure or script which will script out all database level permissions- including permissions to xp_cmdshell for any objects as well as permissions to a sql proxy account. Thanks! --------------Solutions------------- Your problem is

  • How do you Call a Script from a Script? 2011-03-30

    I'm creating what I call a "Script Manager" that will fire a script based on product type. So, if the user inputs Aerial or Radius, the appropriate script is ran. I can get this to run using os.system('python C:\' + parameter variables) The

  • Calling script from shell script - getting command not found 2012-01-25

    I am new to shell scripting. I am trying to work through this. > script to execute in cron ( #!/bin/sh HOST='ahostname' PORT='3306' USER='auser' PASS='apassword' DB='adatabase' . /mnt/stor/backups/ (I also tried source /mnt/stor/b

  • How to open a script in sl4a script editor, programmatically from another script? 2012-03-01

    I'm new to Android python scripting with SL4A, but familliar with python. the question I have is, How to open a script in sl4a script editor, programmatically from another script? my english is not goodenough to explain my problem, so I show it with

  • how to debug asp server-side script using MS script debugger 2009-12-02

    how to debug asp server-side script using MS script debugger. --------------Solutions------------- You may check this article: For debug asp server side you must configure IIS manager

  • document.createElement('script')- adding two scripts with one callback 2009-12-08

    I need to add prototype and then add scriptaculous and get a callback when they are both done loading. I am currently loading prototype like so: var script = document.createElement("script"); script.src = "

  • Difference between launching a script with ./ and . ./ 2009-12-10

    Please tell me what is the difference in bash shell between launching a script with ./ and . ./ --------------Solutions------------- As klausbyskov says, the first form requries that the file have its executable permission bit set.

Copyright (C), All Rights Reserved.

processed in 0.103 (s). 11 q(s)