dbprocessing.dbprocessing.ProcessQueue

class dbprocessing.dbprocessing.ProcessQueue(mission, dryrun=False, echo=False)

Main code used to process the Queue.

Looks in incoming and builds all possible files

Warning

As this object holds a reference to DButils, that database should be closed before the program terminates. Deleting this object will ordinarily suffice.

__init__(mission, dryrun=False, echo=False)

Initializes the process queue

Parameters

missionstr

Mission database, as in DButils. May also be an existing instance of DButils.

dryrunbool, default False

Treat all functionality as “dry run”, make no changes.

echobool, default False

Log all SQL statements (as in DButils).

Methods

buildChildren(file_id[, debug, skip_run, ...])	go through and all the runMe's and add to the runme_list variable
checkIncoming([glb])	Goes out to incoming and grabs all files there adding them to self.queue
diskfileToDB(df)	given a diskfile go through and do all the steps to add it into the db
figureProduct([filename])	Imports inspectors and figures out which inspectors claim the file
importFromIncoming()	Import a file from incoming into the database
mk_tempdir([suffix])	create a secure temp directory
moveToError(fname)	Moves a file from incoming to error
onStartup()	Processes can be defined as output timebase "STARTUP" which means to run them each time to processing chain is run
reprocessByCode(id_in[, startDate, endDate, ...])	Add files made by a code to the queue for reprocessing.
reprocessByDate([startDate, endDate, ...])	Add files to the queue for reprocessing, by file date.
reprocessByInstrument(id_in[, level, ...])	Add files for an instrument to the queue for reprocessing
reprocessByProduct(id_in[, startDate, ...])	Add files of a particular product to the queue for reprocessing.
rm_tempdir()	remove the temp directory
set_filename(filename)	Setter for filename, this is cleaner than just random sets

Attributes

filename

buildChildren(file_id, debug=False, skip_run=False, run_procs=None)

go through and all the runMe’s and add to the runme_list variable

Parameters

file_idint

file ID of the file for which children will be built

skip_runbool, default False

Skip RUN timebase processes if True

run_procsstr, optional

If provided, comma-separated list of process IDs or process names to run; other processes are ignored. (Default: all possible processes).

checkIncoming(glb='*')

Goes out to incoming and grabs all files there adding them to self.queue

Parameters

glbstr, optional

Glob pattern that files must match.

diskfileToDB(df)

given a diskfile go through and do all the steps to add it into the db

Parameters

dfDiskfile

File to add to database.

figureProduct(filename=None)

Imports inspectors and figures out which inspectors claim the file

Parameters

filenamestr, optional

Full path to file to check, default filename.

importFromIncoming()

Import a file from incoming into the database

mk_tempdir(suffix='_dbprocessing_28973')

create a secure temp directory

moveToError(fname)

Moves a file from incoming to error

Parameters

fnamestr

Full path to file to move to error.

onStartup()

Processes can be defined as output timebase “STARTUP” which means to run them each time to processing chain is run

reprocessByCode(id_in, startDate=None, endDate=None, incVersion=None)

Add files made by a code to the queue for reprocessing.

Parameters

id_instr or int

ID or filename of code to reprocess

startDatedatetime, optional

First date to reprocess (default all)

endDatedatetime, optional

Last date to reprocess (default all)

incVersionint {0, 1, 2}, optional

Which version number to increment: major (0), minor (1), subminor (2). Forces reprocessing. (default do not force).

reprocessByDate(startDate=None, endDate=None, incVersion=None, level=None)

Add files to the queue for reprocessing, by file date.

Parameters

startDatedatetime, optional

First date to reprocess (default all)

endDatedatetime, optional

Last date to reprocess (default all)

incVersionint {0, 1, 2}, optional

Which version number to increment: major (0), minor (1), subminor (2). Forces reprocessing (default do not force).

levelfloat, optional

Only reprocess files of this level (default all)

reprocessByInstrument(id_in, level=None, startDate=None, endDate=None, incVersion=None)

Add files for an instrument to the queue for reprocessing

Parameters

id_instr or int

ID or name of instrument to reprocess

levelint, optional

Only reprocess files of this level (default all)

startDatedatetime, optional

First date to reprocess (default all)

endDatedatetime, optional

Last date to reprocess (default all)

incVersionint {0, 1, 2}, optional

Which version number to increment: major (0), minor (1), subminor (2). Forces reprocessing (default do not force).

reprocessByProduct(id_in, startDate=None, endDate=None, incVersion=None)

Add files of a particular product to the queue for reprocessing.

Parameters

id_instr or int

ID or name of code to reprocess

startDatedatetime, optional

First date to reprocess (default all)

endDatedatetime, optional

Last date to reprocess (default all)

incVersionint {0, 1, 2}, optional

Which version number to increment: major (0), minor (1), subminor (2). Forces reprocessing (default do not force).

rm_tempdir()

remove the temp directory

set_filename(filename)

Setter for filename, this is cleaner than just random sets

Parameters

filenamestr

filename to set to filename

filename

Full path to file currently being processed (str)

---

Release: 0.1.0 Doc generation date: Feb 10, 2022