dbprocessing.Utils¶

Various utilities of use throughout this code.

Functions

`chunker`(seq, size)	Return a long iterable in a tuple of shorter lists.
`dateForPrinting`([dt, microseconds, brackets])	Return a string of the date format for printing on the screen
`daterange_to_dates`(daterange)	Given a daterange return the date objects for all days in the range
`datetimeToDate`(dt)	Given an input datetime.datetime or datetime.date return a datetime.date
`dirSubs`(path, filename, utc_file_date, ...)	Do any substitutions that are needed to put thing in the right place
`expandDates`(start_time, stop_time)	Given a start and a stop date make all the dates in between
`flatten`(l)	Flatten an irregularly nested list of lists
`parseDate`(inval)	Given a date of the for yyyy-mm-dd parse to a datetime.
`parseVersion`(inval)	Given a format of the form x.y.z parse to a Version
`processRunning`(pid)	Given a PID see if it is currently running.
`progressbar`(count, blocksize, totalsize[, text])	Print a progress bar with urllib.urlretrieve reporthook functionality
`readconfig`(config_filepath)	Read a database definition config file.
`split_code_args`(args)	Split a string with a bunch of command line arguments into a list
`strargs_to_args`(strargs)	Read in the arguments string from the db and change to a dict
`toBool`(value)	Returns true if passed 'True', 'true', True, 1, 'Yes', 'yes', 'Y', or 'y'
`toDatetime`(dt[, end])	Convert a date, date string, or datetime to a datetime
`toNone`(value)	Returns None if passed '', 'None', 'none', or 'NONE'
`unique`(seq)	Take a list and return only the unique elements in the same order

dbprocessing.Utils.chunker(seq, size)[source]¶

Return a long iterable in a tuple of shorter lists.

Taken from https://stackoverflow.com/questions/434287/what-is-the-most-pythonic-way-to-iterate-over-a-list-in-chunks

Parameters

seqIterable: Iterable to split up
sizeint: Size of each split in the output, last one has the remaining elements of seq

Returns

tuple: tuple of lists of the iterable seq split into len(seq)/size segments

dbprocessing.Utils.dateForPrinting(dt=None, microseconds=False, brackets='[]')[source]¶

Return a string of the date format for printing on the screen

If dt is None return now().

Parameters

dtdatetime, default now(): object to format
microsecondsbool, default False: Include microseconds
bracketsstr, optional: Which brackets to encase the time in, default ('[', ']')

Returns

str: Iso formatted string

Examples

>>> from dbprocessing.Utils import dateForPrinting
>>> print("{0} Something occurred".format(dateForPrinting()))
[2016-03-22T10:51:45]  Something occurred

dbprocessing.Utils.daterange_to_dates(daterange)[source]¶

Given a daterange return the date objects for all days in the range

Parameters

daterangeSequence of datetime: Start and stop dates

Returns

list: All the dates between daterange[0] and daterange[1]

dbprocessing.Utils.datetimeToDate(dt)[source]¶

Given an input datetime.datetime or datetime.date return a datetime.date

Parameters

dtdatetime or date: input to convert

Returns

date: Same date as in dt, but as a date object.

dbprocessing.Utils.dirSubs(path, filename, utc_file_date, utc_start_time, version, dbu=None)[source]¶

Do any substitutions that are needed to put thing in the right place

Todo

This may be useless/could be made more useful

Honored substitutions used as {Y}{PRODUCT}{DATE}

Parameters

pathstr: Path to the file
filenamestr: Name of the time
utc_file_datedatetime: File’s date
utc_start_timedatetime: File’s start time
versionVersion: Version to substitute
dbuDButils, optional: Current database connection. If not specified, creates a new connection.

Returns

str

Notes

Valid subsitutions are:

Y: 4 digit year
m: 2 digit month
b: 3 character month (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)
d: 2 digit day
y: 2 digit year
j: 3 digit day of year
H: 2 digit hour (24-hour time)
M: 2 digit minute
S: 2 digit second
VERSION: version string, interface.quality.revision
DATE: the UTC date from a file, same as Ymd
MISSION: the mission name from the db
SPACECRAFT: the spacecraft name from the db
PRODUCT: the product name from the db

dbprocessing.Utils.expandDates(start_time, stop_time)[source]¶

Given a start and a stop date make all the dates in between

Inclusive on the ends

Parameters

start_timedatetime: Date to start the list
stop_timedatetime: Date to end the list, inclusive

Returns

:class:`list` ofclass:~datetime.datetime: All the dates between start_time and stop_time

dbprocessing.Utils.flatten(l)[source]¶

Flatten an irregularly nested list of lists

Taken from https://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists

Parameters

llist: Nested list of lists to flatten

Returns

list: Flattened list

dbprocessing.Utils.parseDate(inval)[source]¶

Given a date of the for yyyy-mm-dd parse to a datetime.

This is just a wrapper around strptime() If the format is wrong ValueError is raised.

Parameters

invalstr: String date representation of the form YYYY-MM-DD

Returns

datetime: datetime object parsed from the string

dbprocessing.Utils.parseVersion(inval)[source]¶

Given a format of the form x.y.z parse to a Version

This is a wrapper around fromString().

Parameters

invalstr: String Version representation of the form xx.yy.zz

Returns

Version: Version object parsed from the string

dbprocessing.Utils.processRunning(pid)[source]¶

Given a PID see if it is currently running.

Taken from from https://stackoverflow.com/questions/568271/how-to-check-if-there-exists-a-process-with-a-given-pid-in-python

Parameters

pidint: Process ID

Returns

bool: True if pid is running, False otherwise

dbprocessing.Utils.progressbar(count, blocksize, totalsize, text='Download Progress')[source]¶

Print a progress bar with urllib.urlretrieve reporthook functionality

Taken from spacepy

Parameters

countfloat: The current count of the progressbar
blocksizefloat: The size of each block (mostly useful for file downloads)
totalsizefloat: The total size of the job, progress is count*blocksize*100/totalsize
textstr, optional: The text to print in the progressbar

Examples

>>> import spacepy.toolbox as tb
>>> import urllib
>>> urllib.urlretrieve(config['psddata_url'], PSDdata_fname, reporthook=tb.progressbar)

dbprocessing.Utils.readconfig(config_filepath)[source]¶

Read a database definition config file.

Parameters

config_filepathstr: full path to the config file

Returns

dict: Dictionary of key value pairs from config files

dbprocessing

dbprocessing.Utils¶

Previous topic

Next topic

This Page