Release Notes¶
0.1 Series¶
0.1.0 (2022-02-10)¶
This is the first public packaged release of dbprocessing. For the convenience of those working directly from git checkouts, these release notes summarize changes made since the creation of the public repository on 2020-07-15.
New features¶
Support for processes that take no input was added, as part of many enhancements to DBRunner.py (26).
New script linkUningested.py to find files which match product format but are not in database, and symlink them to the incoming directory so they can be ingested (54).
Python 3 support was added (77).
Added options printProcessQueue.py --count,
printProcessQueue.py --exist, and printProcessQueue.py
--quiet to allow flow control in shell scripts based on the process
queue state (87, 88).
Added options printProcessQueue.py --product to filter output
and printProcessQueue.py --sort to sort output
(93).
Initial PostgreSQL support was added; see
Specifying a database and CreateDB.py --dialect
(64,
78
114).
Deprecations and removals¶
None
Dependency requirements¶
SQLAlchemy and dateutil are required. No
minimum version has been established, but dbprocessing is known to
work with SQLAlchemy 1.1 and dateutil 2.6.
Major bugfixes¶
Fixed buildChildren on
older databases without yesterday and tomorrow columns in the
Productprocesslink table (20).
Fixed runMe to always maintain the
output_interface version specified by the code. Updates to a code’s
interface version will increment the quality version of its output (not
the interface); updates to a file’s interface version will increment the
quality version of child files (rather than assuming children were up-to-date
and failing to reprocess them). (63)
Fixed buildChildren()
(specifically, helper method _getRequiredProducts) to only look
for files which are recorded in the database as existing on disk. Most
notably, this means ProcessQueue.py
will not attempt to use nonexistent files as inputs to processing.
Other changes¶
Pull requests merged this release¶
PR 8: Fix unit tests for scripts (44690a95)
PR 13: Circleci project setup (c99ffc43)
PR 18: Fix handling of inspector regex for ECT (61a2215e)
PR 1: Basic respository organization (362a0b72)
PR 19: test_checkIncoming update to guarantee directory state (Closes #17) (16899b2c)
17: test_checkIncoming fails when a direction on my system is not empty
PR 21: Fix _getRequiredProducts/getInputProductID on old DB (Closes #20) (07f3e3a1)
20: ‘Productprocesslink’ has no attribute ‘yesterday’, old database
PR 24: Testing changes in preparation for no-input support (24ca0b57)
PR 27: Run unit tests in CircleCI (1fe1ef46)
PR 26: DBRunner enhancements for ingest, update-only, force (Closes #9) (5c05a165)
9: Support products with no input
PR 31: Speed up getFiles by start/stop time (Closes #23) (b5f30ef3)
23: ProcessQueue “Command Build Progress” is slow
PR 36: addFile: fix utc_start_time/stop_time as dates with Unix time table (333e47dc)
PR 39: CircleCI: update apt cache before installing packages (fix CreateDBsabrs test) (00922ed5)
PR 38: Fix Unix time table calculation to maintain file day (20afffc4)
PR 41: remove the #! to python2.6 in favor of python, this was making my system barf that wants to use 2.7 (8e5d3ae4)
PR 49: Build Sphinx docs in CI; check for warnings (240be3df)
PR 50: More documentation of pull request processing; release notes (b470f436)
PR 51: add docker authentication (af0ab929)
32: Add Docker auth to CircleCI
PR 56: Fix make clean for the autosummary docs (e4eb0c01)
PR 55: Move table definitions to separate, common module. (bfb806af)
PR 54: New linkUningested script: find files not in database and symlink for import (734f37b1)
PR 63: Increment output quality versions when input/code interface version changes (1c153080)
PR 64: Minimal postgresql support in dbUtils (11a417e8)
PR 66: Triage scripts and port to argparse (80d12b59)
PR 67: Fix test_DBfile when called by itself (142a4976)
PR 73: Update postgresql dialect name in CreateDBsabrs (b5ba751b)
PR 72: fast_data: add option to archive files instead of delete. (dc3fb0be)
PR 77: Python 3 support (29b66787)
PR 79: Fix addFromConfig for processes with no output product (3aafa1e1)
PR 78: add postgresql support and unit testing (5c0082c9)
PR 90: Only use products that exist on disk when figuring inputs (0bae771b)
47: _getRequiredProducts should require input files to exist on disc
PR 91: fast_data: skip deleting files that don’t exist on disk (b6711a74)
PR 92: Use docutils <0.16 when building docs in CI (d81efdfb)
PR 88: add count/exist/quiet options to printProcessQueue (5e6e4132)
87: Use exit code of printProcessQueue to indicate if the queue is empty
PR 93: printProcessQueue: add ability to filter by product, sort (6469cb26)
PR 98: Force empty output products to null (Closes #95) (4a8ec7ad)
95: No output fail in postgres
PR 94: Make documentation fully automatic (79363013)
PR 99: test_DButils: clarify no-output tests (34d3cb0b)
PR 100: Update documentation for scripts (fdcc7e38)
PR 101: Close DBrunner db on exit; document the need to close (74cec351)
PR 102: Document tables/database structure (6852abd1)
PR 112: Documentation overhaul (a48f2629)
PR 113: Create “empty” test database in unit tests (4300d704)
PR 114: Convert all unit tests to use Postgres (fb95a08f)
34: Create RBSP_MAGEIS.sqlite database
PR 119: Get unit tests working on Windows (2ca3554f)
PR 120: Fix python 3.9 and sqlalchemy 1.4 deprecations (0a0a7442)
83: deprecations: collections.abc, Engine.table_name
PR 122: Provide support documentation integrated with GitHub and other doc tweaks (6b6ad6b8)
118: Link github page in documentation
PR 123: Updates for release 0.1.0
PR 124: Updated docs for release 0.1.0
Other issues closed this release¶
2: Website not finding CSS
42: PR checklist calls for CHANGELOG update, yet there is no changelog
48: Clean up tags in repo
Release: 0.1.0 Doc generation date: Feb 10, 2022