spacepy.datamodel

Data model conceptually similar to HDF5 and CDF.

For more documentation Datamodel

Authors: Steve Morley and Brian Larsen

Additional Contributors: Charles Kiyanda and Miles Engel

Institution: Los Alamos National Laboratory

Contact: smorley@lanl.gov; balarsen@lanl.gov

Copyright 2010-2016 Los Alamos National Security, LLC.

Functions

convertKeysToStr(SDobject)
createISTPattrs(datatype[, ndims, vartype, ...])	Return set of unpopulated attributes for ISTP compliant variable
dmcopy(dobj)	Generic copy utility to return a copy of a (datamodel) object
dmfilled(shape[, fillval, dtype, order, attrs])	Return a new dmarray of given shape and type, filled with a specified value (default=0).
flatten(dobj)	Collapse datamodel to one level deep
fromCDF(fname)	Create a SpacePy datamodel representation of a NASA CDF file
fromHDF5(fname, **kwargs)	Create a SpacePy datamodel representation of an HDF5 file or netCDF4 file which is HDF5 compliant
fromNC3(fname)
fromRecArray(recarr)	Takes a numpy recarray and returns each field as a dmarray in a SpaceData container
readJSONMetadata(fname, **kwargs)	Read JSON metadata from an ASCII data file
readJSONheadedASCII(fname[, mdata, comment, ...])	read JSON-headed ASCII data files into a SpacePy datamodel
resample(data[, time, winsize, overlap, ...])	resample a SpaceData to a new time interval
toCDF(fname, SDobject[, skeleton, flatten, ...])	Create a CDF file from a SpacePy datamodel representation
toHDF5(fname, SDobject, **kwargs)	Create an HDF5 file from a SpacePy datamodel representation
toHTML(fname, SDobject[, attrs, varLinks, ...])	Create an HTML dump of the structure of a spacedata
toJSONheadedASCII(fname, insd[, metadata, ...])	Write JSON-headed ASCII file of data with metadata from SpaceData object
toRecArray(sdo)	Takes a SpaceData and creates a numpy recarray
unflatten(dobj[, marker])	Collapse datamodel to one level deep
writeJSONMetadata(fname, insd[, depend0, ...])	Scrape metadata from SpaceData object and make a JSON header

Classes

ISTPArray()	Mixin class for array using ISTP metadata.
ISTPContainer()	Mixin class for containers using ISTP metadata.
MetaMixin()	Mixin class that adds a 'meta' attribute that acts like 'attrs'
SpaceData(*args, **kwargs)	Datamodel class extending dict by adding attributes.
dmarray(input_array[, attrs, dtype])	Container for data within a SpaceData object

Exceptions

DMWarning

Warnings class for datamodel, subclassed so it can be set to always

spacepy.datamodel.convertKeysToStr(SDobject)

spacepy.datamodel.createISTPattrs(datatype, ndims=1, vartype=None, units=' ', NRV=False)

Return set of unpopulated attributes for ISTP compliant variable

Parameters:

datatype{‘data’, ‘support_data’, ‘metadata’}

datatype of variable to create metadata for.

ndimsint

number of dimensions, default=1

vartype{‘float’, ‘char’, ‘int’, ‘epoch’, ‘tt2000’}

The type of the variable, default=float

unitsstr

The units of the variable, default=’ ‘

NRVbool

Is the variable NRV (non-record varying), default=False

Returns:

attrsdict

dictionary of attributes for the variable

Examples

>>> import spacepy.datamodel as dm
>>> dm.createISTPattrs('data', ndims=2, vartype='float', units='MeV')
{'CATDESC': '',
 'DISPLAY_TYPE': 'spectrogram',
 'FIELDNAM': '',
 'FILLVAL': -1e+31,
 'FORMAT': 'F18.6',
 'LABLAXIS': '',
 'SI_CONVERSION': ' > ',
 'UNITS': 'MeV',
 'VALIDMIN': '',
 'VALIDMAX': '',
 'VAR_TYPE': 'data',
 'DEPEND_0': 'Epoch',
 'DEPEND_1': ''}

spacepy.datamodel.dmcopy(dobj)

Generic copy utility to return a copy of a (datamodel) object

Parameters:

dobjobject

object to return a copy of

Returns:

copy_obj: object (same type as input)

copy of input oibject

Examples

>>> import spacepy.datamodel as dm
>>> dat = dm.dmarray([2,3], attrs={'units': 'T'})
>>> dat1 = dm.dmcopy(dat)
>>> dat1.attrs['copy': True]
>>> dat is dat1
False
>>> dat1.attrs
{'copy': True, 'units': 'T'}
>>> dat.attrs
{'units': 'T'}

spacepy.datamodel.dmfilled(shape, fillval=0, dtype=None, order='C', attrs=None)

Return a new dmarray of given shape and type, filled with a specified value (default=0).

See also

numpy.ones

Examples

>>> import spacepy.datamodel as dm
>>> dm.dmfilled(5, attrs={'units': 'nT'})
dmarray([ 0.,  0.,  0.,  0.,  0.])

>>> dm.dmfilled((5,), fillval=1, dtype=np.int)
dmarray([1, 1, 1, 1, 1])

>>> dm.dmfilled((2, 1), fillval=np.nan)
dmarray([[ nan],
       [ nan]])

>>> a = dm.dmfilled((2, 1), np.nan, attrs={'units': 'nT'})
>>> a
dmarray([[ nan],
       [ nan]])
>>> a.attrs
{'units': 'nT'}

spacepy.datamodel.flatten(dobj)

Collapse datamodel to one level deep

See also

unflatten

SpaceData.flatten

Examples

>>> import spacepy.datamodel as dm
>>> import spacepy.toolbox as tb
>>> a = dm.SpaceData()
>>> a['1'] = dm.SpaceData(dog = 5, pig = dm.SpaceData(fish=dm.SpaceData(a='carp', b='perch')))
>>> a['4'] = dm.SpaceData(cat = 'kitty')
>>> a['5'] = 4
>>> a.tree()
+
|____1
     |____dog
     |____pig
          |____fish
               |____a
               |____b
|____4
     |____cat
|____5

>>> b = dm.flatten(a)
>>> b.tree()
+
|____1<--dog
|____1<--pig<--fish<--a
|____1<--pig<--fish<--b
|____4<--cat
|____5

>>> a.flatten()
>>> a.tree()
+
|____1<--dog
|____1<--pig<--fish<--a
|____1<--pig<--fish<--b
|____4<--cat
|____5

spacepy.datamodel.fromCDF(fname)

Create a SpacePy datamodel representation of a NASA CDF file

Parameters:

filestring

the name of the cdf file to be loaded into a datamodel

Returns:

outspacepy.datamodel.SpaceData

SpaceData with associated attributes and variables in dmarrays

See also

pycdf.CDF.copy

pycdf.istp.VarBundle

Examples

>>> import spacepy.datamodel as dm
>>> data = dm.fromCDF('test.cdf')

spacepy.datamodel.fromHDF5(fname, **kwargs)

Create a SpacePy datamodel representation of an HDF5 file or netCDF4 file which is HDF5 compliant

Parameters:

filestring

the name of the HDF5/netCDF4 file to be loaded into a datamodel

Returns:

outspacepy.datamodel.SpaceData

SpaceData with associated attributes and variables in dmarrays

Notes

Zero-sized datasets will break in h5py. This is kluged by returning a dmarray containing a None.

This function is expected to work with any HDF5-compliant files, including netCDF4 (not netCDF3) and MatLab save files from v7.3 or later, but some datatypes are not supported, e.g., non-string vlen datatypes, and will raise a warning.

Examples

>>> import spacepy.datamodel as dm
>>> data = dm.fromHDF5('test.hdf')

spacepy.datamodel.fromNC3(fname)

spacepy.datamodel.fromRecArray(recarr)

Takes a numpy recarray and returns each field as a dmarray in a SpaceData container

Parameters:

recarrnumpy record array

object to parse into SpaceData container

Returns:

sd: spacepy.datamodel.SpaceData

dict-like containing arrays of named records in recarr

Examples

>>> import numpy as np
>>> import spacepy.datamodel as dm
>>> x = np.array([(1.0, 2), (3.0, 4)], dtype=[('x', float), ('y', int)])
>>> print(x, x.dtype)
array([(1.0, 2), (3.0, 4)], dtype=[('x', '<f8'), ('y', '<i4')])
>>> sd = dm.fromRecArray(x)
>>> sd.tree(verbose=1)
+
|____x (spacepy.datamodel.dmarray (2,))
|____y (spacepy.datamodel.dmarray (2,))

spacepy.datamodel.readJSONMetadata(fname, **kwargs)

Read JSON metadata from an ASCII data file

Parameters:

fnamestr

Filename to read metadata from

Changed in version 0.5.0: Filename can now be a .gz to indicate the file is gzipped

Returns:

mdata: spacepy.datamodel.SpaceData

SpaceData with the metadata from the file

Other Parameters:

verbosebool (optional)

set verbose output so metadata tree prints on read (default False)

spacepy.datamodel.readJSONheadedASCII(fname, mdata=None, comment='#', convert=False, restrict=None)

read JSON-headed ASCII data files into a SpacePy datamodel

Parameters:

fnamestr or list

Filename(s) to read data from

Changed in version 0.5.0: Filename can now be a .gz to indicate the file is gzipped

Returns:

mdata: spacepy.datamodel.SpaceData

SpaceData with the data and metadata from the file

Other Parameters:

mdataspacepy.datamodel.SpaceData (optional)

supply metadata object, otherwise is read from fname (default None)

comment: str (optional)

comment string in file to be read; lines starting with comment are ignored (default ‘#’)

convert: bool or dict-like (optional)

If True, uses common names to try conversion from string. If a dict- like then uses the functions specified as the dict values to convert each element of ‘key’ to a non-string

restrict: list of strings (optional)

If present, restrict the variables stored to only those on this list

spacepy.datamodel.resample(data, time=None, winsize=0, overlap=0, st_time=None, outtimename='Epoch')

resample a SpaceData to a new time interval

Parameters:

dataSpaceData or dmarray

SpaceData with data to resample or dmarray with data to resample, variables can only be 1d or 2d, if time is specified only variables the same length as time are resampled, otherwise only variables with length equal to the longest length are resampled

timearray-like

dmarray of times the correspond to the data

winsizedatetime.timedelta

Time frame to average the data over

overlapdatetime.timedelta

Overlap in the moving average

st_timedatetime.datetime

Starting time for the resample, if not specified the time of the first data point is used (see spacepy.toolbox.windowMean)

Returns:

ansSpaceData

Resampled data, included keys are in the input keys (with the data caveats above) and Epoch which contains the output time

Examples

>>> import datetime
>>> import spacepy.datamodel as dm
>>> a = dm.SpaceData()
>>> a.attrs['foo'] = 'bar'
>>> a['a'] = dm.dmarray(range(10*2)).reshape(10,2)
>>> a['b'] = dm.dmarray(range(10)) + 4
>>> a['c'] = dm.dmarray(range(3)) + 10
>>> times = [datetime.datetime(2010, 1, 1) + datetime.timedelta(hours=i) for i in range(10)]
>>> out = dm.resample(a, times, winsize=datetime.timedelta(hours=2), overlap=datetime.timedelta(hours=0))
>>> out.tree(verbose=1, attrs=1)
# +
# :|____foo (str [3])
# |____Epoch (spacepy.datamodel.dmarray (4,))
# |____a (spacepy.datamodel.dmarray (4, 2))
# :|____DEPEND_0 (str [5])
#
# Things to note:
#    - attributes are preserved
#    - the output variables have their DEPEND_0 changed to Epoch (or outtimename)
#    - each dimension of a 2d array is resampled individually

spacepy.datamodel.toCDF(fname, SDobject, skeleton='', flatten=False, overwrite=False, autoNRV=False, backward=None, TT2000=None, verbose=False)

Create a CDF file from a SpacePy datamodel representation

Parameters:

fnamestr

Filename to write to

SDobjectspacepy.datamodel.SpaceData

SpaceData with associated attributes and variables in dmarrays

Returns:

None

Other Parameters:

skeletonstr (optional)

create new CDF from a skeleton file (default ‘’)

flattenbool (optional)

flatten incoming datamodel - if SpaceData objects are nested (default False)

overwritebool (optional)

allow overwrite of an existing target file (default False)

autoNRVbool (optional)

attempt automatic identification of non-record varying entries in CDF

backwardbool (optional)

False to create CDF in backward-compatible format; True

to force v3+ compatibility only. (Default: do not change current state, see set_backward()).

Changed in version 0.5.0: Now supports specifying backward compatible or no change; previous versions always wrote v3+ CDFs (even if False).

TT2000bool (optional)

Specify type for variables with names beginning ‘Epoch’. Default CDF_EPOCH for backward-compatible CDF (backward True) and CDF_TT20000 otherwise (backward False or unspecified).

Changed in version 0.5.0: Current handling introduced.

Changed in version 0.3.0: Always write TT2000 variables (due to change in pycdf).

verbosebool (optional)

verbosity flag

Notes

Changed in version 0.5.0: Invalid keyword arguments now raise TypeError rather than being ignored.

spacepy.datamodel.toHDF5(fname, SDobject, **kwargs)

Create an HDF5 file from a SpacePy datamodel representation

Parameters:

fnamestr

Filename to write to

SDobjectspacepy.datamodel.SpaceData

SpaceData with associated attributes and variables in dmarrays

Returns:

None

Other Parameters:

overwritebool (optional)

allow overwrite of an existing target file (default True)

modestr (optional)

HDF5 file open mode (a, w, r) (default ‘a’)

compressionstr (optional)

compress all non-scalar variables using this method (default None) (gzip, shuffle, fletcher32, szip, lzf)

Changed in version 0.4.0: No longer compresses scalars (which usually fails).

compression_optsstr (optional)

options to the compression, see h5py documentation for more details

Examples

>>> import spacepy.datamodel as dm
>>> a = dm.SpaceData()
>>> a['data'] = dm.dmarray(range(100000), dtype=float)
>>> dm.toHDF5('test_gzip.h5', a, overwrite=True, compression='gzip')
>>> dm.toHDF5('test.h5', a, overwrite=True)
>>> # test_gzip.h5 was 118k, test.h5 was 785k

spacepy.datamodel.toHTML(fname, SDobject, attrs=(), varLinks=False, echo=False, tableTag='<table border="1">')

Create an HTML dump of the structure of a spacedata

Parameters:

fnamestr

Filename to write to

SDobjectspacepy.datamodel.SpaceData

SpaceData with associated attributes and variables in dmarrays

Other Parameters:

overwritebool (optional)

allow overwrite of an existing target file (default True)

modestr (optional)

HDF5 file open mode (a, w, r) (default ‘a’)

echobool

echo the html to the screen

varLinksbool

make the variable name a link to a stub page

spacepy.datamodel.toJSONheadedASCII(fname, insd, metadata=None, depend0=None, order=None, **kwargs)

Write JSON-headed ASCII file of data with metadata from SpaceData object

Parameters:

fnamestr

Filename to write to (can also use a file-like object) None can be given in conjunction with the returnString keyword to skip writing output

insdspacepy.datamodel.SpaceData

SpaceData with associated attributes and variables in dmarrays

Returns:

None

Other Parameters:

depend0str (optional)

variable name to use to indicate parameter on which other data depend (e.g. Time)

orderlist (optional)

list of key names in order of start column in output JSON file

metadata: str or file-like (optional)

filename with JSON header to use (or file-like with JSON metadata)

delimiter: str

delimiter to use in ASCII output (default is whitespace), for tab, use ‘ ‘

Examples

>>> import spacepy.datamodel as dm
>>> data = dm.SpaceData()
>>> data.attrs['Global'] = 'A global attribute'
>>> data['Var1'] = dm.dmarray([1,2,3,4,5], attrs={'Local1': 'A local attribute'})
>>> data['Var2'] = dm.dmarray([[8,9],[9,1],[3,4],[8,9],[7,8]])
>>> data['MVar'] = dm.dmarray([7.8], attrs={'Note': 'Metadata'})
>>> dm.toJSONheadedASCII('outFile.txt', data, depend0='Var1', order=['Var1'])
#Note that not all field names are required, those not given will be listed
#alphabetically after those that are specified

spacepy.datamodel.toRecArray(sdo)

Takes a SpaceData and creates a numpy recarray

Parameters:

sdoSpaceData

SpaceData to change to a numpy recarray

Returns:

recarr: numpy record array

numpy.recarray object with the same values (attributes are lost)

Examples

>>> import numpy as np
>>> import spacepy.datamodel as dm
>>> sd = dm.SpaceData()
>>> sd['x'] = dm.dmarray([1.0, 2.0])
>>> sd['y'] = dm.dmarray([2,4])
>>> sd.tree(verbose=1)
+
|____x (spacepy.datamodel.dmarray (2,))
|____y (spacepy.datamodel.dmarray (2,))
>>> ra = dm.toRecArray(sd)
>>> print(ra, ra.dtype)
[(2, 1.0) (4, 2.0)] (numpy.record, [('y', '<i8'), ('x', '<f8')])

spacepy.datamodel.unflatten(dobj, marker='<--')

Collapse datamodel to one level deep

Examples

>>> import spacepy.datamodel as dm
>>> import spacepy.toolbox as tb
>>> a = dm.SpaceData()
>>> a['1'] = dm.SpaceData(dog = 5, pig = dm.SpaceData(fish=dm.SpaceData(a='carp', b='perch')))
>>> a['4'] = dm.SpaceData(cat = 'kitty')
>>> a['5'] = 4
>>> a.tree()
+
|____1
     |____dog
     |____pig
          |____fish
               |____a
               |____b
|____4
     |____cat
|____5

>>> b = dm.flatten(a)
>>> b.tree()
+
|____1<--dog
|____1<--pig<--fish<--a
|____1<--pig<--fish<--b
|____4<--cat
|____5

>>> c = dm.unflatten(b)
>>> c.tree()
+
|____1
     |____dog
     |____pig
          |____fish
               |____a
               |____b
|____4
     |____cat
|____5

spacepy.datamodel.writeJSONMetadata(fname, insd, depend0=None, order=None, verbose=False, returnString=False)

Scrape metadata from SpaceData object and make a JSON header

Parameters:

fnamestr

Filename to write to (can also use a file-like object) None can be given in conjunction with the returnString keyword to skip writing output

insdspacepy.datamodel.SpaceData

SpaceData with associated attributes and variables in dmarrays

Returns:

None (unless returnString keyword is True)

Other Parameters:

depend0str (optional)

variable name to use to indicate parameter on which other data depend (e.g. Time)

orderlist (optional)

list of key names in order of start column in output JSON file

verbose: bool (optional)

verbose output

returnString: bool (optional)

return JSON header as string instead of returning None

exception spacepy.datamodel.DMWarning

Warnings class for datamodel, subclassed so it can be set to always