Fetch API documentation

Fetch values from the Ska engineering telemetry archive.

Classes

class cheta.fetch.MSID(msid, start='1999:204', stop=None, filter_bad=False, stat=None)[source]

Bases: object

Fetch data from the engineering telemetry archive into an MSID object.

The input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_x (ORBITEPHEM1_X) or *pcadmd (AOPCADMD). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

  • msid – name of MSID (case-insensitive)

  • start – start date of telemetry (Chandra.Time compatible)

  • stop – stop date of telemetry (current time if not supplied)

  • filter_bad – automatically filter out bad values

  • stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

MSID instance

fetch = <module 'cheta.fetch' from '/export/jgonzale/github-workflows/shiny-runner-sot/_work/test-actions/test-actions/source/cheta/fetch.py'>
filter_bad(bads=None, copy=False)[source]

Filter out any bad values.

After applying this method the “bads” column will be set to None to indicate that there are no bad values.

Parameters:

  • bads – Bad values mask. If not supplied then self.bads is used.

  • copy – return a copy of MSID object with bad values filtered

filter_bad_times(start=None, stop=None, table=None, copy=False)[source]

Filter out intervals of bad data in the MSID object.

There are three usage options:

  • Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().

  • Supply both start and stop values where each is a single value in a valid DateTime format.

  • Supply a table parameter with one of two forms: (1) list of bad time interval strings, where each string contains the start and stop dates separated by a space; (2) string with the name of a file in the same format.

Examples:

bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msid.filter_bad_times(table=bad_times)
msid.filter_bad_times(table='msid_bad_times.dat')
Parameters:

  • start – Start of time interval to exclude (any DateTime format)

  • stop – End of time interval to exclude (any DateTime format)

  • table – List of str with start and stop for bad time intervals or str with name of file containing list of bad time intervals

  • copy – return a copy of MSID object with bad times filtered

interpolate(dt=None, start=None, stop=None, times=None)[source]

Perform nearest-neighbor interpolation of the MSID to the specified time sequence.

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the first and last times for the MSID.

The MSID times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

If times is provided then this gets used instead of the default linear progression from start and dt.

Parameters:

  • dt – time step (sec, default=328.0)

  • start – start of interpolation period (DateTime format)

  • stop – end of interpolation period (DateTime format)

  • times – array of times for interpolation (default=None)

iplot(fmt='-b', fmt_minmax='-c', **plot_kwargs)[source]

Make an interactive plot for exploring the MSID data.

This method opens a new plot figure (or clears the current figure) and plots the MSID vals versus times. This plot can be panned or zoomed arbitrarily and the data values will be fetched from the archive as needed. Depending on the time scale, iplot displays either full resolution, 5-minute, or daily values. For 5-minute and daily values the min and max values are also plotted.

Once the plot is displayed and the window is selected by clicking in it, the following key commands are recognized:

a: autoscale for full data range in x and y
m: toggle plotting of min/max values
p: pan at cursor x
y: toggle autoscaling of y-axis
z: zoom at cursor x
?: print help

Example:

dat = fetch.Msid('aoattqt1', '2011:001', '2012:001', stat='5min')
dat.iplot()
dat.iplot('.b', '.c', markersize=0.5)

Caveat: the iplot() method is not meant for use within scripts, and may give unexpected results if used in combination with other plotting commands directed at the same plot figure.

Parameters:

  • fmt – plot format for values (default=”-b”)

  • fmt_minmax – plot format for mins and maxes (default=”-c”)

  • plot_kwargs – additional plotting keyword args

logical_intervals(op, val, complete_intervals=False, max_gap=None)[source]

Determine contiguous intervals during which the logical comparison expression “MSID.vals op val” is True. Allowed values for op are:

==  !=  >  <  >=  <=

If complete_intervals is True (default is False) then the intervals are guaranteed to be complete so that the all reported intervals had a transition before and after within the telemetry interval.

If max_gap is specified then any time gaps longer than max_gap are filled with a fictitious False value to create an artificial interval boundary at max_gap / 2 seconds from the nearest data value.

Returns a structured array table with a row for each interval. Columns are:

  • datestart: date of interval start

  • datestop: date of interval stop

  • duration: duration of interval (sec)

  • tstart: time of interval start (CXC sec)

  • tstop: time of interval stop (CXC sec)

Examples:

>>> dat = fetch.MSID('aomanend', '2010:001', '2010:005')
>>> manvs = dat.logical_intervals('==', 'NEND', complete_intervals=True)

>>> dat = fetch.MSID('61PSTS02', '1999:200', '2000:001')
>>> safe_suns = dat.logical_intervals('==', 'SSM', max_gap=66)
Parameters:

  • op – logical operator, one of == != > < >= <=

  • val – comparison value

  • complete_intervals – return only complete intervals (default=True)

  • max_gap – max allowed gap between time stamps (sec, default=None)

Returns:

structured array table of intervals

plot(*args, **kwargs)[source]

Plot the MSID vals using Ska.Matplotlib.plot_cxctime()

This is a convenience function for plotting the MSID values. It is equivalent to:

plot_cxctime(self.times, self.vals, *args, **kwargs)

where *args are additional arguments and **kwargs are additional keyword arguments that are accepted by plot_cxctime().

Example:

dat = fetch.Msid('tephin', '2011:001', '2012:001', stat='5min')
dat.plot('-r', linewidth=2)
property raw_vals

Raw counts corresponding to the string state-code values that are stored in self.vals

remove_intervals(intervals, copy=False)[source]

Remove telemetry points that occur within the specified intervals

This method is the converse of select_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to removing intervals, and that copy is returned. The default is to remove intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then removed and the standard deviation is recomputed. This filters out the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.remove_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
-0.0003688639491030978
>>> figure(2)
>>> aorate2.plot(',')
Parameters:

  • intervals – EventQuery or iterable (N x 2) with start, stop dates/times

  • copy – return a copy of MSID object with intervals removed

select_intervals(intervals, copy=False)[source]

Select telemetry points that occur within the specified intervals

This method is the converse of remove_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to selecting intervals, and that copy is returned. The default is to selecte intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then selected and the mean is recomputed. This highlights the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.select_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
24.764309542605481
>>> figure(2)
>>> aorate2.plot(',')
Parameters:

  • intervals – EventQuery or iterable (N x 2) with start, stop dates/times

  • copy – return a copy of MSID object with intervals selected

property state_codes

List of state codes tuples (raw_count, state_code) for state-valued MSIDs

state_intervals()[source]

Determine contiguous intervals during which the MSID value is unchanged.

Returns a structured array table with a row for each interval. Columns are:

  • datestart: date of interval start

  • datestop: date of interval stop

  • duration: duration of interval (sec)

  • tstart: time of interval start (CXC sec)

  • tstop: time of interval stop (CXC sec)

  • val: MSID value during the interval

Example:

dat = fetch.MSID('cobsrqid', '2010:001', '2010:005')
obsids = dat.state_intervals()
Parameters:

val – state value for which intervals are returned.

Returns:

structured array table of intervals

property tdb

Access the Telemetry database entries for this MSID

write_zip(filename, append=False)[source]

Write MSID to a zip file named filename

Within the zip archive the data for this MSID will be stored in csv format with the name <msid_name>.csv.

Parameters:

  • filename – output zipfile name

  • append – append to an existing zipfile

class cheta.fetch.MSIDset(msids=None, start='1999:204', stop=None, filter_bad=False, stat=None)[source]

Bases: OrderedDict

Fetch a set of MSIDs from the engineering telemetry archive.

Each input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_? (ORBITEPHEM1_X, Y and Z) or aoattqt[1234] (AOATTQT1, 2, 3, and 4). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

  • msids – list of MSID names (case-insensitive)

  • start – start date of telemetry (Chandra.Time compatible)

  • stop – stop date of telemetry (current time if not supplied)

  • filter_bad – automatically filter out bad values

  • stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

Dict-like object containing MSID instances keyed by MSID name

class MSID(msid, start='1999:204', stop=None, filter_bad=False, stat=None)

Bases: object

Fetch data from the engineering telemetry archive into an MSID object.

The input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_x (ORBITEPHEM1_X) or *pcadmd (AOPCADMD). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

  • msid – name of MSID (case-insensitive)

  • start – start date of telemetry (Chandra.Time compatible)

  • stop – stop date of telemetry (current time if not supplied)

  • filter_bad – automatically filter out bad values

  • stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

MSID instance

fetch = <module 'cheta.fetch' from '/export/jgonzale/github-workflows/shiny-runner-sot/_work/test-actions/test-actions/source/cheta/fetch.py'>
filter_bad(bads=None, copy=False)

Filter out any bad values.

After applying this method the “bads” column will be set to None to indicate that there are no bad values.

Parameters:

  • bads – Bad values mask. If not supplied then self.bads is used.

  • copy – return a copy of MSID object with bad values filtered

filter_bad_times(start=None, stop=None, table=None, copy=False)

Filter out intervals of bad data in the MSID object.

There are three usage options:

  • Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().

  • Supply both start and stop values where each is a single value in a valid DateTime format.

  • Supply a table parameter with one of two forms: (1) list of bad time interval strings, where each string contains the start and stop dates separated by a space; (2) string with the name of a file in the same format.

Examples:

bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msid.filter_bad_times(table=bad_times)
msid.filter_bad_times(table='msid_bad_times.dat')
Parameters:

  • start – Start of time interval to exclude (any DateTime format)

  • stop – End of time interval to exclude (any DateTime format)

  • table – List of str with start and stop for bad time intervals or str with name of file containing list of bad time intervals

  • copy – return a copy of MSID object with bad times filtered

interpolate(dt=None, start=None, stop=None, times=None)

Perform nearest-neighbor interpolation of the MSID to the specified time sequence.

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the first and last times for the MSID.

The MSID times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

If times is provided then this gets used instead of the default linear progression from start and dt.

Parameters:

  • dt – time step (sec, default=328.0)

  • start – start of interpolation period (DateTime format)

  • stop – end of interpolation period (DateTime format)

  • times – array of times for interpolation (default=None)

iplot(fmt='-b', fmt_minmax='-c', **plot_kwargs)

Make an interactive plot for exploring the MSID data.

This method opens a new plot figure (or clears the current figure) and plots the MSID vals versus times. This plot can be panned or zoomed arbitrarily and the data values will be fetched from the archive as needed. Depending on the time scale, iplot displays either full resolution, 5-minute, or daily values. For 5-minute and daily values the min and max values are also plotted.

Once the plot is displayed and the window is selected by clicking in it, the following key commands are recognized:

a: autoscale for full data range in x and y
m: toggle plotting of min/max values
p: pan at cursor x
y: toggle autoscaling of y-axis
z: zoom at cursor x
?: print help

Example:

dat = fetch.Msid('aoattqt1', '2011:001', '2012:001', stat='5min')
dat.iplot()
dat.iplot('.b', '.c', markersize=0.5)

Caveat: the iplot() method is not meant for use within scripts, and may give unexpected results if used in combination with other plotting commands directed at the same plot figure.

Parameters:

  • fmt – plot format for values (default=”-b”)

  • fmt_minmax – plot format for mins and maxes (default=”-c”)

  • plot_kwargs – additional plotting keyword args

logical_intervals(op, val, complete_intervals=False, max_gap=None)

Determine contiguous intervals during which the logical comparison expression “MSID.vals op val” is True. Allowed values for op are:

==  !=  >  <  >=  <=

If complete_intervals is True (default is False) then the intervals are guaranteed to be complete so that the all reported intervals had a transition before and after within the telemetry interval.

If max_gap is specified then any time gaps longer than max_gap are filled with a fictitious False value to create an artificial interval boundary at max_gap / 2 seconds from the nearest data value.

Returns a structured array table with a row for each interval. Columns are:

  • datestart: date of interval start

  • datestop: date of interval stop

  • duration: duration of interval (sec)

  • tstart: time of interval start (CXC sec)

  • tstop: time of interval stop (CXC sec)

Examples:

>>> dat = fetch.MSID('aomanend', '2010:001', '2010:005')
>>> manvs = dat.logical_intervals('==', 'NEND', complete_intervals=True)

>>> dat = fetch.MSID('61PSTS02', '1999:200', '2000:001')
>>> safe_suns = dat.logical_intervals('==', 'SSM', max_gap=66)
Parameters:

  • op – logical operator, one of == != > < >= <=

  • val – comparison value

  • complete_intervals – return only complete intervals (default=True)

  • max_gap – max allowed gap between time stamps (sec, default=None)

Returns:

structured array table of intervals

plot(*args, **kwargs)

Plot the MSID vals using Ska.Matplotlib.plot_cxctime()

This is a convenience function for plotting the MSID values. It is equivalent to:

plot_cxctime(self.times, self.vals, *args, **kwargs)

where *args are additional arguments and **kwargs are additional keyword arguments that are accepted by plot_cxctime().

Example:

dat = fetch.Msid('tephin', '2011:001', '2012:001', stat='5min')
dat.plot('-r', linewidth=2)
property raw_vals

Raw counts corresponding to the string state-code values that are stored in self.vals

remove_intervals(intervals, copy=False)

Remove telemetry points that occur within the specified intervals

This method is the converse of select_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to removing intervals, and that copy is returned. The default is to remove intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then removed and the standard deviation is recomputed. This filters out the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.remove_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
-0.0003688639491030978
>>> figure(2)
>>> aorate2.plot(',')
Parameters:

  • intervals – EventQuery or iterable (N x 2) with start, stop dates/times

  • copy – return a copy of MSID object with intervals removed

select_intervals(intervals, copy=False)

Select telemetry points that occur within the specified intervals

This method is the converse of remove_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to selecting intervals, and that copy is returned. The default is to selecte intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then selected and the mean is recomputed. This highlights the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.select_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
24.764309542605481
>>> figure(2)
>>> aorate2.plot(',')
Parameters:

  • intervals – EventQuery or iterable (N x 2) with start, stop dates/times

  • copy – return a copy of MSID object with intervals selected

property state_codes

List of state codes tuples (raw_count, state_code) for state-valued MSIDs

state_intervals()

Determine contiguous intervals during which the MSID value is unchanged.

Returns a structured array table with a row for each interval. Columns are:

  • datestart: date of interval start

  • datestop: date of interval stop

  • duration: duration of interval (sec)

  • tstart: time of interval start (CXC sec)

  • tstop: time of interval stop (CXC sec)

  • val: MSID value during the interval

Example:

dat = fetch.MSID('cobsrqid', '2010:001', '2010:005')
obsids = dat.state_intervals()
Parameters:

val – state value for which intervals are returned.

Returns:

structured array table of intervals

property tdb

Access the Telemetry database entries for this MSID

write_zip(filename, append=False)

Write MSID to a zip file named filename

Within the zip archive the data for this MSID will be stored in csv format with the name <msid_name>.csv.

Parameters:

  • filename – output zipfile name

  • append – append to an existing zipfile

copy() → a shallow copy of od[source]
filter_bad(copy=False, union=False)[source]

Filter bad values for the MSID set.

By default (union=False) the bad values are filtered individually for each MSID.

If union=True this method applies the union (logical-OR) of bad value masks for all MSIDs in the set with the same content type. The result is that the filtered MSID samples are valid for all MSIDs within the content type and the arrays all match up.

For example:

msids = fetch.MSIDset(['aorate1', 'aorate2', 'aogyrct1', 'aogyrct2'],
                      '2009:001', '2009:002')
msids.filter_bad()

Since aorate1 and aorate2 both have content type of pcad3eng they will be filtered as a group and will remain with the same sampling. This will allow something like:

plot(msids['aorate1'].vals, msids['aorate2'].vals)

Likewise the two gyro count MSIDs would be filtered as a group. If this group-filtering is not the desired behavior one can always call the individual MSID.filter_bad() function for each MSID in the set:

for msid in msids.values():
    msid.filter_bad()
Parameters:

copy – return a copy of MSID object with intervals selected

filter_bad_times(start=None, stop=None, table=None, copy=False)[source]

Filter out intervals of bad data in the MSIDset object.

There are three usage options:

  • Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().

  • Supply both start and stop values where each is a single value in a valid DateTime format.

  • Supply a table parameter with one of two forms: (1) list of bad time interval strings, where each string contains the start and stop dates separated by a space; (2) string with the name of a file in the same format.

Examples:

msidset.filter_bad_times()
bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msidset.filter_bad_times(table=bad_times)
msidset.filter_bad_times(table='msid_bad_times.dat')
Parameters:

  • start – Start of time interval to exclude (any DateTime format)

  • stop – End of time interval to exclude (any DateTime format)

  • table – List of str with start and stop for bad time intervals or str with name of file containing list of bad time intervals

  • copy – return a copy of MSID object with intervals selected

interpolate(dt=None, start=None, stop=None, filter_bad=True, times=None, bad_union=False, copy=False)[source]

Perform nearest-neighbor interpolation of all MSID values in the set to a common time sequence. The values are updated in-place.

Times

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the start and stop times for the MSID set.

If times is provided then this gets used instead of the default linear progression from start and dt.

For each MSID in the set the times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

Filtering and bad values

If filter_bad is True (default) then bad values are filtered from the interpolated MSID set. There are two strategies for doing this:

  1. bad_union = False

    Remove the bad values in each MSID prior to interpolating the set to a common time series. This essentially says to use all the available data individually. Each MSID has bad data filtered individually before interpolation so that the nearest neighbor interpolation only finds good data. This strategy is done when filter_union = False, which is the default setting.

  2. bad_union = True

Mark every MSID in the set as bad at the interpolated time if any of them are bad at that time. This stricter version is required when it is important that the MSIDs be truly correlated in time. For instance this is needed for attitude quaternions since all four values must be from the exact same telemetry sample. If you are not sure, this is the safer option.

Parameters:

  • dt – time step (sec, default=328.0)

  • start – start of interpolation period (DateTime format)

  • stop – end of interpolation period (DateTime format)

  • filter_bad – filter bad values

  • times – array of times for interpolation (default=None)

  • bad_union – filter union of bad values after interpolating

  • copy – return a new copy instead of in-place update (default=False)

write_zip(filename)[source]

Write MSIDset to a zip file named filename

Within the zip archive the data for each MSID in the set will be stored in csv format with the name <msid_name>.csv.

Parameters:

filename – output zipfile name

class cheta.fetch.Msid(msid, start='1999:204', stop=None, filter_bad=True, stat=None)[source]

Bases: MSID

Fetch data from the engineering telemetry archive into an MSID object. Same as MSID class but with filter_bad=True by default.

Parameters:

  • msid – name of MSID (case-insensitive)

  • start – start date of telemetry (Chandra.Time compatible)

  • stop – stop date of telemetry (current time if not supplied)

  • filter_bad – automatically filter out bad values

  • stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

  • unit_system – Unit system (cxc|eng|sci, default=current units)

Returns:

MSID instance

class cheta.fetch.Msidset(msids, start='1999:204', stop=None, filter_bad=True, stat=None)[source]

Bases: MSIDset

Fetch a set of MSIDs from the engineering telemetry archive. Same as MSIDset class but with filter_bad=True by default.

Parameters:

  • msids – list of MSID names (case-insensitive)

  • start – start date of telemetry (Chandra.Time compatible)

  • stop – stop date of telemetry (current time if not supplied)

  • filter_bad – automatically filter out bad values

  • stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

  • unit_system – Unit system (cxc|eng|sci, default=current units)

Returns:

Dict-like object containing MSID instances keyed by MSID name

class MSID(msid, start='1999:204', stop=None, filter_bad=False, stat=None)

Bases: object

Fetch data from the engineering telemetry archive into an MSID object.

The input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_x (ORBITEPHEM1_X) or *pcadmd (AOPCADMD). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

  • msid – name of MSID (case-insensitive)

  • start – start date of telemetry (Chandra.Time compatible)

  • stop – stop date of telemetry (current time if not supplied)

  • filter_bad – automatically filter out bad values

  • stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

MSID instance

fetch = <module 'cheta.fetch' from '/export/jgonzale/github-workflows/shiny-runner-sot/_work/test-actions/test-actions/source/cheta/fetch.py'>
filter_bad(bads=None, copy=False)

Filter out any bad values.

After applying this method the “bads” column will be set to None to indicate that there are no bad values.

Parameters:

  • bads – Bad values mask. If not supplied then self.bads is used.

  • copy – return a copy of MSID object with bad values filtered

filter_bad_times(start=None, stop=None, table=None, copy=False)

Filter out intervals of bad data in the MSID object.

There are three usage options:

  • Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().

  • Supply both start and stop values where each is a single value in a valid DateTime format.

  • Supply a table parameter with one of two forms: (1) list of bad time interval strings, where each string contains the start and stop dates separated by a space; (2) string with the name of a file in the same format.

Examples:

bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msid.filter_bad_times(table=bad_times)
msid.filter_bad_times(table='msid_bad_times.dat')
Parameters:

  • start – Start of time interval to exclude (any DateTime format)

  • stop – End of time interval to exclude (any DateTime format)

  • table – List of str with start and stop for bad time intervals or str with name of file containing list of bad time intervals

  • copy – return a copy of MSID object with bad times filtered

interpolate(dt=None, start=None, stop=None, times=None)

Perform nearest-neighbor interpolation of the MSID to the specified time sequence.

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the first and last times for the MSID.

The MSID times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

If times is provided then this gets used instead of the default linear progression from start and dt.

Parameters:

  • dt – time step (sec, default=328.0)

  • start – start of interpolation period (DateTime format)

  • stop – end of interpolation period (DateTime format)

  • times – array of times for interpolation (default=None)

iplot(fmt='-b', fmt_minmax='-c', **plot_kwargs)

Make an interactive plot for exploring the MSID data.

This method opens a new plot figure (or clears the current figure) and plots the MSID vals versus times. This plot can be panned or zoomed arbitrarily and the data values will be fetched from the archive as needed. Depending on the time scale, iplot displays either full resolution, 5-minute, or daily values. For 5-minute and daily values the min and max values are also plotted.

Once the plot is displayed and the window is selected by clicking in it, the following key commands are recognized:

a: autoscale for full data range in x and y
m: toggle plotting of min/max values
p: pan at cursor x
y: toggle autoscaling of y-axis
z: zoom at cursor x
?: print help

Example:

dat = fetch.Msid('aoattqt1', '2011:001', '2012:001', stat='5min')
dat.iplot()
dat.iplot('.b', '.c', markersize=0.5)

Caveat: the iplot() method is not meant for use within scripts, and may give unexpected results if used in combination with other plotting commands directed at the same plot figure.

Parameters:

  • fmt – plot format for values (default=”-b”)

  • fmt_minmax – plot format for mins and maxes (default=”-c”)

  • plot_kwargs – additional plotting keyword args

logical_intervals(op, val, complete_intervals=False, max_gap=None)

Determine contiguous intervals during which the logical comparison expression “MSID.vals op val” is True. Allowed values for op are:

==  !=  >  <  >=  <=

If complete_intervals is True (default is False) then the intervals are guaranteed to be complete so that the all reported intervals had a transition before and after within the telemetry interval.

If max_gap is specified then any time gaps longer than max_gap are filled with a fictitious False value to create an artificial interval boundary at max_gap / 2 seconds from the nearest data value.

Returns a structured array table with a row for each interval. Columns are:

  • datestart: date of interval start

  • datestop: date of interval stop

  • duration: duration of interval (sec)

  • tstart: time of interval start (CXC sec)

  • tstop: time of interval stop (CXC sec)

Examples:

>>> dat = fetch.MSID('aomanend', '2010:001', '2010:005')
>>> manvs = dat.logical_intervals('==', 'NEND', complete_intervals=True)

>>> dat = fetch.MSID('61PSTS02', '1999:200', '2000:001')
>>> safe_suns = dat.logical_intervals('==', 'SSM', max_gap=66)
Parameters:

  • op – logical operator, one of == != > < >= <=

  • val – comparison value

  • complete_intervals – return only complete intervals (default=True)

  • max_gap – max allowed gap between time stamps (sec, default=None)

Returns:

structured array table of intervals

plot(*args, **kwargs)

Plot the MSID vals using Ska.Matplotlib.plot_cxctime()

This is a convenience function for plotting the MSID values. It is equivalent to:

plot_cxctime(self.times, self.vals, *args, **kwargs)

where *args are additional arguments and **kwargs are additional keyword arguments that are accepted by plot_cxctime().

Example:

dat = fetch.Msid('tephin', '2011:001', '2012:001', stat='5min')
dat.plot('-r', linewidth=2)
property raw_vals

Raw counts corresponding to the string state-code values that are stored in self.vals

remove_intervals(intervals, copy=False)

Remove telemetry points that occur within the specified intervals

This method is the converse of select_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to removing intervals, and that copy is returned. The default is to remove intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then removed and the standard deviation is recomputed. This filters out the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.remove_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
-0.0003688639491030978
>>> figure(2)
>>> aorate2.plot(',')
Parameters:

  • intervals – EventQuery or iterable (N x 2) with start, stop dates/times

  • copy – return a copy of MSID object with intervals removed

select_intervals(intervals, copy=False)

Select telemetry points that occur within the specified intervals

This method is the converse of remove_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to selecting intervals, and that copy is returned. The default is to selecte intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then selected and the mean is recomputed. This highlights the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.select_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
24.764309542605481
>>> figure(2)
>>> aorate2.plot(',')
Parameters:

  • intervals – EventQuery or iterable (N x 2) with start, stop dates/times

  • copy – return a copy of MSID object with intervals selected

property state_codes

List of state codes tuples (raw_count, state_code) for state-valued MSIDs

state_intervals()

Determine contiguous intervals during which the MSID value is unchanged.

Returns a structured array table with a row for each interval. Columns are:

  • datestart: date of interval start

  • datestop: date of interval stop

  • duration: duration of interval (sec)

  • tstart: time of interval start (CXC sec)

  • tstop: time of interval stop (CXC sec)

  • val: MSID value during the interval

Example:

dat = fetch.MSID('cobsrqid', '2010:001', '2010:005')
obsids = dat.state_intervals()
Parameters:

val – state value for which intervals are returned.

Returns:

structured array table of intervals

property tdb

Access the Telemetry database entries for this MSID

write_zip(filename, append=False)

Write MSID to a zip file named filename

Within the zip archive the data for this MSID will be stored in csv format with the name <msid_name>.csv.

Parameters:

  • filename – output zipfile name

  • append – append to an existing zipfile

Functions

cheta.fetch.create_msid_data_gap(msid_obj: MSID, data_gap_spec: str)[source]

Make a data gap in the msid_obj (in-place) by removing data points.

This is mostly useful for testing via setting the CHETA_FETCH_DATA_GAP environment variable.

The data_gap_spec string corresponds to a command line argument string with the following options:

--include INCLUDE  Include MSIDs matching glob (default="*", can be repeated)
--exclude EXCLUDE  Exclude MSIDs matching glob (default=None, can be repeated)
--start START      Gap start (CxoTimeLike)
--stop STOP        Gap stop (CxoTimeLike)

For example:

>>> dat = fetch.MSID('aopcadmd', '2010:001', '2010:002')
>>> gap_spec = "--start=-2010:001:12:00:00 --stop=2010:001:12:30:00"
>>> fetch.create_msid_data_gap(dat, gap_spec)
Parameters:

  • msid_obj – MSID object

  • data_gap_spec – data gap specification

cheta.fetch.get_telem(msids, start=None, stop=None, sampling='full', unit_system='eng', interpolate_dt=None, remove_events=None, select_events=None, time_format=None, outfile=None, quiet=False, max_fetch_Mb=1000, max_output_Mb=100)[source]

High-level routine to get telemetry for one or more MSIDs and perform common processing functions:

  • Fetch a set of MSIDs over a time range, specifying the sampling as either full-resolution, 5-minute, or daily data.

  • Filter out bad or missing data.

  • Interpolate (resample) all MSID values to a common uniformly-spaced time sequence.

  • Remove or select time intervals corresponding to specified Kadi event types.

  • Change the time format from CXC seconds (seconds since 1998.0) to something more convenient like GRETA time.

  • Write the MSID telemetry data to a zipfile.

Parameters:

  • msids – MSID(s) to fetch (string or list of strings)’)

  • start – Start time for data fetch (default=<stop> - 30 days)

  • stop – Stop time for data fetch (default=NOW)

  • sampling – Data sampling (full | 5min | daily) (default=full)

  • unit_system – Unit system for data (eng | sci | cxc) (default=eng)

  • interpolate_dt – Interpolate to uniform time steps (secs, default=None)

  • remove_events – Remove kadi events expression (default=None)

  • select_events – Select kadi events expression (default=None)

  • time_format – Output time format (secs|date|greta|jd|…, default=secs)

  • outfile – Output file name (default=None)

  • quiet – Suppress run-time logging output (default=False)

  • max_fetch_Mb – Max allowed memory (Mb) for fetching (default=1000)

  • max_output_Mb – Max allowed memory (Mb) for file output (default=100)

Returns:

MSIDset object

cheta.fetch.get_time_range(msid, format=None)[source]

Get the time range for the given msid.

Parameters:

  • msid – MSID name

  • format – Output format (DateTime format, e.g. ‘secs’, ‘date’, ‘greta’)

Returns:

(tstart, tstop) in CXC seconds

cheta.fetch.get_units()[source]

Get the unit system currently being used for conversions.

cheta.fetch.msid_glob(msid)[source]

Get the archive MSIDs matching msid.

The function returns a tuple of (msids, MSIDs) where msids is a list of MSIDs that is all lower case and (where possible) matches the input msid. The output MSIDs is all upper case and corresponds to the exact MSID names stored in the archive HDF5 files.

Parameters:

msid – input MSID glob

Returns:

tuple (msids, MSIDs)

cheta.fetch.read_bad_times(table)[source]

Include a list of bad times from table in the fetch module bad_times registry. This routine can be called multiple times with different tables and the bad times will be appended to the registry. The table can include any number of bad time interval specifications, one per line. A bad time interval line has three columns separated by whitespace, e.g.:

aogbias1  2008:292:00:00:00  2008:297:00:00:00

The MSID name is not case sensitive and the time values can be in any DateTime format. Blank lines and any line starting with the # character are ignored.

cheta.fetch.set_units(unit_system)[source]

Set the unit system used for output telemetry values. The default is “cxc”. Allowed values for unit_system are:

cxc

FITS standard units used in CXC archive files (basically MKS)

sci

Same as “cxc” but with temperatures in degC instead of Kelvins

eng

OCC engineering units (TDB P009, e.g. degF, ft-lb-sec, PSI)
Parameters:

unit_system – system of units (cxc, sci, eng)

Time and Date formats

The Ska engineering telemetry archive tools support a wide range of formats for representing date-time stamps. Note that within this and other documents for this tool suite, the words ‘time’ and ‘date’ are used interchangably to mean a date-time stamp.

The available formats are listed in the table below:

Format

Description

System

secs

Elapsed seconds since 1998-01-01T00:00:00

tt

numday

DDDD:hh:mm:ss.ss… Elapsed days and time

utc

jd*

Julian Day

utc

mjd*

Modified Julian Day = JD - 2400000.5

utc

date

YYYY:DDD:hh:mm:ss.ss..

utc

caldate

YYYYMonDD at hh:mm:ss.ss..

utc

fits

FITS date/time format YYYY-MM-DDThh:mm:ss.ss..

tt

unix*

Unix time (since 1970.0)

utc

greta

YYYYDDD.hhmmss[sss]

utc

* Ambiguous for input parsing and only available as output formats.

For the date format one can supply only YYYY:DDD in which case 12:00:00.000 is implied.

The default time “System” for the different formats is either tt (Terrestrial Time) or utc (UTC). Since TT differs from UTC by around 64 seconds it is important to be consistent in specifying the time format.