Python interface to MAUDE telemetry server

The maude package provides a low-level Python interface to the MAUDE Chandra telemetry server. For more information on MAUDE check its webpage, which includes extensive documentation.

Setup for authentication

In order to use maude you must have authentication credentials (username and password) to access OCCweb. One can provide those credentials manually to the get_msids() function call, but this gets tiresome.

The preferred method to use this from a secure machine is to edit the file .netrc in your home directory and put in your OCCweb credentials.

IMPORTANT: make sure the file is readable only by you!

chmod og-rwx ~/.netrc

Once you have done that, add these three lines. If there are already other machines defined you need a blank line between the machine configs.

machine  occweb
login    your-occweb-username
password your-occweb-password

Usage

Get MSIDs

The public interface to the maude package is a essentially a single function get_msids() which takes the following arguments:

Arg

Description

msids

Single MSID or list of MSIDs

start

Start time (any Chandra.Time format, default=NOW - 2 days)

stop

Stop time (any Chandra.Time format, default=NOW)

tolerance

MAUDE tolerance parameter (millisec, default=None)

alltimes

MAUDE alltimes parameter (default=False)

channel

MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)

format

MAUDE response format [json | xml | bin] (default=bin)

user

OCCweb user name

password

OCCweb password

As a first example we get the PCAD mode over a short period of time in 2016:

>>> import maude
>>> dat = maude.get_msids('AOPCADMD', start='2016:030:00:00:00', stop='2016:030:00:00:10')

This returns a Python dict structure with four keys:

Key

Description

data

List of telemetry outputs, where each element is a dict.

format

MAUDE binary data format (you normally don’t care about this)

n_msids

Number of MSIDs in the query list

query

Input query values for reference

For the query above, the value of dat is:

>>> dat
{'data': [{'dtype': <type 'numpy.int8'>,
           'flags': {'subset': False, 'tolerance': False},
           'msid': 'AOPCADMD',
           'msid_index': 2416,
           'n_values': 10,
           'raw_values': array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int8),
           'times': array([  5.70499268e+08,   5.70499269e+08,   5.70499270e+08,
                             5.70499271e+08,   5.70499272e+08,   5.70499273e+08,
                             5.70499274e+08,   5.70499275e+08,   5.70499276e+08,
                             5.70499277e+08]),
           'type_code': 0,
           'values': array(['NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT',
                            'NPNT', 'NPNT'],
                           dtype='|S4')}],
 'format': 1,
 'n_msids': 1,
 'query': {'alltimes': False,
           'channel': 'FLIGHT',
           'format': 'bin',
           'msids': ['AOPCADMD'],
           'start': '2016:030:00:00:00.000',
           'stop': '2016:030:00:00:10.000',
           'tolerance': None,
           'url': 'http://t...cfa.harvard.edu/maude/...'}
}

Data structure

Each element of the dat['data'] has the following structure:

Key

Description

dtype

Data type of raw values

msid

MSID name from MAUDE

msid_index

MAUDE numerical index for MSID

n_values

Number of data values

raw_values

Raw telemetry values (different from values for state-code MSIDs)

times

Array of time stamps in CXC seconds (floating point time since 1998.0)

type_code

MAUDE data type code

values

Telemetry values (converted to state-code representation where applicable)

As another more complicated example, we take advantage of the MAUDE functionality to query multiple MSIDs and only return samples that are within a certain tolerance (in msec) of the first MSID in the list. This also sets alltimes=True in order to return the separate time stamps for each MSID instead of the common stamps from the first one:

>>> dat = maude.get_msids(['3TSCPOS', 'AOPCADMD'],
                         start='2016:030:00:00:00', stop='2016:030:00:05:00',
                         tolerance=2000, alltimes=True)
>>> dat
{'data': [{'dtype': <type 'numpy.float64'>,
           'flags': {'subset': False, 'tolerance': True},
           'msid': '3TSCPOS',
           'msid_index': 8274,
           'n_values': 9,
           'raw_values': array([ 75624.,  75624.,  75624.,  75624.,  75624.,  75624.,  75624.,
                                 75624.,  75624.]),
           'times': array([  5.70499291e+08,   5.70499324e+08,   5.70499357e+08,
                             5.70499390e+08,   5.70499422e+08,   5.70499455e+08,
                             5.70499488e+08,   5.70499521e+08,   5.70499554e+08]),
           'type_code': 6,
           'values': array([ 75624.,  75624.,  75624.,  75624.,  75624.,  75624.,  75624.,
                             75624.,  75624.])},
          {'dtype': <type 'numpy.int8'>,
           'flags': {'subset': False, 'tolerance': True},
           'msid': 'AOPCADMD',
           'msid_index': 2416,
           'n_values': 9,
           'raw_values': array([1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int8),
           'times': array([  5.70499291e+08,   5.70499324e+08,   5.70499356e+08,
                             5.70499389e+08,   5.70499422e+08,   5.70499455e+08,
                             5.70499488e+08,   5.70499520e+08,   5.70499553e+08]),
           'type_code': 0,
           'values': array(['NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT',
                            'NPNT'],
                           dtype='|S4')}],
 'format': 3,
 'n_msids': 2,
 'query': {'alltimes': True,
           'channel': 'FLIGHT',
           'format': 'bin',
           'msids': ['3TSCPOS', 'AOPCADMD'],
           'start': '2016:030:00:00:00.000',
           'stop': '2016:030:00:05:00.000',
           'tolerance': 2000,
           'url': 'http://t...cfa.harvard.edu/maude/...'}}

Full resolution data

Normally the MAUDE server will automatically return a sub-sampled version of the telemetry data if the number of points would exceed around 100k. However, if the allow_subset parameter of get_msids() is set to False then this function will prevent sub-sampling by doing multiple small queries. This has an overhead penalty because it may require multiple server requests to piece together the full query. In addition there is a fixed upper limit of 7 days for the query interval (stop - start time) in this mode. As an example:

>>> out = maude.get_msids('aoattqt1', '2016:001', '2016:003', allow_subset=False)
>>> len(out['data'][0]['values'])
168586

Caching

Caching of the request to the MAUDE server can be enabled by setting the configuration variable maude.conf.cache_msid_queries = True, as shown in the example below. This is useful when a request will be repeated verbatim and will cache the last 32 requests.

>>> maude.conf.cache_msid_queries = True
>>> start, stop = '2016:030:00:00:00', '2016:030:01:00:00'
>>> %time dat = maude.get_msids('AOPCADMD', start, stop, allow_subset=False)
CPU times: user 73 ms, sys: 13.6 ms, total: 86.6 ms
Wall time: 296 ms
>>> %time dat = maude.get_msids('AOPCADMD', start, stop, allow_subset=False)
CPU times: user 6.31 ms, sys: 774 µs, total: 7.09 ms
Wall time: 6.24 ms

Get Blobs

The get_blobs() function should be considered experimental at this point as it has not been used extensively. It takes the following arguments:

Arg

Description

start

Start time (any Chandra.Time format, default=NOW - 2 days)

stop

Stop time (any Chandra.Time format, default=NOW)

msids

Optional list of MSIDs (if given, only these MSIDs are returned)

channel

MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)

user

OCCweb user name

password

OCCweb password

It returns a Python dict structure with keys:

Key

Description

blobs

List of telemetry outputs, where each element is a dict.

flags

An integer. Flags returned by MAUDE

query

Input query values for reference

names

The list of MSIDs in the result list

types

The list of dtypes in the query list (corresponding to names)

The following statement retrieves the blobs, including all MSIDs, within a time range:

>>> blob = maude.get_blobs(start='2016:030:00:00:00', stop='2016:030:00:04:00')
>>> blob.keys()
dict_keys(['query', 'blobs', 'flags', 'names', 'types'])

Caching

Caching of the request to the MAUDE server can be enabled by setting the configuration variable maude.conf.cache_blob_queries = True. This is useful when a request will be repeated verbatim and will cache the last 32 requests.

Get Frames

The get_frames() takes the following arguments:

Arg

Description

start

Start time (any Chandra.Time format, default=NOW - 2 days)

stop

Stop time (any Chandra.Time format, default=NOW)

channel

MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)

format

MAUDE response format [json | bin] (default=bin)

user

OCCweb user name

password

OCCweb password

It returns a Python dict structure with four keys:

Key

Description

data

A dict of the form {‘f’: int, ‘frames’: []}, where each entry in result[‘frames’] is a dictionary that depends on the format:

  • If format == ‘json’: {‘t’: float, ‘bytes as hex’: str}

  • If format == ‘bin’: {‘t’: float, ‘bytes’: bytes}

query

Input query values for reference

The following statements retrieve the all VCDU frames, including all MSIDs, within a time range:

>>> frames = maude.get_frames(start='2016:030:00:00:00', stop='2016:030:00:04:00')
>>> frames.keys()
dict_keys(['data', 'query'])
>>> frames['data'].keys()
dict_keys(['frames', 'f'])
>>> frames['data']['frames'][0]['bytes'][:30]
b'A\x89\x9d\x89E\x80\xf5b|\x00\xff\xa0\xb4\x8br\x81p\x83{cG\x10\xac\x17\xa4\x16\xddBA@'
>>> frames = maude.get_frames(start='2016:030:00:00:00', stop='2016:030:00:04:00', format='json')
>>> frames.keys()
dict_keys(['data', 'query'])
>>> frames['data']['frames'][0]['bytes_as_hex'][:30]
'41 89 9D 89 45 80 F5 62 7C 00 '

Caching

Caching of the request to the MAUDE server can be enabled by setting the configuration variable maude.conf.cache_frame_queries = True. This is useful when a request will be repeated verbatim and will cache the last 32 requests.

Logging

To getting logging information, in particular some debug info that shows a bit of what is happening under the hood, do:

>>> maude.set_logger_level('DEBUG')  # for DEBUG
>>> out = maude.get_msids('aoattqt1', '2016:001', '2016:003', allow_subset=False)
get_msids: Using .netrc with user=taldcroft
get_msids_in_chunks: Chunked reading: max samples / major_frame = 32, chunk dt = 82000.0 secs
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016001120000000&tp=2016002040000000
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016002040000000&tp=2016002200000000
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016002200000000&tp=2016003120000000

API documentation

Functions

maude.maude.blobs_to_arrays(blobs)

Convenience method to convert raw blobs into a dictionary.

The output is a dictionary of the form:

{<msid1>: {'times': np.array(...),
           'values': np.array(...)},
 <msid2>: {'times': np.array(...),
           'values': np.array(...)},
 ...}

Example:

>>> import maude
>>> blobs = maude.get_blobs('2022:037:12:00:00.000', '2022:037:12:00:01.500',
...                         msids=['AOPCADMD', 'CVCMNCTR', 'AOATTQT1'])
>>> maude.blobs_to_arrays(blobs)
{'AOATTQT1': {'times': array([7.6053607e+08]),
              'values': array([-0.29576972])},
'AOPCADMD': {'times': array([7.6053607e+08]),
            'values': array(['NPNT'], dtype='<U4')},
'CVCMNCTR': {'times': array([7.60536069e+08, 7.60536070e+08, 7.60536070e+08, 7.60536070e+08,
                             7.60536070e+08, 7.60536071e+08]),
            'values': array([70, 71, 72, 73, 74, 75], dtype=uint8)}
}
Parameters:

blobs – dict Output of maude.get_blobs()

Returns:

dict

maude.maude.blobs_to_table(blobs, names, types, msid_offsets=None, **_)

Convenience method to convert raw blobs into an astropy.Table.

Optionally, some MSIDs can be shifted a fixed number of minor frames. NOTE: If any MSID is shifted, the resulting table is not truncated at the end, which causes the table to have missing MSIDs at the end (since they would be in trailing blobs).

Example usage:

>>> import maude
>>> from chandra_aca import maude_decom
>>> blobs = maude_decom.get_raw_aca_blobs(686111007, 686111009)
>>> maude.blobs_to_table(**blobs)[['TIME', 'CVCMJCTR', 'CVCMNCTR']]
<Table length=2>
     TIME     CVCMJCTR CVCMNCTR
   float64     uint32   uint8
------------- -------- --------
686111007.191     8580       96
686111008.216     8580      100

or:

>>> import maude
>>> blobs = maude.get_blobs(686111007, 686111008)
>>> maude.blobs_to_table(**blobs)[['TIME', 'CVCMJCTR', 'CVCMNCTR']]
<Table length=4>
     TIME     CVCMJCTR CVCMNCTR
   float64     uint32   uint8
------------- -------- --------
686111007.191     8580       96
686111007.448     8580       97
686111007.704     8580       98
 686111007.96     8580       99
Parameters:
  • blobs – dict A dictionary of the form {msid: value … } or the ‘blobs’ entry returned by maude.get_blobs

  • names – list A list of strings (like the ‘names’ entry returned by maude.get_blobs)

  • types – list A list of dtypes (like the ‘types’ entry returned by maude.get_blobs)

  • msid_offsets – dict. Optional A dictionary of the form {msid: int}, where int is the number of minor frames this msid is offset. E.g.: if the offset is 1, the values for this MSID are shifted backwards one frame.

Returns:

maude.maude.get_blobs(start=None, stop=None, msids=(), channel='FLIGHT', data_type=None, format='json', *, highrate=False, allpoints=False, include_calcs=False, include_raw_counts=False, nearest=False, user=None, password=None)

Get blob data for msids from the MAUDE server.

The time values are returned as a CXC seconds for downstream convenience.

If neither start nor stop are provided then the last available blob will be returned.

Details for many of the arguments below are found in the MAUDE documentation available at: https://occweb.cfa.harvard.edu/occweb/FOT/ground_systems/MAUDE/STARTHERE/MAUDE_Interface_Doc_1.0-Beta.pdf

Examples:

>>> maude.get_blobs('2022:037:12:00:00', '2022:037:12:00:01.500',
...                  msids=['aopcadmd', 'aoattqt1'])
{'blobs': [{'f': 0, 'time': 760536069.339, 'values': []},
        {'f': 0, 'time': 760536069.595, 'values': []},
        {'f': 0,
            'time': 760536069.851,
            'values': [{'i': 2688,
                        'n': 'AOATTQT1',
                        'tc': 6,
                        'v': '-0.29576971651476924',
                        'vc': -0.29576971651476924}]},
        {'f': 0,
            'time': 760536070.107,
            'values': [{'i': 2066,
                        'n': 'AOPCADMD',
                        'tc': 0,
                        'v': '1',
                        'vc': 'NPNT'}]},
        {'f': 0, 'time': 760536070.364, 'values': []},
        {'f': 0, 'time': 760536070.62, 'values': []}],
'f': 0,
'format': 0,
'names': ['TIME', 'AOPCADMD', 'AOATTQT1'],
'query': {'channel': 'FLIGHT',
        'msids': ['AOPCADMD', 'AOATTQT1'],
        'start': '2022:037:12:00:00.000',
        'stop': '2022:037:12:00:01.500',
        'url': 'http://telemetry.cfa.harvard.edu/maude/mrest/FLIGHT/blob.json?ts=2022037120000000&tp=2022037120001500'},
'types': [<class 'numpy.float64'>,
        <class 'numpy.int8'>,
        <class 'numpy.float64'>]}

# Get all ASVT VDS "truth" data during a simulation
>>> blobs = maude.get_blobs('2022:043:00:00:00', '2022:043:00:00:01',
...                         channel='TEST', data_type='VDS')
>>> blobs.keys()
dict_keys(['format', 'blobs', 'f', 'query', 'names', 'types'])
>>> len(blobs['blobs'])
15
>>> blobs['blobs'][0].keys()
dict_keys(['time', 'values', 'f'])
Parameters:
  • start – start time (any Chandra.Time format, default=mission start)

  • stop – stop time (any Chandra.Time format, default=NOW)

  • msids – str, list of str single MSID or list of MSIDs (default is to include all MSIDs in blobs)

  • channel – str MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)

  • data_type – str MAUDE data type [SC | MON | VDS] (default=SC for spacecraft)

  • format – str MAUDE response format [json | xml | bin] (default=json)

  • allpoints – bool MAUDE ap parameter: return all points within the specified time range, up to the configured limit on the number of points and/or bytes returned. (default=False)

  • highrate – bool MAUDE hr parameter: include high rate data (default=False)

  • include_calcs – bool, MAUDE icalcs parameter: include calc blobs in result (default=False)

  • include_raw_counts – bool MAUDE ircts parameter: include raw counts blobs in result (default=False)

  • nearest – bool MAUDE nearest parameter: return the item nearest the specified time (default=False)

  • user – OCCweb user name (default=None)

  • password – OCCweb password (default=None)

Returns:

dict of query outputs: {‘blobs’: [], ‘f’: 0, ‘query’: dict, ‘names’: [], ‘types’: np.array}

where ‘blobs’ contains the output from MAUDE, ‘query’ contains the input parameters to the function, and ‘names’ and ‘types’ have the dtypes of the resulting MSIDs. ‘names’ does not include all msids given as input to the function, but only the ones actually in the blobs. ‘f’ stands for ‘flags’ and is an integer returned by MAUDE.

maude.maude.get_frames(start=None, stop=None, channel='FLIGHT', format='bin', *, data_type=None, nearest=False, user=None, password=None)

Get 1025-byte VCDU frames from the MAUDE server.

The time values are converted to CXC seconds for downstream convenience.

If neither start nor stop are provided then the last available frame will be returned.

Details for many of the arguments below are found in the MAUDE documentation available at: https://occweb.cfa.harvard.edu/occweb/FOT/ground_systems/MAUDE/STARTHERE/MAUDE_Interface_Doc_1.0-Beta.pdf

Parameters:
  • start – start time (any Chandra.Time format, default=mission start)

  • stop – stop time (any Chandra.Time format, default=NOW)

  • channel – str MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)

  • format – str MAUDE response format [json | bin] (default=bin)

  • data_type – str MAUDE data type [SC | MON | VDS] (default=SC for spacecraft)

  • nearest – bool MAUDE nearest parameter: return the item nearest the specified time (default=False)

  • user – str OCCweb user name (default=None)

  • password – str OCCweb password (default=None)

Returns:

dict of query outputs. {‘query’: {}, ‘data’: {‘f’: int, ‘frames’: []}} where each frame in result[‘data’][‘frames’] is of the form: - {‘t’: float, ‘bytes’: bytes} if format == ‘bin’ - {‘t’: float, ‘bytes_as_hex’: bytes} if format == ‘json’ (‘f’ stands for ‘flags’ and is an integer returned by MAUDE, ‘t’ stands for time, and is the time of the frame in CXC seconds)

maude.maude.get_last_backorbit_date(msids: str | list[str]) str

Get last date of backorbit telemetry in MAUDE for a given MSID or list of MSIDs.

This function is useful to avoid accessing real-time telemetry from a current or recent comm pass. Real-time telemetry can commonly have drop-outs or data corruption, and may not be suitable for some types of analysis or processing.

The returned date is the earliest date of the last 5-min statistic sample for each MSID in the list of MSID(s). It is then guaranteed that all MSIDs have full-resolution SSR dump telemetry up to this date (at least within 5 minutes). This relies on a feature of the MAUDE server that the 5-minute statistic telemetry is only computed from backorbit (SSR dump) data.

Warning

The MAUDE telemetry ingest process is multi-threaded, so checking the last date of telemetry for one MSID is not a strictly reliable way to know the status of other MSIDs. But if you just need a ballpark estimate for non-production work, you can use the VCDU count MSID “CCSDSVCD”, which is available in every format.

Parameters

msidsstr, list of str

List of MSIDs to check for last telemetry date

Returns

date_laststr

Last date of telemetry for backorbit MAUDE data source (CXC date format)

maude.maude.get_maude_sysinfo(user=None, password=None)

Get system info about the MAUDE server.

Parameters:
  • user – OCCweb user name (default=None)

  • password – OCCweb password (default=None)

Returns:

dict of system information

maude.maude.get_msids(msids, start=None, stop=None, *, tolerance=None, alltimes=False, allpoints=False, highrate=False, nearest=False, channel='FLIGHT', format='bin', user=None, password=None, allow_subset=True)

Get MSID data for msids from the MAUDE server.

For production use format arg should be bin. In this case the binary values are parsed and the returned data structure has data values that are converted to the correct data type, with state-code translations applied as required. The time values are returned as a CXC seconds for downstream convenience.

Normally the MAUDE server will automatically return a sub-sampled version of the telemetry data if the number of points would exceed around 100k. However, if allow_subset is set to False then this function will prevent sub-sampling by doing multiple small queries. This has an overhead penalty because it may require multiple server requests to piece together the full query.

If neither start nor stop are provided then the last available value will be returned.

Details for many of the arguments below are found in the MAUDE documentation available at: https://occweb.cfa.harvard.edu/occweb/FOT/ground_systems/MAUDE/STARTHERE/MAUDE_Interface_Doc_1.0-Beta.pdf

Parameters:
  • msids – str, list of str single MSID or list of MSIDs The MSID or MSIDs may also be a “stat” MSIDs of the form STAT_period_stat_msidname where period is 5MIN or 1DAY, stat is MIN, MAX, MEAN, or COUNT, and msidname is an MSID as listed in MSIDIDX.txt. See “Aggregate MSID Statistics” in the MAUDE documentation.

  • start – start time (any Chandra.Time format, default=mission start)

  • stop – stop time inclusive (any Chandra.Time format, default=NOW)

  • tolerance – int, None MAUDE tolerance parameter: provide for selection of MSID query updates on the basis of time proximity to the updates of the “primary” MSID (in millisec, default=None)

  • alltimes – bool MAUDE alltimes parameter: if True a separate set of times is provided for each set of MSID updates (default=False)

  • allpoints – bool MAUDE ap parameter: return all points within the specified time range, up to the configured limit on the number of points and/or bytes returned. (default=False)

  • highrate – bool MAUDE hr parameter: include high rate data (default=False)

  • nearest – bool MAUDE nearest parameter: return the item nearest the specified time (default=False)

  • channel – str MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)

  • format – str MAUDE response format [json | xml | bin] (default=bin)

  • user – str, None OCCweb user name (default=None)

  • password – str, None OCCweb password (default=None)

  • allow_subset – bool Allow subsetted data, otherwise require full resolution (default=True)

Returns:

dict of query outputs, format depends on input params

maude.maude.get_url(query_type=None, start=None, stop=None, msids=(), channel='FLIGHT', format='bin', **kwargs)

Assemble the URL the request from MAUDE.

Parameters:
  • query_type – str [‘blob’, ‘frame’, ‘msid’]

  • start – start time (any Chandra.Time format)

  • stop – stop time (any Chandra.Time format)

  • msids – list of MSIDs

  • channel – MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)

  • format – MAUDE response format [json | bin] (default=bin)

  • kwargs – Other valid MAUDE parameters (this is not validated)

Returns:

str

maude.maude.set_logger_level(level)

Set logger level (and all handler levels) to level.

Parameters:

level – log level (10=DEBUG, 15=VERBOSE, 20=INFO, 30=WARNING, 40=ERROR, 50=CRITICAL)

Configuration

Configuration for maude.

See https://docs.astropy.org/en/stable/config/index.html#customizing-config-location-in-affiliated-packages and https://github.com/astropy/astropy/issues/12960.

class maude.config.Conf

Configuration parameters for maude.

cache_blob_queries

Cache results blob queries to the MAUDE server. This caches the last 32 queries of any type.

cache_frame_queries

Cache results frame queries to the MAUDE server. This caches the last 32 queries of any type.

cache_msid_queries

Cache results MSID queries to the MAUDE server. This caches the last 32 queries of any type.

class maude.config.ConfigItem(defaultvalue='', description=None, cfgtype=None, module=None, aliases=None)
rootname = 'maude'

Rootname sets the base path for all config files.