Python interface to MAUDE telemetry server¶
The maude
package provides a low-level Python interface to the MAUDE Chandra telemetry
server. For more information on MAUDE check its webpage, which includes extensive
documentation.
Setup for authentication¶
In order to use maude
you must have authentication credentials (username and password)
to access OCCweb. One can provide those credentials manually to the
get_msids()
function call, but this gets tiresome.
The preferred method to use this from a secure machine is to edit the file .netrc
in
your home directory and put in your OCCweb credentials.
IMPORTANT: make sure the file is readable only by you!
chmod og-rwx ~/.netrc
Once you have done that, add these three lines. If there are already other machines defined you need a blank line between the machine configs.
machine occweb
login your-occweb-username
password your-occweb-password
Usage¶
Get MSIDs¶
The public interface to the maude
package is a essentially a single function
get_msids()
which takes the following arguments:
Arg
Description
msids
Single MSID or list of MSIDs
start
Start time (any Chandra.Time format, default=NOW - 2 days)
stop
Stop time (any Chandra.Time format, default=NOW)
tolerance
MAUDE
tolerance
parameter (millisec, default=None)alltimes
MAUDE
alltimes
parameter (default=False)channel
MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
format
MAUDE response format [json | xml | bin] (default=bin)
user
OCCweb user name
password
OCCweb password
As a first example we get the PCAD mode over a short period of time in 2016:
>>> import maude
>>> dat = maude.get_msids('AOPCADMD', start='2016:030:00:00:00', stop='2016:030:00:00:10')
This returns a Python dict
structure with four keys:
Key
Description
data
List of telemetry outputs, where each element is a
dict
.
format
MAUDE binary data format (you normally don’t care about this)
n_msids
Number of MSIDs in the query list
query
Input query values for reference
For the query above, the value of dat
is:
>>> dat
{'data': [{'dtype': <type 'numpy.int8'>,
'flags': {'subset': False, 'tolerance': False},
'msid': 'AOPCADMD',
'msid_index': 2416,
'n_values': 10,
'raw_values': array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int8),
'times': array([ 5.70499268e+08, 5.70499269e+08, 5.70499270e+08,
5.70499271e+08, 5.70499272e+08, 5.70499273e+08,
5.70499274e+08, 5.70499275e+08, 5.70499276e+08,
5.70499277e+08]),
'type_code': 0,
'values': array(['NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT',
'NPNT', 'NPNT'],
dtype='|S4')}],
'format': 1,
'n_msids': 1,
'query': {'alltimes': False,
'channel': 'FLIGHT',
'format': 'bin',
'msids': ['AOPCADMD'],
'start': '2016:030:00:00:00.000',
'stop': '2016:030:00:00:10.000',
'tolerance': None,
'url': 'http://t...cfa.harvard.edu/maude/...'}
}
Data structure¶
Each element of the dat['data']
has the following structure:
Key
Description
dtype
Data type of raw values
msid
MSID name from MAUDE
msid_index
MAUDE numerical index for MSID
n_values
Number of data values
raw_values
Raw telemetry values (different from values for state-code MSIDs)
times
Array of time stamps in CXC seconds (floating point time since 1998.0)
type_code
MAUDE data type code
values
Telemetry values (converted to state-code representation where applicable)
As another more complicated example, we take advantage of the MAUDE functionality to query
multiple MSIDs and only return samples that are within a certain tolerance
(in msec)
of the first MSID in the list. This also sets alltimes=True
in order to return
the separate time stamps for each MSID instead of the common stamps from the first one:
>>> dat = maude.get_msids(['3TSCPOS', 'AOPCADMD'],
start='2016:030:00:00:00', stop='2016:030:00:05:00',
tolerance=2000, alltimes=True)
>>> dat
{'data': [{'dtype': <type 'numpy.float64'>,
'flags': {'subset': False, 'tolerance': True},
'msid': '3TSCPOS',
'msid_index': 8274,
'n_values': 9,
'raw_values': array([ 75624., 75624., 75624., 75624., 75624., 75624., 75624.,
75624., 75624.]),
'times': array([ 5.70499291e+08, 5.70499324e+08, 5.70499357e+08,
5.70499390e+08, 5.70499422e+08, 5.70499455e+08,
5.70499488e+08, 5.70499521e+08, 5.70499554e+08]),
'type_code': 6,
'values': array([ 75624., 75624., 75624., 75624., 75624., 75624., 75624.,
75624., 75624.])},
{'dtype': <type 'numpy.int8'>,
'flags': {'subset': False, 'tolerance': True},
'msid': 'AOPCADMD',
'msid_index': 2416,
'n_values': 9,
'raw_values': array([1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int8),
'times': array([ 5.70499291e+08, 5.70499324e+08, 5.70499356e+08,
5.70499389e+08, 5.70499422e+08, 5.70499455e+08,
5.70499488e+08, 5.70499520e+08, 5.70499553e+08]),
'type_code': 0,
'values': array(['NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT',
'NPNT'],
dtype='|S4')}],
'format': 3,
'n_msids': 2,
'query': {'alltimes': True,
'channel': 'FLIGHT',
'format': 'bin',
'msids': ['3TSCPOS', 'AOPCADMD'],
'start': '2016:030:00:00:00.000',
'stop': '2016:030:00:05:00.000',
'tolerance': 2000,
'url': 'http://t...cfa.harvard.edu/maude/...'}}
Full resolution data¶
Normally the MAUDE server will automatically return a sub-sampled version of the telemetry
data if the number of points would exceed around 100k. However, if the allow_subset
parameter of get_msids()
is set to False
then this function will prevent
sub-sampling by doing multiple small queries. This has an overhead penalty because it
may require multiple server requests to piece together the full query. In addition there
is a fixed upper limit of 7 days for the query interval (stop - start time) in this mode.
As an example:
>>> out = maude.get_msids('aoattqt1', '2016:001', '2016:003', allow_subset=False)
>>> len(out['data'][0]['values'])
168586
Caching¶
Caching of the request to the MAUDE server can be enabled by setting the
configuration variable maude.conf.cache_msid_queries = True
, as shown in the
example below. This is useful when a request will be repeated verbatim and will
cache the last 32 requests.
>>> maude.conf.cache_msid_queries = True
>>> start, stop = '2016:030:00:00:00', '2016:030:01:00:00'
>>> %time dat = maude.get_msids('AOPCADMD', start, stop, allow_subset=False)
CPU times: user 73 ms, sys: 13.6 ms, total: 86.6 ms
Wall time: 296 ms
>>> %time dat = maude.get_msids('AOPCADMD', start, stop, allow_subset=False)
CPU times: user 6.31 ms, sys: 774 µs, total: 7.09 ms
Wall time: 6.24 ms
Get Blobs¶
The get_blobs()
function should be considered experimental at this point as it
has not been used extensively. It takes the following arguments:
Arg
Description
start
Start time (any Chandra.Time format, default=NOW - 2 days)
stop
Stop time (any Chandra.Time format, default=NOW)
msids
Optional list of MSIDs (if given, only these MSIDs are returned)
channel
MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
user
OCCweb user name
password
OCCweb password
It returns a Python dict
structure with keys:
Key
Description
blobs
List of telemetry outputs, where each element is a
dict
.
flags
An integer. Flags returned by MAUDE
query
Input query values for reference
names
The list of MSIDs in the result list
types
The list of dtypes in the query list (corresponding to names)
The following statement retrieves the blobs, including all MSIDs, within a time range:
>>> blob = maude.get_blobs(start='2016:030:00:00:00', stop='2016:030:00:04:00')
>>> blob.keys()
dict_keys(['query', 'blobs', 'flags', 'names', 'types'])
Caching¶
Caching of the request to the MAUDE server can be enabled by setting the configuration
variable maude.conf.cache_blob_queries = True
. This is useful when a request will be repeated
verbatim and will cache the last 32 requests.
Get Frames¶
The get_frames()
takes the following arguments:
Arg
Description
start
Start time (any Chandra.Time format, default=NOW - 2 days)
stop
Stop time (any Chandra.Time format, default=NOW)
channel
MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
format
MAUDE response format [json | bin] (default=bin)
user
OCCweb user name
password
OCCweb password
It returns a Python dict
structure with four keys:
Key
Description
data
A dict of the form {‘f’: int, ‘frames’: []}, where each entry in result[‘frames’] is a dictionary that depends on the format:
If format == ‘json’: {‘t’: float, ‘bytes as hex’: str}
If format == ‘bin’: {‘t’: float, ‘bytes’: bytes}
query
Input query values for reference
The following statements retrieve the all VCDU frames, including all MSIDs, within a time range:
>>> frames = maude.get_frames(start='2016:030:00:00:00', stop='2016:030:00:04:00')
>>> frames.keys()
dict_keys(['data', 'query'])
>>> frames['data'].keys()
dict_keys(['frames', 'f'])
>>> frames['data']['frames'][0]['bytes'][:30]
b'A\x89\x9d\x89E\x80\xf5b|\x00\xff\xa0\xb4\x8br\x81p\x83{cG\x10\xac\x17\xa4\x16\xddBA@'
>>> frames = maude.get_frames(start='2016:030:00:00:00', stop='2016:030:00:04:00', format='json')
>>> frames.keys()
dict_keys(['data', 'query'])
>>> frames['data']['frames'][0]['bytes_as_hex'][:30]
'41 89 9D 89 45 80 F5 62 7C 00 '
Caching¶
Caching of the request to the MAUDE server can be enabled by setting the
configuration variable maude.conf.cache_frame_queries = True
. This is useful
when a request will be repeated verbatim and will cache the last 32 requests.
Logging¶
To getting logging information, in particular some debug info that shows a bit of what is happening under the hood, do:
>>> maude.set_logger_level('DEBUG') # for DEBUG
>>> out = maude.get_msids('aoattqt1', '2016:001', '2016:003', allow_subset=False)
get_msids: Using .netrc with user=taldcroft
get_msids_in_chunks: Chunked reading: max samples / major_frame = 32, chunk dt = 82000.0 secs
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016001120000000&tp=2016002040000000
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016002040000000&tp=2016002200000000
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016002200000000&tp=2016003120000000
API documentation¶
Functions¶
- maude.maude.blobs_to_arrays(blobs)¶
Convenience method to convert raw blobs into a dictionary.
The output is a dictionary of the form:
{<msid1>: {'times': np.array(...), 'values': np.array(...)}, <msid2>: {'times': np.array(...), 'values': np.array(...)}, ...}
Example:
>>> import maude >>> blobs = maude.get_blobs('2022:037:12:00:00.000', '2022:037:12:00:01.500', ... msids=['AOPCADMD', 'CVCMNCTR', 'AOATTQT1']) >>> maude.blobs_to_arrays(blobs) {'AOATTQT1': {'times': array([7.6053607e+08]), 'values': array([-0.29576972])}, 'AOPCADMD': {'times': array([7.6053607e+08]), 'values': array(['NPNT'], dtype='<U4')}, 'CVCMNCTR': {'times': array([7.60536069e+08, 7.60536070e+08, 7.60536070e+08, 7.60536070e+08, 7.60536070e+08, 7.60536071e+08]), 'values': array([70, 71, 72, 73, 74, 75], dtype=uint8)} }
- Parameters:
blobs – dict Output of maude.get_blobs()
- Returns:
dict
- maude.maude.blobs_to_table(blobs, names, types, msid_offsets=None, **_)¶
Convenience method to convert raw blobs into an astropy.Table.
Optionally, some MSIDs can be shifted a fixed number of minor frames. NOTE: If any MSID is shifted, the resulting table is not truncated at the end, which causes the table to have missing MSIDs at the end (since they would be in trailing blobs).
Example usage:
>>> import maude >>> from chandra_aca import maude_decom >>> blobs = maude_decom.get_raw_aca_blobs(686111007, 686111009) >>> maude.blobs_to_table(**blobs)[['TIME', 'CVCMJCTR', 'CVCMNCTR']] <Table length=2> TIME CVCMJCTR CVCMNCTR float64 uint32 uint8 ------------- -------- -------- 686111007.191 8580 96 686111008.216 8580 100
or:
>>> import maude >>> blobs = maude.get_blobs(686111007, 686111008) >>> maude.blobs_to_table(**blobs)[['TIME', 'CVCMJCTR', 'CVCMNCTR']] <Table length=4> TIME CVCMJCTR CVCMNCTR float64 uint32 uint8 ------------- -------- -------- 686111007.191 8580 96 686111007.448 8580 97 686111007.704 8580 98 686111007.96 8580 99
- Parameters:
blobs – dict A dictionary of the form {msid: value … } or the ‘blobs’ entry returned by maude.get_blobs
names – list A list of strings (like the ‘names’ entry returned by maude.get_blobs)
types – list A list of dtypes (like the ‘types’ entry returned by maude.get_blobs)
msid_offsets – dict. Optional A dictionary of the form {msid: int}, where int is the number of minor frames this msid is offset. E.g.: if the offset is 1, the values for this MSID are shifted backwards one frame.
- Returns:
- maude.maude.get_blobs(start=None, stop=None, msids=(), channel='FLIGHT', data_type=None, format='json', *, highrate=False, allpoints=False, include_calcs=False, include_raw_counts=False, nearest=False, user=None, password=None)¶
Get blob data for
msids
from the MAUDE server.The time values are returned as a CXC seconds for downstream convenience.
If neither
start
norstop
are provided then the last available blob will be returned.Details for many of the arguments below are found in the MAUDE documentation available at: https://occweb.cfa.harvard.edu/occweb/FOT/ground_systems/MAUDE/STARTHERE/MAUDE_Interface_Doc_1.0-Beta.pdf
Examples:
>>> maude.get_blobs('2022:037:12:00:00', '2022:037:12:00:01.500', ... msids=['aopcadmd', 'aoattqt1']) {'blobs': [{'f': 0, 'time': 760536069.339, 'values': []}, {'f': 0, 'time': 760536069.595, 'values': []}, {'f': 0, 'time': 760536069.851, 'values': [{'i': 2688, 'n': 'AOATTQT1', 'tc': 6, 'v': '-0.29576971651476924', 'vc': -0.29576971651476924}]}, {'f': 0, 'time': 760536070.107, 'values': [{'i': 2066, 'n': 'AOPCADMD', 'tc': 0, 'v': '1', 'vc': 'NPNT'}]}, {'f': 0, 'time': 760536070.364, 'values': []}, {'f': 0, 'time': 760536070.62, 'values': []}], 'f': 0, 'format': 0, 'names': ['TIME', 'AOPCADMD', 'AOATTQT1'], 'query': {'channel': 'FLIGHT', 'msids': ['AOPCADMD', 'AOATTQT1'], 'start': '2022:037:12:00:00.000', 'stop': '2022:037:12:00:01.500', 'url': 'http://telemetry.cfa.harvard.edu/maude/mrest/FLIGHT/blob.json?ts=2022037120000000&tp=2022037120001500'}, 'types': [<class 'numpy.float64'>, <class 'numpy.int8'>, <class 'numpy.float64'>]} # Get all ASVT VDS "truth" data during a simulation >>> blobs = maude.get_blobs('2022:043:00:00:00', '2022:043:00:00:01', ... channel='TEST', data_type='VDS') >>> blobs.keys() dict_keys(['format', 'blobs', 'f', 'query', 'names', 'types']) >>> len(blobs['blobs']) 15 >>> blobs['blobs'][0].keys() dict_keys(['time', 'values', 'f'])
- Parameters:
start – start time (any Chandra.Time format, default=mission start)
stop – stop time (any Chandra.Time format, default=NOW)
msids – str, list of str single MSID or list of MSIDs (default is to include all MSIDs in blobs)
channel – str MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
data_type – str MAUDE data type [SC | MON | VDS] (default=SC for spacecraft)
format – str MAUDE response format [json | xml | bin] (default=json)
allpoints – bool MAUDE
ap
parameter: return all points within the specified time range, up to the configured limit on the number of points and/or bytes returned. (default=False)highrate – bool MAUDE
hr
parameter: include high rate data (default=False)include_calcs – bool, MAUDE
icalcs
parameter: include calc blobs in result (default=False)include_raw_counts – bool MAUDE
ircts
parameter: include raw counts blobs in result (default=False)nearest – bool MAUDE
nearest
parameter: return the item nearest the specified time (default=False)user – OCCweb user name (default=None)
password – OCCweb password (default=None)
- Returns:
dict of query outputs: {‘blobs’: [], ‘f’: 0, ‘query’: dict, ‘names’: [], ‘types’: np.array}
where ‘blobs’ contains the output from MAUDE, ‘query’ contains the input parameters to the function, and ‘names’ and ‘types’ have the dtypes of the resulting MSIDs. ‘names’ does not include all msids given as input to the function, but only the ones actually in the blobs. ‘f’ stands for ‘flags’ and is an integer returned by MAUDE.
- maude.maude.get_frames(start=None, stop=None, channel='FLIGHT', format='bin', *, data_type=None, nearest=False, user=None, password=None)¶
Get 1025-byte VCDU frames from the MAUDE server.
The time values are converted to CXC seconds for downstream convenience.
If neither
start
norstop
are provided then the last available frame will be returned.Details for many of the arguments below are found in the MAUDE documentation available at: https://occweb.cfa.harvard.edu/occweb/FOT/ground_systems/MAUDE/STARTHERE/MAUDE_Interface_Doc_1.0-Beta.pdf
- Parameters:
start – start time (any Chandra.Time format, default=mission start)
stop – stop time (any Chandra.Time format, default=NOW)
channel – str MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
format – str MAUDE response format [json | bin] (default=bin)
data_type – str MAUDE data type [SC | MON | VDS] (default=SC for spacecraft)
nearest – bool MAUDE
nearest
parameter: return the item nearest the specified time (default=False)user – str OCCweb user name (default=None)
password – str OCCweb password (default=None)
- Returns:
dict of query outputs. {‘query’: {}, ‘data’: {‘f’: int, ‘frames’: []}} where each frame in result[‘data’][‘frames’] is of the form: - {‘t’: float, ‘bytes’: bytes} if format == ‘bin’ - {‘t’: float, ‘bytes_as_hex’: bytes} if format == ‘json’ (‘f’ stands for ‘flags’ and is an integer returned by MAUDE, ‘t’ stands for time, and is the time of the frame in CXC seconds)
- maude.maude.get_last_backorbit_date(msids: str | list[str]) str ¶
Get last date of backorbit telemetry in MAUDE for a given MSID or list of MSIDs.
This function is useful to avoid accessing real-time telemetry from a current or recent comm pass. Real-time telemetry can commonly have drop-outs or data corruption, and may not be suitable for some types of analysis or processing.
The returned date is the earliest date of the last 5-min statistic sample for each MSID in the list of MSID(s). It is then guaranteed that all MSIDs have full-resolution SSR dump telemetry up to this date (at least within 5 minutes). This relies on a feature of the MAUDE server that the 5-minute statistic telemetry is only computed from backorbit (SSR dump) data.
Warning
The MAUDE telemetry ingest process is multi-threaded, so checking the last date of telemetry for one MSID is not a strictly reliable way to know the status of other MSIDs. But if you just need a ballpark estimate for non-production work, you can use the VCDU count MSID “CCSDSVCD”, which is available in every format.
Parameters¶
- msidsstr, list of str
List of MSIDs to check for last telemetry date
Returns¶
- date_laststr
Last date of telemetry for backorbit MAUDE data source (CXC date format)
- maude.maude.get_maude_sysinfo(user=None, password=None)¶
Get system info about the MAUDE server.
- Parameters:
user – OCCweb user name (default=None)
password – OCCweb password (default=None)
- Returns:
dict of system information
- maude.maude.get_msids(msids, start=None, stop=None, *, tolerance=None, alltimes=False, allpoints=False, highrate=False, nearest=False, channel='FLIGHT', format='bin', user=None, password=None, allow_subset=True)¶
Get MSID data for
msids
from the MAUDE server.For production use
format
arg should bebin
. In this case the binary values are parsed and the returned data structure has data values that are converted to the correct data type, with state-code translations applied as required. The time values are returned as a CXC seconds for downstream convenience.Normally the MAUDE server will automatically return a sub-sampled version of the telemetry data if the number of points would exceed around 100k. However, if
allow_subset
is set to False then this function will prevent sub-sampling by doing multiple small queries. This has an overhead penalty because it may require multiple server requests to piece together the full query.If neither
start
norstop
are provided then the last available value will be returned.Details for many of the arguments below are found in the MAUDE documentation available at: https://occweb.cfa.harvard.edu/occweb/FOT/ground_systems/MAUDE/STARTHERE/MAUDE_Interface_Doc_1.0-Beta.pdf
- Parameters:
msids – str, list of str single MSID or list of MSIDs The MSID or MSIDs may also be a “stat” MSIDs of the form STAT_period_stat_msidname where
period
is 5MIN or 1DAY,stat
is MIN, MAX, MEAN, or COUNT, andmsidname
is an MSID as listed in MSIDIDX.txt. See “Aggregate MSID Statistics” in the MAUDE documentation.start – start time (any Chandra.Time format, default=mission start)
stop – stop time inclusive (any Chandra.Time format, default=NOW)
tolerance – int, None MAUDE
tolerance
parameter: provide for selection of MSID query updates on the basis of time proximity to the updates of the “primary” MSID (in millisec, default=None)alltimes – bool MAUDE
alltimes
parameter: if True a separate set of times is provided for each set of MSID updates (default=False)allpoints – bool MAUDE
ap
parameter: return all points within the specified time range, up to the configured limit on the number of points and/or bytes returned. (default=False)highrate – bool MAUDE
hr
parameter: include high rate data (default=False)nearest – bool MAUDE
nearest
parameter: return the item nearest the specified time (default=False)channel – str MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
format – str MAUDE response format [json | xml | bin] (default=bin)
user – str, None OCCweb user name (default=None)
password – str, None OCCweb password (default=None)
allow_subset – bool Allow subsetted data, otherwise require full resolution (default=True)
- Returns:
dict of query outputs, format depends on input params
- maude.maude.get_url(query_type=None, start=None, stop=None, msids=(), channel='FLIGHT', format='bin', **kwargs)¶
Assemble the URL the request from MAUDE.
- Parameters:
query_type – str [‘blob’, ‘frame’, ‘msid’]
start – start time (any Chandra.Time format)
stop – stop time (any Chandra.Time format)
msids – list of MSIDs
channel – MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
format – MAUDE response format [json | bin] (default=bin)
kwargs – Other valid MAUDE parameters (this is not validated)
- Returns:
str
- maude.maude.set_logger_level(level)¶
Set logger level (and all handler levels) to
level
.- Parameters:
level – log level (10=DEBUG, 15=VERBOSE, 20=INFO, 30=WARNING, 40=ERROR, 50=CRITICAL)
Configuration¶
Configuration for maude.
See https://docs.astropy.org/en/stable/config/index.html#customizing-config-location-in-affiliated-packages and https://github.com/astropy/astropy/issues/12960.
- class maude.config.Conf¶
Configuration parameters for maude.
- cache_blob_queries¶
Cache results blob queries to the MAUDE server. This caches the last 32 queries of any type.
- cache_frame_queries¶
Cache results frame queries to the MAUDE server. This caches the last 32 queries of any type.
- cache_msid_queries¶
Cache results MSID queries to the MAUDE server. This caches the last 32 queries of any type.