.. maude documentation master file, created by
sphinx-quickstart on Fri Apr 8 12:18:03 2016.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
.. _MAUDE: https://occweb.cfa.harvard.edu/twiki/bin/view/Software/MAUDE/WebHome
Python interface to MAUDE telemetry server
==========================================
The ``maude`` package provides a low-level Python interface to the MAUDE_ Chandra telemetry
server. For more information on MAUDE_ check its webpage, which includes extensive
documentation.
Setup for authentication
------------------------
In order to use ``maude`` you must have authentication credentials (username and password)
to access OCCweb. One can provide those credentials manually to the
:func:`~maude.maude.get_msids` function call, but this gets tiresome.
The preferred method to use this from a secure machine is to edit the file ``.netrc`` in
your home directory and put in your OCCweb credentials.
**IMPORTANT**: make sure the file is readable only by you!
::
chmod og-rwx ~/.netrc
Once you have done that, add these three lines. If there are already
other machines defined you need a blank line between the machine configs.
::
machine occweb
login your-occweb-username
password your-occweb-password
Usage
-----
Get MSIDs
^^^^^^^^^
The public interface to the ``maude`` package is a essentially a single function
:func:`~maude.maude.get_msids` which takes the following arguments:
========== =================================================================
Arg Description
========== =================================================================
msids Single MSID or list of MSIDs
start Start time (any Chandra.Time format, default=NOW - 2 days)
stop Stop time (any Chandra.Time format, default=NOW)
tolerance MAUDE ``tolerance`` parameter (millisec, default=None)
alltimes MAUDE ``alltimes`` parameter (default=False)
channel MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
format MAUDE response format [json | xml | bin] (default=bin)
user OCCweb user name
password OCCweb password
========== =================================================================
As a first example we get the PCAD mode over a short period of time in 2016::
>>> import maude
>>> dat = maude.get_msids('AOPCADMD', start='2016:030:00:00:00', stop='2016:030:00:00:10')
This returns a Python ``dict`` structure with four keys:
=========== ==================================================================
Key Description
=========== ==================================================================
``data`` List of telemetry outputs, where each element is a ``dict``.
``format`` MAUDE binary data format (you normally don't care about this)
``n_msids`` Number of MSIDs in the query list
``query`` Input query values for reference
=========== ==================================================================
For the query above, the value of ``dat`` is::
>>> dat
{'data': [{'dtype': <type 'numpy.int8'>,
'flags': {'subset': False, 'tolerance': False},
'msid': 'AOPCADMD',
'msid_index': 2416,
'n_values': 10,
'raw_values': array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int8),
'times': array([ 5.70499268e+08, 5.70499269e+08, 5.70499270e+08,
5.70499271e+08, 5.70499272e+08, 5.70499273e+08,
5.70499274e+08, 5.70499275e+08, 5.70499276e+08,
5.70499277e+08]),
'type_code': 0,
'values': array(['NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT',
'NPNT', 'NPNT'],
dtype='|S4')}],
'format': 1,
'n_msids': 1,
'query': {'alltimes': False,
'channel': 'FLIGHT',
'format': 'bin',
'msids': ['AOPCADMD'],
'start': '2016:030:00:00:00.000',
'stop': '2016:030:00:00:10.000',
'tolerance': None,
'url': 'http://t...cfa.harvard.edu/maude/...'}
}
Data structure
""""""""""""""
Each element of the ``dat['data']`` has the following structure:
=========== ==========================================================================
Key Description
=========== ==========================================================================
dtype Data type of raw values
msid MSID name from MAUDE
msid_index MAUDE numerical index for MSID
n_values Number of data values
raw_values Raw telemetry values (different from values for state-code MSIDs)
times Array of time stamps in CXC seconds (floating point time since 1998.0)
type_code MAUDE data type code
values Telemetry values (converted to state-code representation where applicable)
=========== ==========================================================================
As another more complicated example, we take advantage of the MAUDE functionality to query
multiple MSIDs and only return samples that are within a certain ``tolerance`` (in msec)
of the first MSID in the list. This also sets ``alltimes=True`` in order to return
the separate time stamps for each MSID instead of the common stamps from the first one::
>>> dat = maude.get_msids(['3TSCPOS', 'AOPCADMD'],
start='2016:030:00:00:00', stop='2016:030:00:05:00',
tolerance=2000, alltimes=True)
>>> dat
{'data': [{'dtype': <type 'numpy.float64'>,
'flags': {'subset': False, 'tolerance': True},
'msid': '3TSCPOS',
'msid_index': 8274,
'n_values': 9,
'raw_values': array([ 75624., 75624., 75624., 75624., 75624., 75624., 75624.,
75624., 75624.]),
'times': array([ 5.70499291e+08, 5.70499324e+08, 5.70499357e+08,
5.70499390e+08, 5.70499422e+08, 5.70499455e+08,
5.70499488e+08, 5.70499521e+08, 5.70499554e+08]),
'type_code': 6,
'values': array([ 75624., 75624., 75624., 75624., 75624., 75624., 75624.,
75624., 75624.])},
{'dtype': <type 'numpy.int8'>,
'flags': {'subset': False, 'tolerance': True},
'msid': 'AOPCADMD',
'msid_index': 2416,
'n_values': 9,
'raw_values': array([1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int8),
'times': array([ 5.70499291e+08, 5.70499324e+08, 5.70499356e+08,
5.70499389e+08, 5.70499422e+08, 5.70499455e+08,
5.70499488e+08, 5.70499520e+08, 5.70499553e+08]),
'type_code': 0,
'values': array(['NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT', 'NPNT',
'NPNT'],
dtype='|S4')}],
'format': 3,
'n_msids': 2,
'query': {'alltimes': True,
'channel': 'FLIGHT',
'format': 'bin',
'msids': ['3TSCPOS', 'AOPCADMD'],
'start': '2016:030:00:00:00.000',
'stop': '2016:030:00:05:00.000',
'tolerance': 2000,
'url': 'http://t...cfa.harvard.edu/maude/...'}}
Full resolution data
""""""""""""""""""""
Normally the MAUDE server will automatically return a sub-sampled version of the telemetry
data if the number of points would exceed around 100k. However, if the ``allow_subset``
parameter of :func:`~maude.maude.get_msids` is set to ``False`` then this function will prevent
sub-sampling by doing multiple small queries. This has an overhead penalty because it
may require multiple server requests to piece together the full query. In addition there
is a fixed upper limit of 7 days for the query interval (stop - start time) in this mode.
As an example::
>>> out = maude.get_msids('aoattqt1', '2016:001', '2016:003', allow_subset=False)
>>> len(out['data'][0]['values'])
168586
Caching
"""""""
Caching of the request to the MAUDE server can be enabled by setting the
configuration variable ``maude.conf.cache_msid_queries = True``, as shown in the
example below. This is useful when a request will be repeated verbatim and will
cache the last 32 requests.
>>> maude.conf.cache_msid_queries = True
>>> start, stop = '2016:030:00:00:00', '2016:030:01:00:00'
>>> %time dat = maude.get_msids('AOPCADMD', start, stop, allow_subset=False)
CPU times: user 73 ms, sys: 13.6 ms, total: 86.6 ms
Wall time: 296 ms
>>> %time dat = maude.get_msids('AOPCADMD', start, stop, allow_subset=False)
CPU times: user 6.31 ms, sys: 774 µs, total: 7.09 ms
Wall time: 6.24 ms
Get Blobs
^^^^^^^^^
The :func:`~maude.maude.get_blobs` function should be considered experimental at this point as it
has not been used extensively. It takes the following arguments:
========== =================================================================
Arg Description
========== =================================================================
start Start time (any Chandra.Time format, default=NOW - 2 days)
stop Stop time (any Chandra.Time format, default=NOW)
msids Optional list of MSIDs (if given, only these MSIDs are returned)
channel MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
user OCCweb user name
password OCCweb password
========== =================================================================
It returns a Python ``dict`` structure with keys:
============== ==================================================================
Key Description
============== ==================================================================
``blobs`` List of telemetry outputs, where each element is a ``dict``.
``flags`` An integer. Flags returned by MAUDE
``query`` Input query values for reference
``names`` The list of MSIDs in the result list
``types`` The list of dtypes in the query list (corresponding to names)
============== ==================================================================
The following statement retrieves the blobs, including all MSIDs, within a time range::
>>> blob = maude.get_blobs(start='2016:030:00:00:00', stop='2016:030:00:04:00')
>>> blob.keys()
dict_keys(['query', 'blobs', 'flags', 'names', 'types'])
Caching
"""""""
Caching of the request to the MAUDE server can be enabled by setting the configuration
variable ``maude.conf.cache_blob_queries = True``. This is useful when a request will be repeated
verbatim and will cache the last 32 requests.
Get Frames
^^^^^^^^^^
The :func:`~maude.maude.get_frames` takes the following arguments:
========== =================================================================
Arg Description
========== =================================================================
start Start time (any Chandra.Time format, default=NOW - 2 days)
stop Stop time (any Chandra.Time format, default=NOW)
channel MAUDE channel [FLIGHT | FLTCOMP | ASVT | TEST] (default=FLIGHT)
format MAUDE response format [json | bin] (default=bin)
user OCCweb user name
password OCCweb password
========== =================================================================
It returns a Python ``dict`` structure with four keys:
============== ==================================================================
Key Description
============== ==================================================================
``data`` A dict of the form {'f': int, 'frames': []}, where each entry in
result['frames'] is a dictionary that depends on the format:
- **If format == 'json'**: {'t': float, 'bytes as hex': str}
- **If format == 'bin'**: {'t': float, 'bytes': bytes}
``query`` Input query values for reference
============== ==================================================================
The following statements retrieve the all VCDU frames, including all MSIDs, within a time range::
>>> frames = maude.get_frames(start='2016:030:00:00:00', stop='2016:030:00:04:00')
>>> frames.keys()
dict_keys(['data', 'query'])
>>> frames['data'].keys()
dict_keys(['frames', 'f'])
>>> frames['data']['frames'][0]['bytes'][:30]
b'A\x89\x9d\x89E\x80\xf5b|\x00\xff\xa0\xb4\x8br\x81p\x83{cG\x10\xac\x17\xa4\x16\xddBA@'
>>> frames = maude.get_frames(start='2016:030:00:00:00', stop='2016:030:00:04:00', format='json')
>>> frames.keys()
dict_keys(['data', 'query'])
>>> frames['data']['frames'][0]['bytes_as_hex'][:30]
'41 89 9D 89 45 80 F5 62 7C 00 '
Caching
"""""""
Caching of the request to the MAUDE server can be enabled by setting the
configuration variable ``maude.conf.cache_frame_queries = True``. This is useful
when a request will be repeated verbatim and will cache the last 32 requests.
Logging
-------
To getting logging information, in particular some debug info that shows a bit of
what is happening under the hood, do::
>>> maude.set_logger_level('DEBUG') # for DEBUG
>>> out = maude.get_msids('aoattqt1', '2016:001', '2016:003', allow_subset=False)
get_msids: Using .netrc with user=taldcroft
get_msids_in_chunks: Chunked reading: max samples / major_frame = 32, chunk dt = 82000.0 secs
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016001120000000&tp=2016002040000000
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016002040000000&tp=2016002200000000
get_msids: Getting URL http://t...cfa.harvard.edu/...&ts=2016002200000000&tp=2016003120000000
API documentation
-----------------
Functions
^^^^^^^^^
.. automodule:: maude.maude
:members:
Configuration
^^^^^^^^^^^^^
.. automodule:: maude.config
:members: