Data Utilities
--------------

Fetching data can be a complex task. Data can be can requested based on a time range or an OBSID.
Image telemetry data can be accessed through MAUDE, using either
blob or VCDU frame queries, or read from mica level0 files. The image data is enriched
with data from other sources to provide information on attitude and star catalogs. These sources
include the REST API on the kadi web server and various packages that access data on disk.
The actual fetching of the data can optionally be done on multiple processes.

Aca-view includes utilities (known as *services*) to fetch the data and notify the application as
the data becomes available. It also includes utilities to fetch data in standalone mode (no GUI).

- :ref:`data-fetch`
- :ref:`data-config`
- :ref:`data-api`

  - :ref:`low-level-functions`. Simple functions to get the data.
  - :ref:`data-services`.

    - :ref:`data-service`. The top-level class to encapsulate data access.
    - :ref:`Multi-process Data Service`. A collection of tools to fetch data in multiple processes.
    - :ref:`File Service`. A worker class to read data from files.
    - :ref:`MAUDE Service`.A worker class to get data from MAUDE.

  - :ref:`data-service-other`. Other useful functions.
- :ref:`aca_view_fetch`. A script to fetch data in standalone mode with the same arguments as
  aca_view

.. _data-fetch:

Fetching Data
^^^^^^^^^^^^^

The :any:`fetch <aca_view.data.fetch.fetch>` function and the :ref:`aca_view_fetch`
script are high-level utilities to get data without the graphical user interface. They wrap
all fetching functionality and are configured in the same way as aca_view configures data fetching.

The following statement reads a list of mica level0 files and outputs an
:any:`AcaTelemetryTimeline <aca_view.aca_telemetry_timeline.AcaTelemetryTimeline>` using only local
data:

.. code-block:: python

    from pathlib import Path
    from aca_view import tests
    from aca_view.data.fetch import fetch
    TEST_DATA_DIR = Path(tests.__file__).parent / 'data'
    timeline = fetch(
        filenames = TEST_DATA_DIR.glob('aca*fits.gz'),
        mica = True,
        ska_data_sources=['local']
    )

and the following statement fetches the first 100 seconds of OBSID 8008 image telemetry from MAUDE:

.. code-block:: python

    from aca_view.data.fetch import fetch
    timeline = fetch(
        obsid = 8008,
        stop = 100,
        maude = True,
        ska_data_sources=['ska_api']
    )

.. _data-config:

Configuring Data Fetching
^^^^^^^^^^^^^^^^^^^^^^^^^

The :any:`validate_config <aca_view.data.config.validate_config>` function produces a standard
dictionary with settings for the data fetching task. It tries to provide sensible defaults
for arguments that are not given, and tries to make sure the input arguments are consistent.

Within aca-view, the input to this function is a combination of settings in the persistent store
and arguments passed in the command line or the data-service dialog.

.. autofunction:: aca_view.data.config.validate_config

.. _data-api:

API
^^^

.. _low-level-functions:

Low-level Fetching Functions
""""""""""""""""""""""""""""

These are simple functions that can be called directly and do not depend on Qt. They are the ones
that actually get the data.

.. autofunction:: aca_view.data.core.fetch.from_file

.. autofunction:: aca_view.data.core.fetch.from_maude

.. autofunction:: aca_view.data.core.fetch.from_blobs

.. _data-services:

Services
""""""""

.. _data-service:

Data Service
''''''''''''

The top-level class encapsulating all data access

.. autoclass:: aca_view.data.data_service.DataService


.. _multi-process data service:

Multi-process Data Service
'''''''''''''''''''''''''''

A collection of tools to spread the data fetching into multiple child processes (both files or from MAUDE).

.. autoclass:: aca_view.data.multiprocess_service.MultiProcessWorker

.. autoclass:: aca_view.data.multiprocess_server.MultiProcessServer

.. autofunction:: aca_view.data.multiprocess_server.run

.. _file service:

File Service
''''''''''''

A worker class to read data from files. This has been superseded by the MultiProcessWorker.

.. autoclass:: aca_view.data.file_service.FileServiceWorker

.. _maude service:

MAUDE Service
'''''''''''''

A worker class to get data from MAUDE. This has been superseded by the MultiProcessWorker.

.. autoclass:: aca_view.data.maude_service.MaudeServiceWorker

.. _data-service-other:

Other
"""""

.. autofunction:: aca_view.data.core.add_centroids.add_centroids

.. autofunction:: aca_view.data.core.add_dark_cal.add_dark_cal

.. autofunction:: aca_view.data.core.add_magnitudes.add_magnitudes

.. autofunction:: aca_view.data.core.add_non_img_data.add_non_img_data

.. autofunction:: aca_view.data.core.add_residuals.add_residuals

.. autofunction:: aca_view.data.core.add_starcat.add_starcat

.. autofunction:: aca_view.data.core.add_status_strings.add_status_strings

.. autofunction:: aca_view.data.fetch.fetch

.. _aca_view_fetch:

aca_view_fetch
^^^^^^^^^^^^^^^

.. argparse::
   :ref: aca_view.scripts.fetch.get_parser
   :prog: aca_view_fetch