Chandra Data Archive and Ocat

The mica.archive.cda.services module provides a Python interface to the Chandra Data Archive (CDA) web services and an interface to a local disk copy of the Observation Catalog (Ocat) if that is available.

The CDA web services interface provides access to the following. Note however that the CDA is not accessible from the GRETA network.

  • Ocat details: full details from the Ocat (124 fields) for the mission, plus access to ACIS windows, roll requirements and time requirements for applicable observations.

  • Ocat summary: summary data from the Ocat (26 fields).

  • Proposal abstracts: abstract information for each observation.

  • Archive file list: list of raw data files in the Chandra archive for each observation.

On the HEAD and GRETA networks, a copy of the Ocat details table (124 fields) is maintained and updated daily. This is a compressed HDF5 file which is approximately 5 Mb and can be synced to a local (laptop) computer. It is located at:

${SKA}/data/mica/archive/ocat_target_table.h5

This local file version can be used for faster, network-free queries of the Ocat.

This is the type of content that can be seen directly at:

https://cda.harvard.edu/srservices/ocatDetails.do?obsid=2121

Local Ocat access

The get_ocat_local() function provides fast access to the Ocat if the local Ocat HDF5 file is available on disk. In particular reading the entire Ocat or querying on a target name substring is at least 10 times faster than using the Web access to CDA.

The local version of the Ocat is cached in memory the first time it is read in full, so this means subsequent full reads or searches for target names will be much faster. The cache expires in one hour in case you have a long-running job.

Any of the fields in the returned table can be used for filtering the query via exact matches of the parameters:

>>> from mica.archive.cda import get_ocat_local
>>> dat = get_ocat_local(obsid=5438)
>>> dat['target_name']
'R Aqr'

>>> dat = get_ocat_local()  # Get the whole Ocat for the mission
>>> dat = get_ocat_local(instr='HRC-S', grat='LETG')  # All HRC-S LETG obs
>>> dat
<Table length=734>
seq_num  status  obsid  pr_num  target_name ... evfil evfil_lo evfil_ra efficient spwin
                                            ...         keV      keV
  str6   str11   int64   str8      str29    ...  str1 float64  float64     str1    str1
------- -------- ----- -------- ----------- ... ----- -------- -------- --------- -----
        archived 62635                      ...     0       --       --         0     0
        archived 62636                      ...     0       --       --         0     0
    ...      ...   ...      ...         ... ...   ...      ...      ...       ...   ...
 901342 archived 20944 18900560     Mkn 421 ...     0       --       --         0     0
 901410 archived 21389 20900088 PKS0405-123 ...     0       --       --         0     0
 901410 archived 21955 20900088 PKS0405-123 ...     0       --       --         0     0

The target_name is handled specially. By default, this parameter matches any substring in the Ocat target name field, ignoring spaces:

# Targets with 'jet' in name.
>>> dat = get_ocat_local(target_name='jet')

However, if the target name is a valid catalog source name that can be resolved by the CDS name resolver, then the target name can be used for a radial cone search around the target position by specifying resolve_name=True:

# Observations within 4 arcmin of 3C273
>>> dat = get_ocat_local(target_name='3c273', resolve_name=True, radius=4)

# Observations within 1 degree of RA, Dec = 10, 10
>>> dat = get_ocat_local(ra=10, dec=10, radius=60)

One convenience in this function is that when the obsid is specified, then by default the result is returned as a dict instead of a Table. This saves the boilerplate step of selecting element-0 of a Table.

Advanced users may consider use of the where keyword argument, which allows fast in-kernel filtering of the table contents. See https://www.pytables.org/usersguide/condition_syntax.html for details. This feature is used internally for the exact parameter matching as well as for the positional cone search around a coordinate.

Web access to CDA

The Chandra Data Archive hosts a web service that provides access to various elements of the archive. Examples follow.

Ocat

The get_ocat_web() function for web access to the Ocat is similar to the local file access documented above, and all of those examples will work just by changing get_ocat_local to get_ocat_web.

Additional functionality is described in the function docs get_ocat_web(). In particular the Web API parameters can be used via the function call.

Proposal abstracts

The get_proposal_abstract() function allows getting information about the proposal and the abstract:

>>> from mica.archive.cda import get_proposal_abstract
>>> get_proposal_abstract(obsid=8000)
{'abstract': 'We propose the Chandra-COSMOS survey which will provide an '
             'unprecedented combination of contiguous area, depth and '
             'resolution. 36 densely tiled observations will cover the central '
             '0.7 sq.deg. COSMOS field to a uniform 200ksec depth. COSMOS '
             'explores the coupled evolution of galaxies, dark matter halos '
             'and AGNs (massive black holes) largely free of cosmic variance. '
             'COSMOS is a comprehensive survey including: HST, Spitzer, '
             'Subaru, VLT, Magellan, VLA, MAMBO, GALEX, & potentially EVLA & '
             'ALMA. Chandra resolution & sensitivity enables the study of '
             'large scale phenomena: (1) influence of the surrounding '
             'environment; (2) interaction between galaxies; (3) influence of '
             'groups and clusters',
 'principal_investigator': 'Martin Elvis',
 'proposal_number': '08900073',
 'proposal_title': 'THE CHANDRA-COSMOS SURVEY'}

Archive file list

The get_archive_file_list() function allows getting a list of archive files for given obsid, detector, level, and dataset (and possibly other parameters).

Example:

>>> get_archive_file_list(obsid=2365, detector='pcad',
...                           subdetector='aca', level=1, obi=2)
<Table length=27>
        Filename            Filesize      Timestamp
            str30               int64          str19
------------------------------ -------- -------------------
pcadf126690624N007_asol1.fits  7300800 2021-04-09 08:04:29
pcadf02365_002N001_asol1.fits  4728960 2021-04-09 08:04:30
                        ...      ...                 ...
pcadf126695890N007_adat61.fits  1293120 2021-04-09 08:04:28
pcadf126695890N007_adat71.fits  1293120 2021-04-09 08:04:28

>>> get_archive_file_list(obsid=400, detector='acis', level=2, filetype='evt2')
<Table length=1>
        Filename         Filesize      Timestamp
        str24            int64          str19
------------------------ -------- -------------------
acisf00400N007_evt2.fits  4619520 2011-07-08 13:52:57

Ocat table fields

Some of these fields may be described in the chaser help at:

https://cda.harvard.edu/chaser/chaserFieldHelp.html

Column

Description

seq_num

Sequence Number

status

Status (unobserved, archived, untriggered, etc)

obsid

Obsid

pr_num

Proposal Number

target_name

Target name

grid_name

database id of grid name if grid observation

instr

Instrument

grat

Grating (HETG, LETG, NONE)

type

Observation type (TOO, GO, GTO)

obs_cycle

Observation cycle

prop_cycle

Proposal cycle

charge_cycle

Charge cycle

start_date

Observation start date

public_avail

Date publicly available

readout_detector

Detector (which HRC detector or string of actual ACIS ccds)

datamode

Instrument data mode

joint

Joint observatories (string)

hst

HST time (orbits?)

noao

NOAO time

nrao

NRAO time

rxte

RXTE time

spitzer

SPITZER time

suzaku

SUZAKU time

xmm

XMM time

swift

SWIFT time

nustar

NUSTAR time

category

Science category

seg_max_num

prop_title

Proposal Title

pi_name

Principal investigator name

observer

Observer name

app_exp

Approved exposure time (ks)

exp_time

Actual exposure time (ks)

ra

Target Right Ascension

dec

Target Declination

soe_roll

Roll from SOE

time_crit

Time critical flag

y_off

Y offset

z_off

Z offset

x_sim

X SIM

z_sim

Z SIM

raster

Raster flag

obj_type

Object type

obj

Solar system object name

photo

Photometry flag

vmag

V Mag of object

est_cnt_rate

Estimated count rate

forder_cnt_rate

First order count rate

count_rate

event_count

dither

Dither flag

y_amp

Dither Y amplitude if custom dither

y_freq

Dither Y frequency if custom dither

y_phase

Dither Y phase if custom dither

z_amp

Dither Z amplitude if custom dither

z_freq

Dither Z frequency if custom dither

z_phase

Dither Z phase if custom dither

roll

Roll constraint flag

window

Window constraint flag

unint

Uninterrupt constraint flag

pointing_update

Pointing update constraint flag

monitor

Monitor series flag

pre_id

Obsid of previous observation in monitor series

mon_min

Min days from pre_id for monitor observation

mon_max

Max days from pre_id for monitor observation

group_id

Database group id

constr

epoch

period

pstart

ps_marg

pend

pe_marg

multitel

multitel_obs

multitel_int

constr_rmk

Constraint in remarks flag

too_type

too_start

too_stop

alt_group

alt_trig

simode

Science Instrument (SI) mode

hrc

spect_mode

blank_en

u_hi

v_hi

u_lo

v_lo

timing

z_blk

acis

mode

ACIS mode (CC or TE)

bep_pack

ACIS BEP PACK (F, G, VF, F+B)

dropped_chip_cnt

Dropped chip count

i0

ACIS I0 ccd status (Y, N, optional with number, or D if dropped)

i1

ACIS I1 ccd status (Y, N, optional with number, or D if dropped)

i2

ACIS I2 ccd status (Y, N, optional with number, or D if dropped)

i3

ACIS I3 ccd status (Y, N, optional with number, or D if dropped)

s0

ACIS S0 ccd status (Y, N, optional with number, or D if dropped)

s1

ACIS S1 ccd status (Y, N, optional with number, or D if dropped)

s2

ACIS S2 ccd status (Y, N, optional with number, or D if dropped)

s3

ACIS S3 ccd status (Y, N, optional with number, or D if dropped)

s4

ACIS S4 ccd status (Y, N, optional with number, or D if dropped)

s5

ACIS S5 ccd status (Y, N, optional with number, or D if dropped)

spectra_max_count

Spectra Max Count

multiple_spectral_lines

Multiple spectral lines expected (Y, N)

subary

ACIS subarray (CUSTOM, NONE)

strt_row

Start row of ACIS subarray

row_cnt

Number of rows of ACIS subarray

d_cyc

sec_cnt

pr_time

sec_time

f_time

oc_sum

oc_row

oc_col

evfil

evfil_lo

evfil_ra

efficient

ACIS use most efficient (Y, N)

spwin

Spatial window (Y, N)