Ska Package Helpers

Ska_helpers is a collection of utilities for the Ska3 runtime environment.

ska_helpers.test(*args, **kwargs)

Run py.test unit tests.

Chandra Models Data

Logging

ska_helpers.logging.basic_logger(name, format='%(asctime)s %(funcName)s: %(message)s', propagate=False, **kwargs)

Create logger name using logging.basicConfig.

This is a thin wrapper around logging.basicConfig, except:

• Uses logger named name instead of the root logger

• Defaults to a standard format for Ska applications. Specify format=None to use the default basicConfig format.

• Not recommended for multithreaded or multiprocess applications due to using a temporary monkey-patch of a global variable to create the logger. It will probably work but it is not guaranteed.

This function does nothing if the name logger already has handlers configured, unless the keyword argument force is set to True. It is a convenience method intended to do one-shot creation of a logger.

The default behaviour is to create a StreamHandler which writes to sys.stderr, set a formatter using the format string "%(asctime)s %(funcName)s: %(message)s", and add the handler to the name logger with a level of WARNING.

By default the created logger will not propagate to parent loggers. This is to prevent unexpected logging from other packages that set up a root logger. To propagate to parent loggers, set propagate=True. See https://docs.python.org/3/howto/logging.html#logging-flow, in particular how the log level of parent loggers is ignored in message handling.

Example:

# In __init__.py for a package or in any module
from ska_helpers.logging import basic_logger
logger = basic_logger(__name__, level='INFO')

# In other submodules within a package the normal usage is to inherit
# the package logger.
import logging
logger = logging.getLogger(__name__)

A number of optional keyword arguments may be specified, which can alter the default behaviour.

filename Specifies that a FileHandler be created, using the specified

filename, rather than a StreamHandler.

filemode Specifies the mode to open the file, if filename is specified

(if filemode is unspecified, it defaults to ‘a’).

format Use the specified format string for the handler.

datefmt Use the specified date/time format.

style If a format string is specified, use this to specify the

type of format string (possible values ‘%’, ‘{’, ‘$’, for %-formatting, str.format() and string.Template - defaults to ‘%’).

level Set the name logger level to the specified level. This can be

a number (10, 20, …) or a string (‘NOTSET’, ‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’) or logging.DEBUG, etc.

stream Use the specified stream to initialize the StreamHandler. Note

that this argument is incompatible with ‘filename’ - if both are present, ‘stream’ is ignored.

handlers If specified, this should be an iterable of already created

handlers, which will be added to the name handler. Any handler in the list which does not have a formatter assigned will be assigned the formatter created in this function.

force If this keyword is specified as true, any existing handlers

attached to the name logger are removed and closed, before carrying out the configuration as specified by the other arguments.

Note that you could specify a stream created using open(filename, mode) rather than passing the filename and mode in. However, it should be remembered that StreamHandler does not close its stream (since it may be using sys.stdout or sys.stderr), whereas FileHandler closes its stream when the handler is closed.

Note this function is probably not thread-safe.

Parameters

namestr

logger name

formatstr

format string for handler

propagate: bool

propagate to parent loggers (default=False)

**kwargsdict

other keyword arguments for logging.basicConfig

Returns

loggerLogger object

Retry

Retry package initially copied from https://github.com/invl/retry.

This project appears to be abandoned so moving it to ska_helpers.

LICENSE:

Copyright 2014 invl

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

exception ska_helpers.retry.RetryError(failures)

Keep track of the stack of exceptions when trying multiple times.

Parameters

failureslist of dict, each with keys ‘type’, ‘value’, ‘trace’.

ska_helpers.retry.retry(exceptions=<class 'Exception'>, tries=-1, delay=0, max_delay=None, backoff=1, jitter=0, logger=<Logger ska_helpers.retry.api (WARNING)>)

Returns a retry decorator.

Parameters

• exceptions – an exception or a tuple of exceptions to catch. default: Exception.

• tries – the maximum number of attempts. default: -1 (infinite).

• delay – initial delay between attempts. default: 0.

• max_delay – the maximum value of delay. default: None (no limit).

• backoff – multiplier applied to delay between attempts. default: 1 (no backoff).

• jitter – extra seconds added to delay between attempts. default: 0. fixed if a number, random if a range tuple (min, max)

• logger – logger.warning(fmt, error, delay) will be called on failed attempts. default: retry.logging_logger. if None, logging is disabled.

Returns

a retry decorator.

ska_helpers.retry.retry_call(f, args=None, kwargs=None, exceptions=<class 'Exception'>, tries=-1, delay=0, max_delay=None, backoff=1, jitter=0, logger=<Logger ska_helpers.retry.api (WARNING)>)

Calls a function and re-executes it if it failed.

Parameters

• f – the function to execute.

• args – the positional arguments of the function to execute.

• kwargs – the named arguments of the function to execute.

• exceptions – an exception or a tuple of exceptions to catch. default: Exception.

• tries – the maximum number of attempts. default: -1 (infinite).

• delay – initial delay between attempts. default: 0.

• max_delay – the maximum value of delay. default: None (no limit).

• backoff – multiplier applied to delay between attempts. default: 1 (no backoff).

• jitter – extra seconds added to delay between attempts. default: 0. fixed if a number, random if a range tuple (min, max)

• logger – logger.warning(fmt, error, delay) will be called on failed attempts. default: retry.logging_logger. if None, logging is disabled.

Returns

the result of the f function.

ska_helpers.retry.tables_open_file(*args, **kwargs)

Call tables.open_file(*args, **kwargs) with retry up to 3 times.

This only catches tables.exceptions.HDF5ExtError. After an initial failure it will try again after 2 seconds and once more after 4 seconds.

Parameters

• *args –

  args passed through to tables.open_file()

• **kwargs –

  kwargs passed through to tables.open_file()

Returns

tables file handle

Setup Helpers

ska_helpers.setup_helper.duplicate_package_info(vals, name_in, name_out)

Duplicate a list or dict of values inplace, replacing name_in with name_out.

Normally used in setup.py for making a namespace package that copies a flat one. For an example see setup.py in the ska_sun or Ska.Sun repo.

Parameters

• vals – list or dict of values

• name_in – string to replace at start of each value

• name_out – output string

Utilities

class ska_helpers.utils.LazyDict(load_func, *args, **kwargs)

Dict which is lazy-initialized using supplied function load_func.

This class allows defining a module-level dict that is expensive to initialize, where the initialization is done lazily (only when actually needed).

Parameters

load_funcfunction

Reference to a function that returns a dict to init this dict object

*args

Arguments list for load_func

**kwargs

Keyword arguments for load_func

Examples

from ska_helpers.utils import LazyDict

def load_func(a, b):
    # Some expensive function in practice
    print('Here in load_func')
    return {'a': a, 'b': b}

ONE = LazyDict(load_func, 1, 2)

print('ONE is defined but not yet loaded')
print(ONE['a'])

copy() → a shallow copy of D

get(key, default=None, /)

Return the value for key if key is in the dictionary, else default.

items() → a set-like object providing a view on D's items

keys() → a set-like object providing a view on D's keys

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

values() → an object providing a view on D's values

class ska_helpers.utils.LazyVal(load_func, *args, **kwargs)

Value which is lazy-initialized using supplied function load_func.

This class allows defining a module-level value that is expensive to initialize, where the initialization is done lazily (only when actually needed).

The lazy value is accessed using the val property.

Parameters

load_funcfunction

Reference to a function that returns a dict to init this dict object

*args

Arguments list for load_func

**kwargs

Keyword arguments for load_func

Examples

from ska_helpers.utils import LazyVal

def load_func(a):
    # Some expensive function in practice
    print('Here in load_func')
    return a

ONE = LazyVal(load_func, 1)

print('ONE is defined but not yet loaded')
print(ONE.val)

Version Info

The ska_helpers.version module provides utilities to handle package versions. The version of a package is determined using pkg_resources if it is installed, and setuptools_scm otherwise.

ska_helpers.version.get_version(package, distribution=None)

Get version string for package with optional distribution name.

If the package is not from an installed distribution then get version from git using setuptools_scm.

Parameters

package :

package name, typically __package__

distribution :

name of distribution if different from package (Default value = None)

Returns

str

Version string

ska_helpers.version.parse_version(version)

Parse version string and return a dictionary with version information. This only handles the default scheme.

Parameters

version :

str

Returns

dict

version information

Default versioning scheme

What follows is the scheme as described in setuptools_scm’s documentation.

In the standard configuration setuptools_scm takes a look at three things:

1. latest tag (with a version number)

2. the distance to this tag (e.g. number of revisions since latest tag)

3. workdir state (e.g. uncommitted changes since latest tag)

and uses roughly the following logic to render the version:

no distance and clean:

{tag}

distance and clean:

{next_version}.dev{distance}+{scm letter}{revision hash}

no distance and not clean:

{tag}+dYYYMMMDD

distance and not clean:

{next_version}.dev{distance}+{scm letter}{revision hash}.dYYYMMMDD

The next version is calculated by adding 1 to the last numeric component of the tag.

For Git projects, the version relies on git describe, so you will see an additional g prepended to the {revision hash}.

Due to the default behavior it’s necessary to always include a patch version (the 3 in 1.2.3), or else the automatic guessing will increment the wrong part of the SemVer (e.g. tag 2.0 results in 2.1.devX instead of 2.0.1.devX). So please make sure to tag accordingly.

Run time information

The ska_helpers.run_info module provides convenience functions to get and print relevant run time information such as machine name, user name, date, program version, and so on. This is aimed at executable scripts and cron jobs.

ska_helpers.run_info.get_run_info(opt=None, *, version=None, stack_level=1)

Get run time information as dict.

Parameters

opt :

argparse options (Default value = None)

version :

program version (default=__version__ in calling module)

stack_level :

stack level for getting calling module (Default value = 1)

Returns

dcit

run information

ska_helpers.run_info.get_run_info_lines(opt=None, *, version=None, stack_level=2)

Get run time information as formatted lines.

Parameters

opt :

argparse options (Default value = None)

version :

program version (default=__version__ in calling module)

stack_level :

stack level for getting calling module (Default value = 2)

Returns

list

formatted information lines

ska_helpers.run_info.log_run_info(log_func, opt=None, *, version=None, stack_level=3)

Output run time information as formatted lines via log_func.

Each formatted line is passed to log_func.

Parameters

log_func :

logger output function (e.g. logger.info)

opt :

argparse options (Default value = None)

version :

program version (default=__version__ in calling module)

stack_level :

stack level for getting calling module (Default value = 3)

Indices and tables

• Index

• Module Index

• Search Page