analytics_controller¶

The main class of the analytics system is analytics_controller. This is invoked with das_analytics [options] [config file(s)]. All options can be specified either in a configuration file (executed as python source, so global variables in the file become configuration options) or specified on the command line (--help will list the available options). Options on the command line take precedence. Jobs can only be specified in a command line. Upon start, the server blocks until killed (rather than daemonising - this does not appear to agree with the python multiprocessing module).

The controller runs a task scheduler, which handles the execution of tasks, and a web server, which provides the principal information and control interface for the analytics system.

The controller creates a mongo capped collection in which to store the results of the tasks it runs, and the log messages generated by tasks and the analytics system itself. This defaults to 64MiB in size.

scheduler¶

The scheduler owns a pool of worker processes, and submits jobs from an internal queue to the pool at their submission time. The jobs then run and on completion will either be automatically rescheduled based on their requested interval, can request a custom resubmission time and can also create new child tasks. Jobs that fail are automatically retried a small number of times before being abandoned. A minimum submission interval is enforced to try and prevent badly configured jobs performing a DoS attack on the server. The scheduler runs in a separate thread.

webserver¶

The analytics webserver is run automatically on starting the server. By default, it serves from http://host:8213/analytics/. It does not include any security itself, and contains control functions (add/remove task, alter configuration, etc) which should not be generally available, so should be accessible only by SSH forwarding or similar.

The webserver allows one to: * View the current schedule, tasks currently executing and change the resubmission time. * View the raw output of recently run tasks, and the configuration of each task. * Reports showing better presented outputs from tasks or the system in general. * Control the server, either changing the system configuration or adding new tasks. * View a small amount of online help.

tasks¶

An analytics task is a python class which will be run at regular intervals by the analytics server. Tasks receive a set of standard arguments at init in addition to those specified when the job was configured, then are called with no arguments to run. They should then return a dictionary containing any information they wish to make visible on the webserver, along with special arguments such as a requested resubmission time or new tasks they wish to spawn.

Tasks are specified in a python configuration file like:

Task(name, classname, interval, **kwargs)

where kwargs include any task-specific arguments necessary, classname is the name of a class in DAS.analytics.tasks and name is a display identifier for the task.

Additionally, you can supply special kwargs only_once, only_before, max_runs to limit the task running. Time-like arguments are given as seconds in the unix time convention.

When the task is called, it receives a kwargs dictionary containing

plus any kwargs given when the task was defined. The task should use the supplied logger to record any log messages (which will then be stored along with the task result.

The actual task should be carried out in the __call__ method of the class object, which should finally return a dictionary containing any useful information, and any special information the task wishes to pass back to the scheduler.

The following special values are understood in the return dictionary:

Upon startup, the server indexes all classes in DAS.analytics.tasks searching for task classes, but it is recommended that tasks be placed in a file of the same name. The indexer will look up the following attributes if they exist.

• task_title: display name for the task (default task.__class__.__name__)

• task_info: longer text description (default task.__doc__)

• task_options: list of dictionaries, each containing string attributes name, default, type, help

  used by the web interface to make adding tasks easier

• hide: don’t index this task

standalone tasks¶

Tasks can be run independently of the analytics scheduler with the script das_analytics_task. This will run the task as normal and report if new tasks were requested or resubmission altered, but obviously no new tasks will actually be spawned. Useful for testing or running one-off jobs. Example, to run the test task:

das_analytics_task -c Test -o "{'message': 'hello world'}"

cli interface¶

A CLI script, das_analytics_cli is provided which allows some of the webserver control options to be triggered without a web browser. This still makes HTTP calls so requires the web server to be running and reachable.

reports¶

Reports are classes that are intended to provide richer views of the output of either tasks, analytics in general or DAS in general. A report consists of a class in DAS.analytics.reports (optionally inheriting from DAS.analytics.utils.Report), and one or more cheetah templates used for presenting information.

Code in a report class is executed by the web thread handling the request, and so should involve a minimum of actual computation. If computation is required for a report, a task should be created which runs intermittently performing the computation and returning the result, which can then be retrieved by the report from the analytics mongodb collection.

The __call__ method of the report should return a two-tuple of the name of the template desired and a dictionary required by cheetah to format the template. The __call__ method receives any extra parameters to the report URL as kwargs.

Reports are indexed similarly to tasks, with the following parameters understood.

• report_group: For grouping reports (default “general”)
• report_title: Display name (default report.__class__.__name__)
• report_info: Longer information (default report.__doc__)
• hide: Don’t index this class.

By convention, the templates should be called analytics_report_<reportname>.tmpl

Templates wishing to generate plots can generate URLs to a plotfairy instance, if the location of one has been supplied to analytics. See existing examples.

implemented tasks¶

generate_task(self, item, count, epoch_start, epoch_end)

    Return a new task dictionary for the selected item, eg
    a QueryMaintainer task if queries are the items being selected

make_one_summary(self, start, finish)

    Return an item->count mapping for the start and end times specified

report(self)

    Return a dictionary of extra keys to go in the return value

preselect_items(self, items)

    Remove unwanted keys (eg, those containing wildcards if you want to
    not consider them), then return items

mutate_items(self, items)

    Perform any merging of selected keys, eg determining which queries
    are supersets of each other, then return items