Package glue :: Module pipeline

Module pipeline

This modules contains objects that make it simple for the user to create python scripts that build Condor DAGs to run code on the LSC Data Grid.

This file is part of the Grid LSC User Environment (GLUE)

GLUE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

Version: 8cbd1b7187ce3ed9a825d6ed11cc432f3cfde9a5

Date: 2017-12-05 15:29:36 +0000

Author: Duncan Brown <duncan@gravity.phys.uwm.edu>

Classes

[hide private]

AnalysisChunk
An AnalysisChunk is the unit of data that a node works with, usually some subset of a ScienceSegment.

AnalysisJob
Describes a generic analysis job that filters LIGO data as configured by an ini file.

AnalysisNode
Contains the methods that allow an object to be built to analyse LIGO data in a Condor DAG.

CondorDAG
A CondorDAG is a Condor Directed Acyclic Graph that describes a collection of Condor jobs and the order in which to run them.

CondorDAGError

CondorDAGJob
A Condor DAG job never notifies the user on completion and can have variable options that are set for a particular node in the DAG.

CondorDAGJobError

CondorDAGManJob
Condor DAGMan job class.

CondorDAGManNode
Condor DAGMan node class.

CondorDAGNode
A CondorDAGNode represents a node in the DAG.

CondorDAGNodeError

CondorError
Error thrown by Condor Jobs

CondorJob
Generic condor job class.

CondorJobError

CondorSubmitError

DeepCopyableConfigParser
The standard SafeConfigParser no longer supports deepcopy() as of python 2.7 (see http://bugs.python.org/issue16058).

LDBDCJob
A ldbdc job can be used to insert data or fetch data from the database.

LDBDCNode
Runs an instance of ldbdc in a Condor DAG.

LSCDataFindJob
An LSCdataFind job used to locate data.

LSCDataFindNode
A DataFindNode runs an instance of LSCdataFind in a Condor DAG.

LigolwAddJob
A ligolw_add job can be used to concatenate several ligo lw files

LigolwAddNode
Runs an instance of ligolw_add in a Condor DAG.

LigolwCutJob
A ligolw_cut job can be used to remove parts of a ligo lw file

LigolwCutNode
Runs an instance of ligolw_cut in a Condor DAG.

LigolwSqliteJob
A LigolwSqlite job.

LigolwSqliteNode
A LigolwSqlite node.

LsyncCache

NoopJob
A Noop Job does nothing.

NoopNode
Run an noop job in a Condor DAG.

ScienceData
An object that can contain all the science data used in an analysis.

ScienceSegment
A ScienceSegment is a period of time where the experimenters determine that the inteferometer is in a state where the data is suitable for scientific analysis.

SegmentError

SqliteJob
A cbc sqlite job adds to CondorDAGJob and AnalysisJob features common to jobs which read or write to a sqlite database.

SqliteNode
A cbc sqlite node adds to the standard AnalysisNode features common to nodes which read or write to a sqlite database.

Functions

[hide private]

recurse_pfn_cache(node, caches=[]) source code

s2play(t)
Return True if t is in the S2 playground, False otherwise t = GPS time to test if playground

source code

Variables

[hide private]

PEGASUS_BASEDIR_SCRIPT = '\n#!/bin/bash\n\nTMP_EXEC_DIR=%s\nUE...

PEGASUS_PROPERTIES = '\n######################################...

PEGASUS_SCRIPT = '#!/bin/bash\nTMP_EXEC_DIR=%s\nIHOPE_RUN_DIR=...

__package__ = 'glue'

Variables Details

[hide private]

PEGASUS_BASEDIR_SCRIPT

Value:

'''
#!/bin/bash

TMP_EXEC_DIR=%s
UEBER_CONCRETE_DAG=%s

usage()
{
...

PEGASUS_PROPERTIES

Value:

'''
######################################################################
#########
# Pegasus properties file generated by pipeline
#
######################################################################
#########
# Catalog Properties
...

PEGASUS_SCRIPT

Value:

'''#!/bin/bash
TMP_EXEC_DIR=%s
IHOPE_RUN_DIR=%s
UEBER_CONCRETE_DAG=%s
usage()
{
  echo "Usage: pegasus_submit_dag [-f] [-h]"
  echo
...