Lidax
Quick Start Guide

Daniel Sigg, May 2001


  1. First Steps
    1. Source Selection
    2. Destination Selection
    3. Progress Report
    4. Save/Restore
    5. Monitor Selection
    6. Run
  2. Understanding Universal Data Set Names (UDN)
  3. Advanced Setup
    1. Environment Variables
    2. Setup of UDN Lists
    3. Command Line Arguments


First Steps

The LIGO Data Access Tool (Lidax) is a graphical user interface to retrieve data from the Laser Interferometer Gravitational-wave Observatory. Start Lidax by typing:

lidax
This will bring up the following window:


 

Source Selection

The source area lets the user select the data server, the name of the data set, the channels of interest and the measurement time. If more than one server is selected, the data streams are merged as they are read. After selecting one or multiple data servers the user is presented with a list of available data set names in the UDN (Universal Data set Name) selection box. Selecting a UDN enables the channel selection window and the time selection window which should present a list of channels and measurement intervals, respectively.

Server Selection

Lidax currently supports the following data servers and data sources:

  • Lars (LIGO Archive Server): Access to the LIGO archives at Caltech and at the two observatory sites. This is the default method to get offline frame data. In order to be granted access the user must obtain an ldas user name and password and use the ldaspw utility to set it up from the command line.
  • Local file system: Access frames stored on a local disk system.
  • Local tape drive/robot: Read frames directly from a tape drive.
  • NDS (Network Data Server): Get online data from the NDS proxy servers at the two observatroy sites.
  • Shared memory partition: Get data from a DMT shared memory region. This requires a producer such as Dpush which copies data from file or the online broadcasting system.

  • UDN Selection

    A few examples of valid data set names (see Understanding UDNs for more details):

  • lars:////ligo/raw/lho/e4 : describes the data of the 4th enginering run (LHO).
  • nds://red.ligo-wa/minute-trend : descibes the online minute trend available at LHO.
  • dir:///export/raid2/E3/01-03-09_18:02:19.@ : describes a series of numbered directories located on a local disk.
  • file:///export/raid2/E3/*/*.F : describes all files matching the specified wildcard on a local disk.
  • file:///export/raid2/E3/filelist.txt : descibes a text file containing a list of UDNs (one per line).
  • tape:///dev/rmt/0n : describes a device name for a magnetic tape drive (files must have been written to tape using tar).
  • tape:///dev/rmt/0n -r CY0@1:10 : describes a drive in a Cybernetics tape library; in this example tapes have been loaded into slots 1 to 10.
  • dmt:///LHO_Online : describes a DMT shared memory partition with name LHO_Online.

  • Channel Selection

    The wildcard selection accepts * (star), ? (question mark) and [] (square brackets) and it supports the same matching as the unix command line (see fnmatch).

    Time Selection

    Destination Selection

    The client area lets the user select the data destination, the name of the data set, the channels of interest and the frame format. If more than one destination is selected, the data stream is split as it is written. After selecting one or multiple data sinks the user is presented with a list of available destinations in the UDN selection box. At this point it is also possible to specify a destination channel selection which has to be a subset of the combined list of source channels. Being able to set a channel list separately for each client makes it possible to split a frame stream into multiple data subsets concurrently. The user should also select an output frame format.

    Client Selection

    The output selection is limited to:

  • Local file system: Store data on a local disk system.
  • Local tape drive/robot: Write frames directly to a tape drive.
  • Shared memory partition: Send data to a DMT shared memory region.

  • UDN Selection

    Selecting a data set name is essentially the same as for a server. For more details about UDNs see the next section.

    Channel Selection

    This is identical to the source channel selection.

    Format Selection

    The frame file format is defined by the length of each frame in seconds, the number of frames per file and the compression method.

    Progress Report


    A data retrieveal or converting process may take a long time and being able to monitor the progress can be convenient. A user has the choice between several different reporting options:

  • Log file: The log file reports each processed frame - both in the input and the output data streams.
  • Wep page: The web page presents a summary of progress so far and is auto-reloadable.
  • e-mail: A notfication message can be sent after everything is done.
  • Dialog box: A dialog box with a progress bar is shown. This dialog box can be closed without terminating the process.

  • Save/Restore

    The settings of the Lidax program can saved to file in the LIGO lightweight data format (XML). Saved settings can be restored from file as well.

    Monitor Selection

    This option is useful if the data is directly fed into one ore multiple DMT monitor processes. To make it work at least on of the destinations must be a shared memory partition. Monitors are typically started before the data retrieval process begins and the name of the shared memory partition is passed as a command line argument (denoted by "$part").

    Run

    When the run button is pressed, the program forks in the background and starts the data retrieval and conversion process. It is not necessary to keep the terminal open.



    Understanding Universal Data Set Names (UDN)

    A detailed description of UDNs can be found in T010062.



    Advanced Setup

    Environment Variables

    The following environmental variables can be set:

  • LIGOSMPART : Name of the default shared memory partition.
  • LIGOSMPARTS : A set of shared memory partitions separated by colons.
  • NDSSERVER : The name(s) of the Network Data Servers separated by commas. The format is 'ip address'[:'port number'].
  • UDNFILE : A set of file names separated by colons. Each file lists a set of available data set names located on the local disk system.

  •  

     

    Examples:

    setenv LIGOSMPART LHO_Online
    setenv LIGOSMPARTS offline_1:offline_2
    setenv NDSSERVER red.ligo-wa.caltech.edu,london.ligo-la.caltech.edu
    setenv UDNFILE $HOME/file1.udn:$HOME/file2.udn


    Example $HOME/file1.udn:

    # E2 data on fortress
    dir:///export/raid3/E2/00-11-08_19:59:35.@
    dir:///export/raid2/E2/00-11-09_09:17:47.@
    dir:///export/raid2/E2/00-11-12_17:10:19.@
    dir:///export/raid2/E2/00-11-14_00:25:08.@


    Setup of UDN Lists

    To accelerate disk access and make it more convenient for repeated tasks one can generate a UDN list and an associated channel list. These two lists can then be read by LiDaX/DTT without accessing the files first. For example:

    udnls "/samraw/S3/L0/LHO/H-R-75*/*.gwf" > mylist.udn
    finfo -ce "/samraw/S3/L0/LHO/H-R-7555/H-R-755500000-16.gwf" > mylist.chn
    will create a UDN list in mylist.udn and a channel list in mylist.chn. Pointing DTT to the frames  directly would take a very very long time to load.  The UDN file will look like this (where the c-option is used to denote how many files are continuously added to the series):
    file:///samraw/S3/L0/LHO/H-R-7511/H-R-751117376-16.gwf -c 36
    file:///samraw/S3/L0/LHO/H-R-7511/H-R-751119776-16.gwf -c 122
    file:///samraw/S3/L0/LHO/H-R-7511/H-R-751122560-16.gwf -c 189
    file:///samraw/S3/L0/LHO/H-R-7511/H-R-751198368-16.gwf -c 62
    file:///samraw/S3/L0/LHO/H-R-7512/H-R-751203392-16.gwf -c 19
    file:///samraw/S3/L0/LHO/H-R-7512/H-R-751210592-16.gwf -c 63
    ...
    The channel file will look like this:
    H1:SUS-MC1_URPOS_GAIN                16
    H1:IOO-MC_F                                       16384
    H2:LSC-ASPD1_TempMon                    16
    H1:SUS-SM_SPDMon                               16
    H2:SUS-RM_ULCOIL_INMON            16
    H1:IOO-MC_I                                        16384
    H1:SUS-SM_ASCPIT_GAIN                 16
    If you these are raw frames, DTT will spend most time building up the channel selection lists. You can further accelerate this by only listing the channels that are actually use. In most circumstances you can get rid of the slow channels. So,
    finfo -ce "/samraw/S3/L0/LHO/H-R-7555/H-R-755500000-16.gwf" | awk '$2>16' >  mylist.chn
    may be a faster option in most cases.

    Command Line Arguments

    The Lidax program has the following usage format:

    usage: lidax [-p] [-o 'dir' [-f 'format']]
               lidax -h
               lidax -H 'env'
               -p : don't fork (useful for pipe)
               -o 'dir' : output is written into specified directory
               -f 'format' : output frame format
               -h : this help
               -H 'env' : help for an environment variable