Low Latency Injections in O4

During the lead-up to O4, a replay of O3 data was available which contained an addition stream of injections buried in the original O3 data. These injections were used to measure the sensitivity of analysis pipelines, and the test suite was developed to provide support for the analysis of these injections.

Previously, however, there was no support for streaming injections in live/current hoft during the observing run. The low latency injection streamer solves this problem. Given a frame cache of previously generated injections (embedded in silence), the low latency injection streamer adds these injections on the fly to live hoft data, and writes a new frame to disk containing the original hoft data, and a new channel containing the injections embedded in the hoft data.

This output is currently used to monitor the sensitivity of the gstlal pipeline, but could in theory additionally be used to analyze injections for the RnP group in real-time and remove the need for additional offline analysis of such injections.

This script has been reviewed by the gstlal review team during several meetings, and was part of an sccb approved gstlal-ugly release.

The most recent version of the streamer can be found on the gstlal master branch here.

Location and Monitoring

The low latency injection streamer is run as a systemd service on the lowlatency1 headnode at CIT. Those given the appropriate permissions can manage this service via the lowlatency.gstlal account on this headnode. To login: ssh lowlatency.gstlal@lowlatency1.gstlal.ligo.caltech.edu

There is one service for each detector. The service files are named as lldd-live-injections@{ifo}.service and output the injection frame files in /dev/shm/{ifo}_injections/.

To check on the status of the service, systemctl --user status lldd-live-injections@{ifo}.serivce If running stably, this will output a handful of lines. You’re looking to see that the service is “active” and that there are no errors reported in the last couple of lines of log output.

To see more than the log 4 lines of log output, journalctl --user -u lldd-live-injections@{ifo}.service

and you can always ls -lthr /dev/shm/{ifo}_injections/ to sanity check if the injections written to disk are recent.

There are two Icinga monitors which check that these frames written to disk are recent. They can be found here

After the frames are written to disk, they need to be pushed to kafka in order to be streamed to nodes at CIT and ICDS. This process is managed by Patrick Brockill and is not accessible to the lowlatency.gstlal user. The distirubtion of injection frames to ICDS is monitored on Icinga by this check. If this check is critical, but the frames are being generated on the lowlatency node, please contact Ron Tapia or Patrick Brockill.

Debugging

If the services are running, but the frames are not being generated, you can try to restart the service. systemctl --user restart lldd-live-injections@{ifo}.service

Otherwise, use the journal to track down any error messages. journalctl --user -u lldd-live-injections@{ifo}.service

Maintaining

The injection files for the streamer must be copied to the lowlatency node itself in order to keep up with real time. The node only has space for two months of injections at a time. Before the first of each month, that month’s worth of injection frames must be rsynced to the node, and a frame cache must be made. The service should then be restarted during the CIT maitenance proceeding the first of the month with the frame cache option replaced to point to the new location.

To update the monthly frames:

The injection frames live in: /data/online_injections

First, remove the last month’s worth of data rm -rf /last/months/data

Injection files currently live in Reed’s dirs by month ls /home/reed.essick/rates+pop/rpo4-injections/online-injections/ In those dirs, you want ls /home/reed.essick/rates+pop/rpo4-injections/online-injections/MONTH_DIR_HERE/HKLV-rpo4_online_14_16384/*

To rsync the next month’s worth of data example. Replace month-specific info. rsync -azP /home/reed.essick/rates+pop/rpo4-injections/online-injections/rpo4-2023_09-1377561600-2592768/HKLV-rpo4_online_14_16384 rpo4-2023_09-1377561600-2592768

Then make a frame cache for that month ls MONTH_FRAME_DIR/*/*.gwf | lalpath2cache > MONTH_FRAME_DIR.cache

If not restarting on the first of the month at midnight (lol who is) might want to append the last day or so of the previous month to this cache. To do so, first make cache with just the last bit of the previous month. e.g. ls MONTH_FRAME_DIR/LAST_NUM_DIR/*.gwf | lal_path2cache > SOME_NAME.cache

To update the service file:

The service file currently lives in ~/services/live_injections/lldd-live-injections@.service

Open that file, and change the --inj-frame-cache option to point to the new location.

Move updated service file to systemd configuration location cp lldd-live-injections@.service $HOME/.config/systemd/user

Reload the service file you just copied into the systemctl daemon systemctl --user daemon-reload

If service previously running, restart. systemctl --user restart lldd-live-injections@H1.service

Alternatively, kick up the service from scratch systemctl --user start lldd-live-injections@H1.service