1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

  1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

Offline Filtering Rota

Table of Contents


Overview

The offline results are reranked using online results, and some data that were dropped in online analysis for some reason need to be refiltered for completeness. Also, the R&P group has produced injection sets to analyze offline. For both dropped data and injection sets, we will filter these injections using the same whitened & decomposed templates that were used online, hereafter referred to simply as SVDs. The non-injection and injection triggers output by this filtering will be included in the final combined reranks. The injection filtering workflow is now done using a Makefile. The Makefile can be used with some modifications for running an offline filtering dag.

Instructions (For dropped data filtering, see this page for a updated instruction.)

  1. Locate the latest container from offline_inj_filtering or dropped_data_refiltering branch (for filtering injections and refiltering the dropped data respectively) of the o4b-containers repository. The builds on the shared account are located at

    • Bob: /home/gstlalcbc.offline/observing/4/a/builds/
    • Rick: /ligo/home/ligo.org/gstlalcbc/observing/4/a/offline/builds/ and follow the naming convention offline_abcdefgh where abcdefgh is the first 8 characters of the git commit hash.

  2. Create an analysis directory and copy the Makefile from the build.
    injection filtering
    /home/gstlalcbc.offline/observing/4/a/injections at CIT,
    /ligo/home/ligo.org/gstlalcbc/observing/4/a/offline/injections at ICDS and
    dropped data refiltering
    /home/gstlalcbc.offline/observing/4/a/refiltering_dropped_data at CIT,
    /ligo/home/ligo.org/gstlalcbc/observing/4/a/offline/refiltering_dropped_data at ICDS.

    Note: The Makefile should point to the latest singularity build. If this is a routine setup for offline injection filering you may run

    make all_till_injections N=<N>

    and proceed to step 11.

  3. Copy the mass model and the appropriate manifest file from the run directory. (Has to be done only once)

    make copy_mass_model_and_manifest_file

  4. Copy the config to appropriate directories.

    For offline injection filtering we break the data into chunks decided by the online analysis rewhitening times. For o4b we rewhiten every 604800s on Tuesdays. We assume the SVDs were swapped at 16:00 UTC for simplicity. O4b started on 1396796418. First rewhitening was done on the following Tuesday (GPS: 1397318418) So the first chunk (N = 0) is 1396796418-1397318418, the second chunk (N = 1) is 1397318418-139792318 and so on. Note: All chunks except the first one are seperated by 604800s.

    make update_config N=<N> (N = 1, 2, 3 ...)

    For the first chunk (N=0) run

    make update_config START=1396796418 STOP=1397318418

    Use the start and stop times instead of N=0 in all steps for the first chunk.

    The number at the end denotes which RPO4 injection set is used, 000, 001, 002, or 003. Start with 000, instructions at the end describe shortcuts that can then be used for the other three injection sets for that time. Note that 000/-003/ is not needed for refiltering dropped data. If you are doing dropped data refiltering remonve the injection stuff.

  5. Create/renew x509 proxy Run

    make proxy

    which is equivalent to running

    X509_USER_PROXY=x509_proxy ligo-proxy-init albert.einstein

  6. Create the Makefile.

    make makefile N=<N> (N = 1, 2, 3 ...)

  7. Assuming that the offline PSD dag has not been run for this stretch of time, make the PSD dag and submit it.

    make submit_psd_dag N=<N> (N = 1, 2, 3 ...)

  8. Copy the online SVDs to the correct location.

    For all but the first week run

    make copy_svd N=<N> (N = 1, 2, 3 ...)

    For the first week copy the SVDs from the appropriate location manually. For Rick run

    rsync -rzvP /ligo/home/ligo.org/gstlalcbc.offline/observing/4/b/runs/rick_svd_bank_ref/svd_bank trigs.rick_1396796418_522000/input_data/
cd trigs.rick_1396796418_522000/input_data/svd_bank
for f in *; do \
   new_name=$(echo "$f" | sed -E 's/0-0/1396796418-522000/g') \
   mv "$f" "$new_name" \
done
cd ../../../

  9. Generate everything related to injections. (not needed for dropped data filtering)

    make injections N=<N> (N = 1, 2, 3 ...)

  10. (only for dropped data refiltering). Populate bin-specific options in the manifest file.

    singularity exec <build-dir> gstlal_inspiral_set_svdbin_option --config config.yml -w filter

  11. Make the injection filter dag and submit it. Run this after the PSD dags have finished.

    make submit_inj_filter_dag N=<N> (N = 1, 2, 3 ...)

  12. After a chunk has been filtered we will delete the SVDs to free up space on the cluster.

    make clean N=<N> (N = 1, 2, 3 ...)

Rick and Bob SVDs for o4b

To get the SVD for chunk trigs.analysis_START_DURATION go to /home/gstlalcbc.offline/observing/4/b/rewhiten/$(START-1008000)-604800/analysis_name/svd_bank This is not valid for the first chunk (N=0).

Edward Analysis Times for o4a

The Edward SVDs were replaced on Tuesdays. Week divisions are at 16:00 UTC. Note about week 6: the SVD policy hadn’t been finalized yet and the SVD was swapped out on July 1. We’ll be conservative and use the older SVD for the whole week.

Week Dates Start End SVD URL
01 May 24, 2023 - May 30, 2023 1368975618 1369497618 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.05.30/filter/svd_bank
02 May 30, 2023 - Jun 06, 2023 1369497618 1370102418 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.07.01/filter/svd_bank
03 Jun 06, 2023 - Jun 13, 2023 1370102418 1370707218 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.07.01/filter/svd_bank
04 Jun 13, 2023 - Jun 20, 2023 1370707218 1371312018 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.07.01/filter/svd_bank
05 Jun 20, 2023 - Jun 27, 2023 1371312018 1371916818 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.07.01/filter/svd_bank
06 Jun 27, 2023 - Jul 04, 2023 1371916818 1372521618 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.07.01/filter/svd_bank
07 Jul 04, 2023 - Jul 11, 2023 1372521618 1373126418 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.07.11/filter/svd_bank
08 Jul 11, 2023 - Jul 18, 2023 1373126418 1373731218 /home/gstlalcbc.online/observing/4/a/runs/trigs.edward_o4a/archive/23.07.18/filter/svd_bank
09 Jul 18, 2023 - Jul 25, 2023 1373731218 1374336018 /home/gstlalcbc.offline/observing/4/a/rewhiten/edward.230707_230714/svd_bank/13727
10 Jul 25, 2023 - Aug 01, 2023 1374336018 1374940818 /home/gstlalcbc.offline/observing/4/a/rewhiten/edward.230714_230721/svd_bank/13733
11 Aug 01, 2023 - Aug 08, 2023 1374940818 1375545618 /home/gstlalcbc.offline/observing/4/a/rewhiten/edward.230721_230728/svd_bank/13739
12 Aug 08, 2023 - Aug 15, 2023 1375545618 1376150418 /home/gstlalcbc.offline/observing/4/a/rewhiten/1374537618-604800/edward/svd_bank/13745
13 Aug 15, 2023 - Aug 22, 2023 1376150418 1376755218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1375142418-604800/edward/svd_bank/13751
14 Aug 22, 2023 - Aug 29, 2023 1376755218 1377360018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1375747218-604800/edward/svd_bank/13757
15 Aug 29, 2023 - Sep 05, 2023 1377360018 1377964818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1376352018-604800/edward/svd_bank/13763
16 Sep 05, 2023 - Sep 12, 2023 1377964818 1378569618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1376956818-604800/edward/svd_bank/13769
17 Sep 12, 2023 - Sep 19, 2023 1378569618 1379174418 /home/gstlalcbc.offline/observing/4/a/rewhiten/1377561618-604800/edward/svd_bank/13775
18 Sep 19, 2023 - Sep 26, 2023 1379174418 1379779218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1378166418-604800/edward/svd_bank/13781
19 Sep 26, 2023 - Oct 03, 2023 1379779218 1380384018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1378771218-604800/edward/svd_bank/13787
20 Oct 03, 2023 - Oct 10, 2023 1380384018 1380988818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1379376018-604800/edward/svd_bank/13793
21 Oct 10, 2023 - Oct 17, 2023 1380988818 1381593618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1379980818-604800/edward/svd_bank/13799
22 Oct 17, 2023 - Oct 24, 2023 1381593618 1382198418 /home/gstlalcbc.offline/observing/4/a/rewhiten/1380585618-604800/edward/svd_bank/13805
23 Oct 24, 2023 - Oct 31, 2023 1382198418 1382803218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1381190418-604800/edward/svd_bank/13811
24 Oct 31, 2023 - Nov 07, 2023 1382803218 1383408018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1381795218-604800/edward/svd_bank/13817
25 Nov 07, 2023 - Nov 14, 2023 1383408018 1384012818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1382400018-604800/edward/svd_bank/13824
26 Nov 14, 2023 - Nov 21, 2023 1384012818 1384617618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1383004818-604800/edward/svd_bank/13830
27 Nov 21, 2023 - Nov 28, 2023 1384617618 1385222418 /home/gstlalcbc.offline/observing/4/a/rewhiten/1383609618-604800/edward/svd_bank/13836
28 Nov 28, 2023 - Dec 05, 2023 1385222418 1385827218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1384214418-604800/edward/svd_bank/13842
29 Dec 05, 2023 - Dec 12, 2023 1385827218 1386432018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1384819218-604800/edward/svd_bank/13848
30 Dec 12, 2023 - Dec 19, 2023 1386432018 1387036818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1385424018-604800/edward/svd_bank/13854
31 Dec 19, 2023 - Dec 26, 2023 1387036818 1387641618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1386028818-604800/edward/svd_bank/13860
32 Dec 26, 2023 - Jan 02, 2023 1387641618 1388246418 /home/gstlalcbc.offline/observing/4/a/rewhiten/1386633618-604800/edward/svd_bank/13866
33 Jan 02, 2023 - Jan 09, 2023 1388246418 1388851218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1387238418-604800/edward/svd_bank/13872
34 Jan 09, 2023 - Jan 16, 2023 1388851218 1389456018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1387843218-604800/edward/svd_bank/13878

Jacob Analysis Times for o4a

The Jacob SVDs were replaced on Thursdays until December 20, then the SVDs were replaced on Wednesdays. Week divisions are at 16:00 UTC.

Week Dates Start End SVD URL
01 May 24, 2023 - Jun 01, 2023 1368975618 1369670418 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.06.01/filter/svd_bank
02 Jun 01, 2023 - Jun 08, 2023 1369670418 1370275218 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.06.30/filter/svd_bank
03 Jun 08, 2023 - Jun 15, 2023 1370275218 1370880018 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.06.30/filter/svd_bank
04 Jun 15, 2023 - Jun 22, 2023 1370880018 1371484818 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.06.30/filter/svd_bank
05 Jun 22, 2023 - Jun 29, 2023 1371484818 1372089618 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.06.30/filter/svd_bank
06 Jun 29, 2023 - Jul 06, 2023 1372089618 1372694418 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.07.13/filter/svd_bank
07 Jul 06, 2023 - Jul 13, 2023 1372694418 1373299218 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.07.13/filter/svd_bank
08 Jul 13, 2023 - Jul 20, 2023 1373299218 1373904018 /ligo/home/ligo.org/gstlalcbc/observing/4/a/runs/trigs.jacob_o4a/archive/23.07.20/filter/svd_bank
09 Jul 20, 2023 - Jul 27, 2023 1373904018 1374508818 /home/gstlalcbc.offline/observing/4/a/rewhiten/jacob.230707_230714/svd_bank/13727
10 Jul 27, 2023 - Aug 03, 2023 1374508818 1375113618 /home/gstlalcbc.offline/observing/4/a/rewhiten/jacob.230714_230721/svd_bank/13733
11 Aug 03, 2023 - Aug 10, 2023 1375113618 1375718418 /home/gstlalcbc.offline/observing/4/a/rewhiten/jacob.230721_230728/svd_bank/13739
12 Aug 10, 2023 - Aug 17, 2023 1375718418 1376323218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1374537618-604800/jacob/svd_bank/13745
13 Aug 17, 2023 - Aug 24, 2023 1376323218 1376928018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1375142418-604800/jacob/svd_bank/13751
14 Aug 24, 2023 - Aug 31, 2023 1376928018 1377532818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1375747218-604800/jacob/svd_bank/13757
15 Aug 31, 2023 - Sep 07, 2023 1377532818 1378137618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1376352018-604800/jacob/svd_bank/13763
16 Sep 07, 2023 - Sep 14, 2023 1378137618 1378742418 /home/gstlalcbc.offline/observing/4/a/rewhiten/1376956818-604800/jacob/svd_bank/13769
17 Sep 14, 2023 - Sep 21, 2023 1378742418 1379347218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1377561618-604800/jacob/svd_bank/13775
18 Sep 21, 2023 - Sep 28, 2023 1379347218 1379952018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1378166418-604800/jacob/svd_bank/13781
19 Sep 28, 2023 - Oct 05, 2023 1379952018 1380556818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1378771218-604800/jacob/svd_bank/13787
20 Oct 05, 2023 - Oct 12, 2023 1380556818 1381161618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1379376018-604800/jacob/svd_bank/13793
21 Oct 12, 2023 - Oct 19, 2023 1381161618 1381766418 /home/gstlalcbc.offline/observing/4/a/rewhiten/1379980818-604800/jacob/svd_bank/13799
22 Oct 19, 2023 - Oct 26, 2023 1381766418 1382371218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1380585618-604800/jacob/svd_bank/13805
23 Oct 26, 2023 - Nov 02, 2023 1382371218 1382976018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1381190418-604800/jacob/svd_bank/13811
24 Nov 02, 2023 - Nov 09, 2023 1382976018 1383580818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1381795218-604800/jacob/svd_bank/13817
25 Nov 09, 2023 - Nov 16, 2023 1383580818 1384185618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1382400018-604800/jacob/svd_bank/13824
26 Nov 16, 2023 - Nov 22, 2023 1384185618 1384704018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1383004818-604800/jacob/svd_bank/13830
27 Nov 22, 2023 - Nov 29, 2023 1384704018 1385308818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1383609618-604800/jacob/svd_bank/13836
28 Nov 29, 2023 - Dec 07, 2023 1385308818 1386000018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1384214418-604800/jacob/svd_bank/13842
29 Dec 07, 2023 - Dec 14, 2023 1386000018 1386604818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1384819218-604800/jacob/svd_bank/13848
30 Dec 14, 2023 - Dec 20, 2023 1386604818 1387123218 /home/gstlalcbc.offline/observing/4/a/rewhiten/1385424018-604800/jacob/svd_bank/13854
31 Dec 20, 2023 - Dec 27, 2023 1387123218 1387728018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1386028818-604800/jacob/svd_bank/13860
32 Dec 27, 2023 - Jan 03, 2023 1387728018 1388332818 /home/gstlalcbc.offline/observing/4/a/rewhiten/1386633618-604800/jacob/svd_bank/13866
33 Jan 03, 2023 - Jan 10, 2023 1388332818 1388937618 /home/gstlalcbc.offline/observing/4/a/rewhiten/1387238418-604800/jacob/svd_bank/13872
34 Jan 10, 2023 - Jan 16, 2023 1388937618 1389456018 /home/gstlalcbc.offline/observing/4/a/rewhiten/1387843218-604800/jacob/svd_bank/13878