1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

  1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

Online Operations Manual

Table of Contents


Common Errors and Known Solutions

Node issue

When running your setup jobs online_setup_dag.dag, you may find some of the gstlal_inspiral_svd_bank jobs failed with an error:

Traceback (most recent call last):
	...
	...
	ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory
	...
	...
	ModuleNotFoundError: No module named '_lal'

which is possibly due to the incompatibility between some of the cluster nodes and gstlal jobs. You can try condor_release job_ID and let the jobs land on other workable nodes.

If condor_release is not helping, try touch ~/.nointel and log out of the cluster. Next, you can log back in, source env.sh, and repeat the submission of online_setup_dag.dag. nointel tells the cluster to avoid running the gstlal jobs on the INTEL nodes.

File corruption

Errors of the following form often indicate file corruption:

xml.sax._exceptions.SAXParseException: <unknown>:1:0: syntax error

This can happen if Git Large File Storage is not installed when large files are downloaded from a repository. Try running git lfs install, and then redownloading any files that were checked into the repo using lfs. These files can be identified by an LFS icon next to the file name in the gitlab web interface.

Environment variables

GSTLAL_CHECK_TIMESTAMPS

The check_timestamps element is used for debugging purposes. If the environment variable GSTLAL_CHECK_TIMESTAMPS is not set, the check_timestamps element will be skipped. If GSTLAL_CHECK_TIMESTAMPS is exported, the check_timestamps element will be called.

GST_DEBUG

Sets gstreamer logging level.