ocs-pysmurf-monitor sqlite database lock issue

[BrianJKoopman]

Discussed in https://github.com/simonsobs/daq-discussions/discussions/122

This relates to the pysmurf-monitor and suprsync agents accessing the same sqliteDB.

^{Originally posted by ykyohei November 19, 2024}
Version: ocs-pysmurf-agent:v0.5.0-5-g6688b61-dev

Error log

2024-11-19T13:18:44+0000 New file: /data/smurf_data/20241119/ufm_mv35/1732022101/plots/1732022283_fknee_summary.png
2024-11-19T13:18:48+0000 New file: /data/smurf_data/20241119/ufm_mv35/1732022101/plots/1732022283_asd_summary.png
2024-11-19T13:18:48+0000 New file: /data/smurf_data/20241119/ufm_mv35/1732022101/outputs/1732022283_take_noise.npy
2024-11-19T13:19:12+0000 New file: /data/so/timestreams/17320/ufm_mv27/1732022318_000.g3
2024-11-19T13:19:18+0000 run:0 CRASH: [Failure instance: Traceback: <class 'sqlalchemy.exc.OperationalError'>: (sqlite3.OperationalError) database is locked
[SQL: INSERT INTO supersync_v0 (local_path, local_md5sum, archive_name, remote_path, timestamp, remote_md5sum, copied, removed, failed_copy_attempts, deletable, "ignore") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: ('/data/so/timestreams/17320/ufm_mv27/1732022318_000.g3', '1a55bc6a1b8b1dd01772dbb3fe68da7f', 'timestreams', '17320/ufm_mv27/1732022318_000.g3', 1732022353.3419797, None, None, None, 0, 1, 0)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
/usr/lib/python3.8/threading.py:932:_bootstrap_inner
/usr/lib/python3.8/threading.py:870:run
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_threadworker.py:49:work
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_team.py:192:doWork
--- <exception caught here> ---
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:269:inContext
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:285:<lambda>
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:117:callWithContext
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:82:callWithContext
/usr/local/lib/python3.8/dist-packages/ocs/ocs_agent.py:984:_running_wrapper
/usr/local/lib/python3.8/dist-packages/socs/agents/pysmurf_monitor/agent.py:210:run
/usr/lib/python3.8/contextlib.py:120:__exit__
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py:1173:_maker_context_manager
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/util.py:237:__exit__
/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py:70:__exit__
/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py:207:raise_
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/util.py:233:__exit__
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py:829:commit
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py:808:_prepare_impl
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py:3367:flush
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py:3507:_flush
/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py:70:__exit__
/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py:207:raise_
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py:3467:_flush
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py:456:execute
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py:630:execute
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py:245:save_obj
/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py:1238:_emit_insert_statements
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py:1631:_execute_20
/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py:325:_execute_on_connection
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py:1498:_execute_clauseelement
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py:1862:_execute_context
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py:2043:_handle_dbapi_exception
/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py:207:raise_
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py:1819:_execute_context
/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py:732:do_execute
]
2024-11-19T13:19:18+0000 run:0 Status is now "done".
2024-11-19T13:19:18+0000 New file: /data/smurf_data/20241119/ufm_mv27/1732022101/plots/1732022316_white_noise_summary.png
2024-11-19T13:19:19+0000 New file: /data/smurf_data/20241119/ufm_mv27/1732022101/plots/1732022316_fknee_summary.png
```</div>