Linux IOCs is failing after some days when running USB camera at 163 frames/s

[MarkRivers]

I am seeing ADSpinnaker IOCs fail after 1-3 days when running a USB camera at 163 frames/s.

The following is the output of procServ for 2 occurrences.

In this one the IOC completed booting on June 9 at 10:47 AM. On June 10 at 11:53 acquisition was started. 3 days later on June 13 we see a bunch of warnings that the scan tasks are running too slowly. There is then a "double free or corruption" error and the IOC crashes and automatically reboots.

Tue Jun  9 10:47:30 2020epics>
Tue Jun  9 10:47:30 2020epics> 2020/06/09 10:47:30.740 Param[BIN_X] GenICamFeature::write: node BinningHorizontal is not writable
Tue Jun  9 10:47:30 2020
Tue Jun  9 10:47:31 2020auto_settings.sav: 2874 of 2874 PV's connected
Tue Jun  9 10:47:31 2020
Tue Jun  9 10:47:32 2020epics>
Tue Jun  9 10:47:32 2020epics> 2020/06/10 11:52:47.397 ADSpinnaker::grabImage [PG1] ERROR: pData is NULL!
Wed Jun 10 11:53:02 20202020/06/10 11:53:02.024 Param[GC_FRAMERATE] GenICamFeature::write: node AcquisitionFrameRate is not writable
Wed Jun 10 11:53:02 2020
Sat Jun 13 10:31:27 2020
Sat Jun 13 10:31:27 2020dbScan warning from '.1 second' scan thread:
Sat Jun 13 10:31:27 2020        Scan processing averages 0.757 seconds (0.154 .. 1.276).
Sat Jun 13 10:31:27 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:31:27 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:32:21 2020
Sat Jun 13 10:32:21 2020dbScan warning from '.1 second' scan thread:
Sat Jun 13 10:32:21 2020        Scan processing averages 0.492 seconds (0.185 .. 0.995).
Sat Jun 13 10:32:21 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:32:21 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:32:26 2020
Sat Jun 13 10:32:26 2020dbScan warning from '.2 second' scan thread:
Sat Jun 13 10:32:26 2020        Scan processing averages 0.835 seconds (0.334 .. 1.583).
Sat Jun 13 10:32:26 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:32:26 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:32:50 2020
Sat Jun 13 10:32:50 2020dbScan warning from '.1 second' scan thread:
Sat Jun 13 10:32:50 2020        Scan processing averages 0.563 seconds (0.125 .. 0.997).
Sat Jun 13 10:32:50 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:32:50 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:32:57 2020
Sat Jun 13 10:32:57 2020dbScan warning from '.2 second' scan thread:
Sat Jun 13 10:32:57 2020        Scan processing averages 0.493 seconds (0.242 .. 0.754).
Sat Jun 13 10:32:57 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:32:57 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:33:32 2020
Sat Jun 13 10:33:32 2020dbScan warning from '.1 second' scan thread:
Sat Jun 13 10:33:32 2020        Scan processing averages 0.772 seconds (0.343 .. 1.403).
Sat Jun 13 10:33:32 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:33:32 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:33:33 2020
Sat Jun 13 10:33:33 2020dbScan warning from '.2 second' scan thread:
Sat Jun 13 10:33:33 2020        Scan processing averages 0.772 seconds (0.293 .. 1.488).
Sat Jun 13 10:33:33 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:33:33 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:34:15 2020
Sat Jun 13 10:34:15 2020dbScan warning from '.2 second' scan thread:
Sat Jun 13 10:34:15 2020        Scan processing averages 0.487 seconds (0.211 .. 0.824).
Sat Jun 13 10:34:15 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:34:15 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:34:16 2020
Sat Jun 13 10:34:16 2020dbScan warning from '.1 second' scan thread:
Sat Jun 13 10:34:16 2020        Scan processing averages 0.549 seconds (0.165 .. 0.874).
Sat Jun 13 10:34:16 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:34:16 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:34:22 2020
Sat Jun 13 10:34:22 2020dbScan warning from '.5 second' scan thread:
Sat Jun 13 10:34:22 2020        Scan processing averages 0.885 seconds (0.603 .. 1.599).
Sat Jun 13 10:34:22 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:34:22 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:34:55 2020
Sat Jun 13 10:34:55 2020dbScan warning from '.1 second' scan thread:
Sat Jun 13 10:34:55 2020        Scan processing averages 0.593 seconds (0.185 .. 1.282).
Sat Jun 13 10:34:55 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:34:55 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:35:03 2020
Sat Jun 13 10:35:03 2020dbScan warning from '.2 second' scan thread:
Sat Jun 13 10:35:03 2020        Scan processing averages 0.646 seconds (0.206 .. 1.197).
Sat Jun 13 10:35:03 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:35:03 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:35:08 2020
Sat Jun 13 10:35:08 2020dbScan warning from '.5 second' scan thread:
Sat Jun 13 10:35:08 2020        Scan processing averages 1.019 seconds (0.583 .. 1.395).
Sat Jun 13 10:35:08 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:35:08 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:35:14 2020
Sat Jun 13 10:35:14 2020dbScan warning from '.1 second' scan thread:
Sat Jun 13 10:35:14 2020        Scan processing averages 0.592 seconds (0.108 .. 1.205).
Sat Jun 13 10:35:14 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:35:14 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:35:23 2020
Sat Jun 13 10:35:23 2020dbScan warning from '.2 second' scan thread:
Sat Jun 13 10:35:23 2020        Scan processing averages 0.853 seconds (0.228 .. 1.648).
Sat Jun 13 10:35:23 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:35:23 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:35:26 2020
Sat Jun 13 10:35:26 2020dbScan warning from '.5 second' scan thread:
Sat Jun 13 10:35:26 2020        Scan processing averages 1.061 seconds (0.504 .. 1.491).
Sat Jun 13 10:35:26 2020        Over-runs have now happened 10 times in a row.
Sat Jun 13 10:35:26 2020        To fix this, move some records to a slower scan rate.
Sat Jun 13 10:35:32 2020double free or corruption (out)
Sat Jun 13 10:41:03 2020/corvette/home/epics/support/CARS/iocBoot/ioc13ida_pg1/start_epics: line 4: 16762 Aborted                 (core dumped) /corvette/home/epics/support/areaDetector/ADSpinnaker/iocs/spinnakerIOC/bin/linux-x86_64-centos8/spinnakerApp st.cmd

On June 16 the same scan tasks errors happened. I think the IOC may have been restarted by the user in this case, rather than crashing. But something is still going wrong.

Sat Jun 13 10:41:09 2020# These PVs are in auto_settings.req but don't have PINI=Yes
Sat Jun 13 10:41:09 2020dbpf("13IDAPG1:cam1:GC_OnBoaColProEna.PROC", "1")
Sat Jun 13 10:41:09 2020DBF_UCHAR:          1 = 0x1
Sat Jun 13 10:41:09 2020dbpf("13IDAPG1:cam1:GC_VideoMode.PROC", "1")
Sat Jun 13 10:41:09 2020DBF_UCHAR:          1 = 0x1
Sat Jun 13 10:41:09 2020dbpf("13IDAPG1:cam1:GC_DevLinThrLimit.PROC", "1")
Sat Jun 13 10:41:09 2020DBF_UCHAR:          1 = 0x1
Sat Jun 13 10:41:09 2020epics> 2020/06/13 10:41:10.025 Param[BIN_X] GenICamFeature::write: node BinningHorizontal is not writable
Sat Jun 13 10:41:10 2020
Sat Jun 13 10:41:10 2020auto_settings.sav: 2875 of 2875 PV's connected
Mon Jun 15 10:39:59 20202020/06/15 10:39:59.692 ADSpinnaker::grabImage [PG1] ERROR: pData is NULL!
Tue Jun 16 11:21:30 2020
Tue Jun 16 11:21:30 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:21:30 2020        Scan processing averages 0.536 seconds (0.316 .. 0.881).
Tue Jun 16 11:21:30 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:21:30 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:21:32 2020
Tue Jun 16 11:21:32 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:21:32 2020        Scan processing averages 0.491 seconds (0.266 .. 0.831).
Tue Jun 16 11:21:32 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:21:32 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:21:46 2020
Tue Jun 16 11:21:46 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:21:46 2020        Scan processing averages 0.650 seconds (0.272 .. 1.153).
Tue Jun 16 11:21:46 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:21:46 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:21:47 2020
Tue Jun 16 11:21:47 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:21:47 2020        Scan processing averages 0.764 seconds (0.504 .. 1.203).
Tue Jun 16 11:21:47 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:21:47 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:21:47 2020
Tue Jun 16 11:21:47 2020dbScan warning from '.5 second' scan thread:
Tue Jun 16 11:21:47 2020        Scan processing averages 0.874 seconds (0.550 .. 1.609).
Tue Jun 16 11:21:47 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:21:47 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:22:07 2020
Tue Jun 16 11:22:07 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:22:07 2020        Scan processing averages 0.530 seconds (0.148 .. 0.734).
Tue Jun 16 11:22:07 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:22:07 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:22:21 2020
Tue Jun 16 11:22:21 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:22:21 2020        Scan processing averages 0.523 seconds (0.237 .. 1.008).
Tue Jun 16 11:22:21 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:22:21 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:22:26 2020
Tue Jun 16 11:22:26 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:22:26 2020        Scan processing averages 0.600 seconds (0.233 .. 1.410).
Tue Jun 16 11:22:26 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:22:26 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:22:53 2020
Tue Jun 16 11:22:53 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:22:53 2020        Scan processing averages 1.052 seconds (0.383 .. 1.800).
Tue Jun 16 11:22:53 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:22:53 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:22:53 2020
Tue Jun 16 11:22:53 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:22:53 2020        Scan processing averages 1.090 seconds (0.317 .. 1.850).
Tue Jun 16 11:22:53 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:22:53 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:22:57 2020
Tue Jun 16 11:22:57 2020dbScan warning from '.5 second' scan thread:
Tue Jun 16 11:22:57 2020        Scan processing averages 1.359 seconds (0.529 .. 3.146).
Tue Jun 16 11:22:57 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:22:57 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:23:24 2020
Tue Jun 16 11:23:24 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:23:24 2020        Scan processing averages 0.719 seconds (0.170 .. 1.880).
Tue Jun 16 11:23:24 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:23:24 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:23:29 2020
Tue Jun 16 11:23:29 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:23:29 2020        Scan processing averages 0.778 seconds (0.345 .. 1.239).
Tue Jun 16 11:23:29 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:23:29 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:23:54 2020
Tue Jun 16 11:23:54 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:23:54 2020        Scan processing averages 0.649 seconds (0.201 .. 1.800).
Tue Jun 16 11:23:54 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:23:54 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:24:07 2020
Tue Jun 16 11:24:07 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:24:07 2020        Scan processing averages 0.544 seconds (0.215 .. 1.001).
Tue Jun 16 11:24:07 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:24:07 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:24:36 2020
Tue Jun 16 11:24:36 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:24:36 2020        Scan processing averages 1.222 seconds (0.456 .. 1.580).
Tue Jun 16 11:24:36 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:24:36 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:24:36 2020
Tue Jun 16 11:24:36 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:24:36 2020        Scan processing averages 1.274 seconds (0.150 .. 2.574).
Tue Jun 16 11:24:36 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:24:36 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:24:37 2020
Tue Jun 16 11:24:37 2020dbScan warning from '.5 second' scan thread:
Tue Jun 16 11:24:37 2020        Scan processing averages 1.424 seconds (0.717 .. 2.374).
Tue Jun 16 11:24:37 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:24:37 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:24:49 2020
Tue Jun 16 11:24:49 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:24:49 2020        Scan processing averages 0.377 seconds (0.211 .. 0.602).
Tue Jun 16 11:24:49 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:24:49 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:25:17 2020
Tue Jun 16 11:25:17 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:25:17 2020        Scan processing averages 0.940 seconds (0.437 .. 2.260).
Tue Jun 16 11:25:17 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:25:17 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:25:21 2020
Tue Jun 16 11:25:21 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:25:21 2020        Scan processing averages 0.980 seconds (0.487 .. 2.310).
Tue Jun 16 11:25:21 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:25:21 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:25:21 2020
Tue Jun 16 11:25:21 2020dbScan warning from '.5 second' scan thread:
Tue Jun 16 11:25:21 2020        Scan processing averages 1.303 seconds (0.881 .. 1.734).
Tue Jun 16 11:25:21 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:25:21 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:25:35 2020
Tue Jun 16 11:25:35 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:25:35 2020        Scan processing averages 0.972 seconds (0.373 .. 1.899).
Tue Jun 16 11:25:35 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:25:35 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:25:35 2020
Tue Jun 16 11:25:35 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:25:35 2020        Scan processing averages 1.025 seconds (0.423 .. 1.949).
Tue Jun 16 11:25:35 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:25:35 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:25:57 2020
Tue Jun 16 11:25:57 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:25:57 2020        Scan processing averages 0.752 seconds (0.304 .. 1.125).
Tue Jun 16 11:25:57 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:25:57 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:25:57 2020
Tue Jun 16 11:25:57 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:25:57 2020        Scan processing averages 0.791 seconds (0.386 .. 1.175).
Tue Jun 16 11:25:57 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:25:57 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:26:06 2020
Tue Jun 16 11:26:06 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:26:06 2020        Scan processing averages 0.396 seconds (0.194 .. 0.891).
Tue Jun 16 11:26:06 2020        Over-runs have now happened 11 times in a row.
Tue Jun 16 11:26:06 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:26:23 2020
Tue Jun 16 11:26:23 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:26:23 2020        Scan processing averages 0.614 seconds (0.193 .. 1.286).
Tue Jun 16 11:26:23 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:26:23 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:26:39 2020
Tue Jun 16 11:26:39 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:26:39 2020        Scan processing averages 0.331 seconds (0.151 .. 0.902).
Tue Jun 16 11:26:39 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:26:39 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:26:50 2020
Tue Jun 16 11:26:50 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:26:50 2020        Scan processing averages 0.351 seconds (0.140 .. 0.677).
Tue Jun 16 11:26:50 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:26:50 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:27:05 2020
Tue Jun 16 11:27:05 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:27:05 2020        Scan processing averages 0.464 seconds (0.112 .. 0.869).
Tue Jun 16 11:27:05 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:27:05 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:27:06 2020
Tue Jun 16 11:27:06 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:27:06 2020        Scan processing averages 0.456 seconds (0.314 .. 0.683).
Tue Jun 16 11:27:06 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:27:06 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:27:20 2020
Tue Jun 16 11:27:20 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:27:20 2020        Scan processing averages 0.585 seconds (0.143 .. 1.159).
Tue Jun 16 11:27:20 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:27:20 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:27:22 2020
Tue Jun 16 11:27:22 2020dbScan warning from '.5 second' scan thread:
Tue Jun 16 11:27:22 2020        Scan processing averages 0.984 seconds (0.603 .. 1.690).
Tue Jun 16 11:27:22 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:27:22 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:27:23 2020
Tue Jun 16 11:27:23 2020dbScan warning from '.2 second' scan thread:
Tue Jun 16 11:27:23 2020        Scan processing averages 0.588 seconds (0.243 .. 1.110).
Tue Jun 16 11:27:23 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:27:23 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:27:39 2020
Tue Jun 16 11:27:39 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:27:39 2020        Scan processing averages 0.303 seconds (0.101 .. 0.541).
Tue Jun 16 11:27:39 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:27:39 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:27:54 2020
Tue Jun 16 11:27:54 2020dbScan warning from '.1 second' scan thread:
Tue Jun 16 11:27:54 2020        Scan processing averages 0.414 seconds (0.100 .. 0.794).
Tue Jun 16 11:27:54 2020        Over-runs have now happened 10 times in a row.
Tue Jun 16 11:27:54 2020        To fix this, move some records to a slower scan rate.
Tue Jun 16 11:28:17 2020/corvette/home/epics/support/CARS/iocBoot/ioc13ida_pg1/start_epics: line 4: 18108 Killed                  /corvette/home/epics/support/areaDetector/ADSpinnaker/iocs/spinnakerIOC/bin/linux-x86_64-centos8/spinnakerApp st.cmd
Tue Jun 16 11:28:19 2020
Tue Jun 16 11:28:20 2020@@@ @@@ @@@ @@@ @@@
Tue Jun 16 11:28:20 2020@@@ Received a sigChild for process 18091. Normal exit status = 137
Tue Jun 16 11:28:20 2020@@@ Current time: Tue Jun 16 11:28:20 2020
Tue Jun 16 11:28:20 2020@@@ Child process is shutting down, a new one will be restarted shortly
Tue Jun 16 11:28:20 2020@@@ ^R or ^X restarts the child, ^Q quits the server
Tue Jun 16 11:28:20 2020@@@ Restarting child "13IDA_PG1_IOC"
Tue Jun 16 11:28:20 2020@@@    (as /corvette/home/epics/support/CARS/iocBoot/ioc13ida_pg1/start_epics)
Tue Jun 16 11:28:21 2020@@@ The PID of new child "13IDA_PG1_IOC" is: 10523
Tue Jun 16 11:28:21 2020@@@ @@@ @@@ @@@ @@@
Tue Jun 16 11:28:22 2020< envPaths
Tue Jun 16 11:28:22 2020epicsEnvSet("IOC","ioc13ida_pg1")
Tue Jun 16 11:28:22 2020epicsEnvSet("TOP","/corvette/home/epics/support/CARS")
Tue Jun 16 11:28:22 2020epicsEnvSet("SUPPORT","/corvette/home/epics/support")

[MarkRivers]

Watching the IOC task with "top" does not show any memory leak, the virtual memory is not changing.