Error in the image processing loop

[mkaivs]

I have a pipeline that ran for hours before this error occurred.

66:52:26.688001436 11645 0x564a34d30590 ERROR               pylonsrc gstpylonsrc.c:2204:gst_pylonsrc_create:<camera> Error in the image processing loop.
66:52:26.688060908 11645 0x564a34d30590 INFO                 basesrc gstbasesrc.c:2839:gst_base_src_loop:<camera> pausing after gst_base_src_get_range() = error
66:52:26.688103115 11645 0x564a34d30590 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<camera> error: Internal data stream error.
66:52:26.688124190 11645 0x564a34d30590 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<camera> error: streaming stopped, reason error (-5)
66:52:26.688175968 11645 0x564a34d30590 INFO        GST_ERROR_SYSTEM gstelement.c:2145:gst_element_message_full_with_details:<camera> posting message: Internal data stream error.
66:52:26.688271008 11645 0x564a34d30590 INFO        GST_ERROR_SYSTEM gstelement.c:2172:gst_element_message_full_with_details:<camera> posted error message: Internal data stream error.
ERROR from element camera: Internal data stream error.
66:52:26.688349520 11645 0x564a34d30590 INFO                    task gsttask.c:316:gst_task_func:<camera:src> Task going to paused
Error details: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:ds-custom-pipeline/GstBin:source-bin-00/GstPylonsrc:camera:
streaming stopped, reason error (-5)
Returned, stopping playback

I looked into the source code and the error originated from this function:

static GstFlowReturn gst_pylonsrc_create (GstPushSrc *src, GstBuffer **buf)
{  
  GstPylonsrc *pylonsrc = GST_PYLONSRC (src);
  GENAPIC_RESULT res;
  size_t bufferIndex;
  PylonGrabResult_t grabResult;
  _Bool bufferReady;  
  GstMapInfo mapInfo;

  // Wait for the buffer to be filled  (up to 1 s)  
  res = PylonWaitObjectWait(pylonsrc->waitObject, 300000, &bufferReady);
  PYLONC_CHECK_ERROR(pylonsrc, res);
  if(!bufferReady) {
    GST_MESSAGE_OBJECT(pylonsrc, "Camera couldn't prepare the buffer in time. Probably dead.");    
    goto error;
  }

  res = PylonStreamGrabberRetrieveResult (pylonsrc->streamGrabber, &grabResult, &bufferReady);
  PYLONC_CHECK_ERROR(pylonsrc, res);
  if(!bufferReady) {
    GST_MESSAGE_OBJECT(pylonsrc, "Couldn't get a buffer from the camera. Basler said this should be impossible. You just proved them wrong. Congratulations!");    
    goto error;
  }

if(!pylonsrc->continuousMode) {
     // Trigger the next picture while we process this one
    if(PylonDeviceFeatureIsAvailable(pylonsrc->deviceHandle, "AcquisitionStatus")) {
    _Bool isReady = FALSE;
     do {
       res = PylonDeviceGetBooleanFeature(pylonsrc->deviceHandle, "AcquisitionStatus", &isReady);
       PYLONC_CHECK_ERROR(pylonsrc, res);
     } while (!isReady);
   }
    res = PylonDeviceExecuteCommandFeature(pylonsrc->deviceHandle, "TriggerSoftware");
   PYLONC_CHECK_ERROR(pylonsrc, res);
  }

  // Process the current buffer
  bufferIndex = (size_t) grabResult.Context;
  if(grabResult.Status == Grabbed) {        
    //TODO: See if I can avoid memcopy and record directly into the gst buffer map.

    // Copy the image into the buffer that will be passed onto the next GStreamer element
    *buf = gst_buffer_new_and_alloc(pylonsrc->payloadSize);
    gst_buffer_map(*buf, &mapInfo, GST_MAP_WRITE);
    orc_memcpy(mapInfo.data, grabResult.pBuffer, mapInfo.size);
    gst_buffer_unmap(*buf, &mapInfo);        

    // Release frame's memory
    res = PylonStreamGrabberQueueBuffer( pylonsrc->streamGrabber, grabResult.hBuffer, (void*) bufferIndex );
    PYLONC_CHECK_ERROR(pylonsrc, res);
  } else {
    GST_ERROR_OBJECT(pylonsrc, "Error in the image processing loop.");    
    goto error;
  }

I'm not sure what this error means and how to start fixing it. Please give me some explanation and pointers to help me deal with this error.

[mrstecklo]

@mkaivs, you are using an outdated code. Try pulling newest version. Also GST_DEBUG=2,pylonsrc:6 environment variable helps understanding errors

[mkaivs]

@mrstecklo, Thanks, I'm installing the newest code now. I will report back.

[MhdKAT]

Any updates on this @mkaivs ? I am having same issue

[joshdoe]

I didn't write this section of code, but have some guesses. It could be that there were missing packets due to network/system overloading, generating an incomplete buffer. To help identify this, I would add to that error statement the value of the grab result. If it's due to incomplete buffers, an element property could be added to ignore a certain number of incomplete buffers, or possibly fill in missing data from the previous frame. Do either of you have a way to reliably recreate this error condition?

[MhdKAT]

Thanks for the reply @joshdoe. Actually, I don't know if it is reproducible but I am getting this error when running this simple pipeline on Adlink Neon Vision Camera which has a basler acA2500-14uc USB3 camera connected to Jetson TX2 board.
Pipeline : gst-launch-1.0 -v pylonsrc ! fpsfilter ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! nvegltransform ! nveglglessink sync=0. I have tried contacting Nvidia but according to them it is a camera "instability".

[thiesmoeller]

Hi @MhdKAT,

have you applied the recommendations in the installation readme in the pylon for Arm64 package?

e.g. Increase usbfs memory, increase transfer size in camera.

If not you generate a very high rate of smaller transfers, which could overload the host controller and can result in issues you observe

[MhdKAT]

Thank you @thiesmoeller I checked that usbfs was increased as in the installation recommandations' readme.
Transfer size in the camera defaults to 65536. Is there a way to change it through plugin params ? I am currently setting the camera params through a saved config file but I don't think that param gets saved in that pfs file.

[mrstecklo]

@MhdKAT, every parameter that is displayed active in Pylon Viewer will be saved in PFS file. You can open PFS with text editor if you are unsure.

[MhdKAT]

Thank you @mrstecklo but I don't think it's there . Here is my PFS file :
GainSelector All
GainRaw 8
GainSelector All
GainSelector All
GainAuto Off
GainSelector All
GainSelector All
AutoGainLowerLimitRaw 8
GainSelector All
GainSelector All
AutoGainUpperLimitRaw 127
GainSelector All
BlackLevelSelector All
BlackLevelRaw 0
BlackLevelSelector All
GammaRaw 1000
BslColorSpaceMode sRGB
Width 1296
Height 972
OffsetX 0
OffsetY 0
BinningHorizontalMode Average
BinningHorizontal 2
BinningVerticalMode Average
BinningVertical 2
ReverseX 0
ReverseY 0
PixelFormat YCbCr422_8
TestPattern Off
SensorShutterMode Rolling
OverlapMode On
BslImmediateTriggerMode Off
ExposureTimeRaw 22677
ExposureAuto Off
AutoExposureTimeLowerLimitRaw 100
AutoExposureTimeUpperLimitRaw 100000
TriggerSelector FrameStart
TriggerMode Off
TriggerSelector FrameStart
TriggerSelector FrameStart
TriggerSource Software
TriggerSelector FrameStart
TriggerSelector FrameStart
TriggerActivation RisingEdge
TriggerSelector FrameStart
ExposureMode Timed
AcquisitionFramePeriodRaw 10
LightSourcePreset Daylight5000K
BalanceRatioSelector Red
BalanceRatioRaw 64
BalanceRatioSelector Green
BalanceRatioRaw 77
BalanceRatioSelector Blue
BalanceRatioRaw 103
BalanceRatioSelector Red
BalanceWhiteAuto Off
BslHueValue 0
BslSaturationValueRaw 1000
ContrastEnhancementRaw 0
BslContrastMode Linear
BslBrightnessRaw 0
BslContrastRaw 0
PgiMode On_AutomaticNoiseReduction
SharpnessEnhancementRaw 0
DefectPixelCorrectionMode On
LineSelector Line1
LineMode Input
LineSelector Line2
LineMode Input
LineSelector Line1
LineSelector Line1
LineInverter 0
LineSelector Line2
LineInverter 0
LineSelector Line1
LineSelector Line1
LineSource UserOutput1
LineSelector Line2
LineSource UserOutput2
LineSelector Line1
LineSelector Line1
LineDebouncerTimeRaw 0
LineSelector Line2
LineDebouncerTimeRaw 0
LineSelector Line1
UserOutputSelector UserOutput1
UserOutputValue 0
UserOutputSelector UserOutput2
UserOutputValue 0
UserOutputSelector UserOutput1
AutoTargetBrightnessRaw 30
AutoTargetBrightnessDampingRaw 3
AutoBacklightCompensationRaw 0
AutoFunctionProfile Smart
AutoFunctionROISelector ROI1
AutoFunctionROIWidth 1296
AutoFunctionROISelector ROI2
AutoFunctionROIWidth 1296
AutoFunctionROISelector ROI1
AutoFunctionROISelector ROI1
AutoFunctionROIHeight 972
AutoFunctionROISelector ROI2
AutoFunctionROIHeight 972
AutoFunctionROISelector ROI1
AutoFunctionROISelector ROI1
AutoFunctionROIOffsetX 0
AutoFunctionROISelector ROI2
AutoFunctionROIOffsetX 0
AutoFunctionROISelector ROI1
AutoFunctionROISelector ROI1
AutoFunctionROIOffsetY 0
AutoFunctionROISelector ROI2
AutoFunctionROIOffsetY 0
AutoFunctionROISelector ROI1
DeviceLinkSelector 0
DeviceLinkThroughputLimitMode On
DeviceLinkSelector 0
DeviceLinkSelector 0
DeviceLinkThroughputLimit 163000000
DeviceLinkSelector 1
DeviceLinkThroughputLimit 0
DeviceLinkSelector 0
DeviceIndicatorMode Active

[mrstecklo]

@MhdKAT. Try adjusting Transfer Size a bit in Pylon Viewer. Probably some defaults are not exported to PFS. When property is represented in PFS it can be manually edited with text editor

[thiesmoeller]

It is not in the camera node map, but in the streamgrabber nodemap. It is a config param of the grab enginem

In pseudo code:

cam.StreamgrabberNodemap.MaxTransferSize = 4 * 1024 * 1024

[thiesmoeller]

I added the feature in my fork ... ( https://github.com/thiesmoeller/gst-plugins-vision/tree/feature/gst-pylonsrc_add_usb3_maxtransfersize )
You can now call to get e.g. 4MB transfers like described above:

gst-launch-1.0 pylonsrc camera=0 max-transfer-size=4194304 ! videoconvert ! autovideosink