Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UDP Stream cuts out after one second #19

Closed
frankSDeviation opened this issue Aug 17, 2020 · 6 comments
Closed

UDP Stream cuts out after one second #19

frankSDeviation opened this issue Aug 17, 2020 · 6 comments
Labels
bug Something isn't working question Further information is requested

Comments

@frankSDeviation
Copy link

frankSDeviation commented Aug 17, 2020

Hello, my name is Frank and i have been playing with your plugin for the last few weeks. I can successfully display my Basler camera on my host machine using xvimagesink. My goal is to be able to stream the cameras image over UDP with the lowest latency possible. After trying many different pipelines, I was finally able to send a UDP stream to another computer. The stream only comes up for about one second and then dies. This is the pipeline I am currently running:

gst-launch-1.0 -c -v pylonsrc imageformat=mono8 ! "video/x-raw,format=GRAY8" ! decodebin ! videoconvert ! x264enc tune=zerolatency ! rtph264pay ! udpsink host=192.168.0.44 port=5000

As you can see it is nothing too crazy. I ran a GST_DEBUG=pylonsrc:5 command and go these results:

GST_DEBUG=pylonsrc:5 gst-launch-1.0 -c -v pylonsrc imageformat=mono8 ! "video/x-raw,format=GRAY8" ! decodebin ! videoconvert ! x264enc tune=zerolatency ! rtph264pay ! udpsink host=192.168.0.44 port=5000
0:00:00.014948009 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:407:gst_pylonsrc_init:<GstPylonsrc@0x5627a37c38e0> Initialising defaults
0:00:00.014975256 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:489:gst_pylonsrc_set_property:<pylonsrc0> Setting a property.
0:00:00.018648026 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:895:gst_pylonsrc_get_caps:<pylonsrc0> Received a request for caps.
0:00:00.018658038 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:897:gst_pylonsrc_get_caps:<pylonsrc0> Could not send caps - no camera connected.
0:00:00.018672705 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:895:gst_pylonsrc_get_caps:<pylonsrc0> Received a request for caps.
0:00:00.018676089 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:897:gst_pylonsrc_get_caps:<pylonsrc0> Could not send caps - no camera connected.
0:00:00.018686528 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:895:gst_pylonsrc_get_caps:<pylonsrc0> Received a request for caps.
0:00:00.018692039 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:897:gst_pylonsrc_get_caps:<pylonsrc0> Could not send caps - no camera connected.
Setting pipeline to PAUSED ...
0:00:00.277341070 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:977:gst_pylonsrc_start:<pylonsrc0> pylonsrc: found 1 Basler device(s).
0:00:00.277381749 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:2224:pylonc_connect_camera:<pylonsrc0> Connecting to the camera...
0:00:00.453028250 25125 0x5627a3799530              pylonsrc gstpylonsrc.c:2269:pylonc_print_camera_info:<pylonsrc0> Status: Using camera "daA1600-60um" (serial number: 22720403, id: 0). Custom ID: None
0:00:00.453070418 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1067:gst_pylonsrc_start:<pylonsrc0> Setting horizontal binning to 1
0:00:00.466538411 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1070:gst_pylonsrc_start:<pylonsrc0> Setting vertical binning to 1
0:00:00.482873790 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1100:gst_pylonsrc_start:<pylonsrc0> Max resolution is 1600x1200.
0:00:00.487613002 25125 0x5627a3799530              pylonsrc gstpylonsrc.c:1129:gst_pylonsrc_start:<pylonsrc0> Setting resolution to 1600x1200.
0:00:00.487849165 25125 0x5627a3799530 WARN                pylonsrc gstpylonsrc.c:1141:gst_pylonsrc_start:<pylonsrc0> The camera doesn't seem to allow offset centering. Skipping...
0:00:00.490453327 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1193:gst_pylonsrc_start:<pylonsrc0> Flipping X: False, Flipping Y: False.
0:00:00.490547673 25125 0x5627a3799530              pylonsrc gstpylonsrc.c:1253:gst_pylonsrc_start:<pylonsrc0> Using Mono8 image format.
0:00:00.506502190 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1264:gst_pylonsrc_start:<pylonsrc0> Pixel is 8 bits large.
0:00:00.506571957 25125 0x5627a3799530 WARN                pylonsrc gstpylonsrc.c:1284:gst_pylonsrc_start:<pylonsrc0> The camera doesn't support test image mode.
0:00:00.506602598 25125 0x5627a3799530 WARN                pylonsrc gstpylonsrc.c:1305:gst_pylonsrc_start:<pylonsrc0> Camera does not support changing the readout mode.
0:00:00.506836614 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1311:gst_pylonsrc_start:<pylonsrc0> Limiting camera's bandwidth.
0:00:00.508356259 25125 0x5627a3799530 WARN                pylonsrc gstpylonsrc.c:1386:gst_pylonsrc_start:<pylonsrc0> This camera doesn't have any lightsource presets
0:00:00.508474011 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1393:gst_pylonsrc_start:<pylonsrc0> Disabling automatic exposure.
0:00:00.508996807 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1417:gst_pylonsrc_start:<pylonsrc0> Disabling automatic gain.
0:00:00.509485747 25125 0x5627a3799530 WARN                pylonsrc gstpylonsrc.c:1458:gst_pylonsrc_start:<pylonsrc0> This camera doesn't support automatic white balance.
0:00:00.509529365 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1529:gst_pylonsrc_start:<pylonsrc0> Using the auto profile currently saved on the device.
0:00:00.509560342 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1700:gst_pylonsrc_start:<pylonsrc0> This camera doesn't support adjusting colours. Skipping...
0:00:00.509579753 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1822:gst_pylonsrc_start:<pylonsrc0> This camera doesn't support transforming colours. Skipping...
0:00:00.509664973 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1833:gst_pylonsrc_start:<pylonsrc0> Exposure property not set, using the saved exposure setting.
0:00:00.509738944 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1845:gst_pylonsrc_start:<pylonsrc0> Setting gain to 0.00
0:00:00.511129385 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1857:gst_pylonsrc_start:<pylonsrc0> Setting black level to 0.00
0:00:00.512892869 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1866:gst_pylonsrc_start:<pylonsrc0> Setting gamma to 1.00
0:00:00.527165833 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1910:gst_pylonsrc_start:<pylonsrc0> Basler's PGI is not supported. Skipping.
0:00:00.527186000 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1914:gst_pylonsrc_start:<pylonsrc0> Setting trigger mode.
0:00:00.528561322 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:1958:gst_pylonsrc_start:<pylonsrc0> Using "FrameStart" trigger selector. Software trigger mode is Off.
0:00:00.532853935 25125 0x5627a3799530 WARN                pylonsrc gstpylonsrc.c:2039:gst_pylonsrc_start:<pylonsrc0> Couldn't determine link speed.
0:00:00.532874297 25125 0x5627a3799530 WARN                pylonsrc gstpylonsrc.c:2051:gst_pylonsrc_start:<pylonsrc0> Couldn't determine sensor readout time.
0:00:00.533073381 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:2061:gst_pylonsrc_start:<pylonsrc0> The resulting framerate is 60 fps.
0:00:00.533092392 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:2062:gst_pylonsrc_start:<pylonsrc0> Each frame is 1920000 bytes big (1.9 MB). That's 114.9MB/s.
0:00:00.535110592 25125 0x5627a3799530              pylonsrc gstpylonsrc.c:2076:gst_pylonsrc_start:<pylonsrc0> Initialised successfully.
Pipeline is live and does not need PREROLL ...
0:00:00.535302155 25125 0x5627a37d3630 DEBUG               pylonsrc gstpylonsrc.c:895:gst_pylonsrc_get_caps:<pylonsrc0> Received a request for caps.
0:00:00.535332798 25125 0x5627a37d3630 DEBUG               pylonsrc gstpylonsrc.c:934:gst_pylonsrc_get_caps:<pylonsrc0> The following caps were sent: video/x-raw, GRAY8, 1600x1200, variable fps.
0:00:00.535379742 25125 0x5627a37d3630 DEBUG               pylonsrc gstpylonsrc.c:945:gst_pylonsrc_set_caps:<pylonsrc0> Setting caps to video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstPylonsrc:pylonsrc0.GstPad:src: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
New clock: GstSystemClock
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1600, height=(int)1200, framerate=(fraction)0/1, format=(string)Y444
Redistribute latency...
/GstPipeline:pipeline0/GstX264Enc:x264enc0.GstPad:sink: caps = video/x-raw, width=(int)1600, height=(int)1200, framerate=(fraction)0/1, format=(string)Y444
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstDecodePad:src_0.GstProxyPad:proxypad1: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstDecodeBin:decodebin0.GstGhostPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1200, framerate=(fraction)0/1
0:00:00.604895874 25125 0x5627a37d3630 DEBUG               pylonsrc gstpylonsrc.c:895:gst_pylonsrc_get_caps:<pylonsrc0> Received a request for caps.
0:00:00.604931874 25125 0x5627a37d3630 DEBUG               pylonsrc gstpylonsrc.c:934:gst_pylonsrc_get_caps:<pylonsrc0> The following caps were sent: video/x-raw, GRAY8, 1600x1200, variable fps.
/GstPipeline:pipeline0/GstX264Enc:x264enc0.GstPad:src: caps = video/x-h264, codec_data=(buffer)01f40028ffe1001d67f40028919640190097b016a0202028000003000bb9aca00078c1924001000568ebcc4480, stream-format=(string)avc, alignment=(string)au, level=(string)4, profile=(string)high-4:4:4, width=(int)1600, height=(int)1200, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstRtpH264Pay:rtph264pay0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, profile-level-id=(string)f40028, sprop-parameter-sets=(string)"Z/QAKJGWQBkAl7AWoCAgKAAAAwALuaygAHjBkkA\=\,aOvMRIA\=", payload=(int)96, ssrc=(uint)1945388410, timestamp-offset=(uint)1437368984, seqnum-offset=(uint)32064
/GstPipeline:pipeline0/GstUDPSink:udpsink0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, profile-level-id=(string)f40028, sprop-parameter-sets=(string)"Z/QAKJGWQBkAl7AWoCAgKAAAAwALuaygAHjBkkA\=\,aOvMRIA\=", payload=(int)96, ssrc=(uint)1945388410, timestamp-offset=(uint)1437368984, seqnum-offset=(uint)32064
/GstPipeline:pipeline0/GstRtpH264Pay:rtph264pay0.GstPad:sink: caps = video/x-h264, codec_data=(buffer)01f40028ffe1001d67f40028919640190097b016a0202028000003000bb9aca00078c1924001000568ebcc4480, stream-format=(string)avc, alignment=(string)au, level=(string)4, profile=(string)high-4:4:4, width=(int)1600, height=(int)1200, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstRtpH264Pay:rtph264pay0: timestamp = 1437371943
/GstPipeline:pipeline0/GstRtpH264Pay:rtph264pay0: seqnum = 32064
0:00:01.356149721 25125 0x5627a37d3630 ERROR               pylonsrc gstpylonsrc.c:2136:gst_pylonsrc_create:<pylonsrc0> Error in the image processing loop.
ERROR: from element /GstPipeline:pipeline0/GstPylonsrc:pylonsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstPylonsrc:pylonsrc0:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.820767721
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
0:00:01.371119497 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:2154:gst_pylonsrc_stop:<pylonsrc0> stop
0:00:01.422230471 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:2200:pylonc_disconnect_camera:<pylonsrc0> Camera disconnected.
Setting pipeline to NULL ...
Freeing pipeline ...
0:00:01.423028394 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:2165:gst_pylonsrc_dispose:<pylonsrc0> dispose
0:00:01.423061434 25125 0x5627a3799530 DEBUG               pylonsrc gstpylonsrc.c:2173:gst_pylonsrc_finalize:<pylonsrc0> finalize

The error message that comes up is "Error in the image processing loop."
I have looked at this in issue #5 but that solution did not work for me. I did exactly as instructed in that thread but it did not fix my issue.

This is the pipeline I am running on the external machine to display the UDP stream. I used it to play a recorded video file from the Basler camera and it works fine.

gst-launch-1.0 -v udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink sync=false

I am not sure if this is the correct way of doing things.Any help will be greatly appreciated.

Thank you,
Frank

@zingmars
Copy link
Owner

zingmars commented Aug 18, 2020

Hi,
The problem with the "Error in the image processing loop" error is that it's very hard to diagnose. As you might've read in #5, it usually means that there was a corrupted frame present, and debugging it is quite hard. Now, when you say "I did exactly as instructed", do you mean you replaced the code block as shown in here #5 (comment)?

For debugging, first I would recommend you to use the pylonsrc from https://github.com/joshdoe/gst-plugins-vision (you can also try https://github.com/AB-Eskild/gst-plugins-vision/tree/bugfix/pylonsrc_handle_corupt_frames, which is a branch for a PR that attempts to tackle this issue). That repository has a slightly refactored version of plugin that should be better at showing where the error was (although it will still fail if a frame will fail to grab iirc).

One thing to try is to run the pipeline, and then open pylon viewer and run it for a while to see if there are any errors/warnings there. Basler cameras save settings in ram, and running the pipeline will set a lot of values to the plugin's defaults thus what you'll get should be very close to what it would be outputting to gstreamer. joshdoe/gst-plugins-vision#14 (comment) reports that changing framerate/resolution and other parameters seem to affect the frequency of any warnings/errors, so you might also want to try that.

@zingmars zingmars added bug Something isn't working question Further information is requested labels Aug 18, 2020
@frankSDeviation
Copy link
Author

Hello Zingmar,

Thank you for the response. The only thing I did in issue number 5 was change "goto error" to "return GST_FLOW_OK;" I was not to sure where to place that code block.

I am having issues using the vision plug ins. I am sure I've installed everything correctly but I cant be 100%. I am not sure if I am doing thigns correctly. After installing the plugin, I should just be able to run the pylonsrc plugin correct? or is there another way I am supposed to use the vision plug in?

I have ran the Basler pylonviewer after running the pipeline and everything is running just fine. I do not get any errors at all. I also want to mention that I am able to see the video stream locally by running this pipeline:

gst-launch-1.0 pylonsrc camera=0 imageformat=mono8 width=1600 height=1200 ! videoconvert ! xvimagesink

I will take a look at issue #14 in the vision plugin. I also would like to say that I am not a software engineer. My background is primarily hardware design so I do apologize if I may say some silly things or ask dumb questions regarding compiling or installing software.

Thank you,
Frank

@zingmars
Copy link
Owner

zingmars commented Aug 19, 2020

Try running the UDP pipeline with continuous=false set

I am having issues using the vision plug ins. I am sure I've installed everything correctly but I cant be 100%. I am not sure if I am doing thigns correctly. After installing the plugin, I should just be able to run the pylonsrc plugin correct? or is there another way I am supposed to use the vision plug in?

From what I remember you should be able to just make install it and run, though you'll probably need to uninstall this version first. Also, you might need to clean gstreamer cache (~/.gstreamer-1.0 iirc?)

Thank you for the response. The only thing I did in issue number 5 was change "goto error" to "return GST_FLOW_OK;" I was not to sure where to place that code block.

That probably didn't work because you told gstreamer that the plugin succeeded to grab a frame when it didn't actually provide a frame. I've uploaded two patch files that you can try - 0001-Retry-capture-when-a-frame-failed-to-grab.patch is the solution provided in #5. 0001-Grab-frame-after-failure.patch will make the plugin try another capture if it fails (although I haven't tested it). You can apply a patch file using git am --3way <patch file>.

Download: pylonsrc-patches.zip

@frankSDeviation
Copy link
Author

frankSDeviation commented Aug 20, 2020

Hello Zingmars,

Thank you for all of your help. I first want to say that I tried applying the patch but for some reason it just wouldn't take. After that I began to play with the gst-plugins-vision code and was finally able to stream a UDP stream with almost 0 latency! This is the pipeline I am using on my host machine:

gst-launch-1.0 -v pylonsrc camera=1 pixel-format=mono8 width=1600 height=1200 ! "video/x-raw,format=GRAY8" ! videoflip method=vertical-flip ! videoconvert ! x264enc bitrate=30000 speed-preset=superfast qp-min=30 tune=zerolatency ! rtph264pay ! udpsink host=XXX.XXX.XXX.XXX port=XXXX

And this is the pipeline I am running on the destination machine:

gst-launch-1.0 -v udpsrc port=XXXX caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! queue ! decodebin ! videoconvert ! videoscale ! autovideosink sync=true

I have another question, is it possible to send two udp streams from two different cameras at the same time? My application requires me to be able to two cameras running on the same machine. Also since this question was answered, should I start another issue and ask this question there?

thank you,
Frank

@zingmars
Copy link
Owner

zingmars commented Aug 21, 2020 via email

@frankSDeviation
Copy link
Author

Thanks again Zingmars! I am currently working on getting two video streams to be sent over UDP. I have been successful in doing this with the following pipline:

st-launch-1.0 pylonsrc camera=0 pixel-format=mono8 width=800 height=600 ! "video/x-raw,format=GRAY8" ! videoflip method=vertical-flip ! videoconvert ! x264enc bitrate=30000 speed-preset=superfast qp-min=30 tune=zerolatency ! rtph264pay ! udpsink host=XXX.XXX.XXX port=XXXX | gst-launch-1.0 pylonsrc camera=1 pixel-format=mono8 width=800 height=600 ! "video/x-raw,format=GRAY8" ! videoflip method=vertical-flip ! videoconvert ! x264enc bitrate=30000 speed-preset=superfast qp-min=30 tune=zerolatency ! rtph264pay ! udpsink host=XXX.XXX.XXX port=XXXX

I will start playing with the videomixer to see if I can send them over UDP in one window. Thanks a ton for the help. I will close this issue now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants