Raspberry Pi requires multi-planar buffer type #15

FallingSnow · 2020-11-12T05:24:06Z

Raspberry Pi only supports the multi-planar API.

Could we get another method (or modify the current v4l::output::device::Device.enum_formats method) that allows us to specify the buffer type?

We need to allow

libv4l-rs/src/output/device.rs

Line 97 in 2ab400d

v4l2_fmt.type_ = v4l2_buf_type_V4L2_BUF_TYPE_VIDEO_OUTPUT;

to be

 v4l2_fmt.type_ = v4l2_buf_type_V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;

Reference v4l::buffer::BufferType::VideoOutputMplane.

The text was updated successfully, but these errors were encountered:

FallingSnow · 2020-11-12T05:25:34Z

I should also point out that this may only be true on Raspberry Pi 4. That is what I'm currently testing on.

raymanfx · 2020-11-12T08:43:19Z

Hi, I'm very interested in getting multi-planar format support going. Unfortunately I don't have a camera device which supports/needs it. What device are you testing this with? I'm pretty sure standard USB webcams will work using the single-planar (aka packed) v4l2 layer even on the RPi 4, no?

From your first comment I read that you need multi-planar support for the output device, is that correct? Maybe we can tackle it for input and output at the same time even.

Apart from the format change, I can envision some more edits to be required, e.g. a new Buffer struct to represent planar buffers. Perhaps using a data structure like Vec<&[u8]>.

FallingSnow · 2020-11-12T09:04:51Z

I'm using /dev/video10 on a raspberry pi 4 (kernel: 5.4) in an attempt to decode h264. It seems this device only supports multi-planar buffers. With the v4l2_buf_type_V4L2_BUF_TYPE_VIDEO_OUTPUT buffer type you get Invalid Argument errors.

Using v4l2-ctl I still haven't been able to get a h264 stream decoded, keep running into DECODER STOP messages.

However it seems I've taken a few steps back. I tried to install aarch64 arch linux (kernel: 4.8.9) and now I don't get any v4l2 devices at all.

Anyway,

From your first comment I read that you need multi-planar support for the output device, is that correct? Maybe we can tackle it for input and output at the same time even.

Yes. If I can help I'll try.

Apart from the format change, I can envision some more edits to be required, e.g. a new Buffer struct to represent planar buffers. Perhaps using a data structure like Vec<&[u8]>.

I'm very new to this space and I'm not really even sure what multi-planar is. If I understand correctly it's buffers for a device that needs a discontiguous buffer? (going off https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/planar-apis.html#planar-apis)

raymanfx · 2020-11-12T10:20:11Z

I'm using /dev/video10 on a raspberry pi 4 (kernel: 5.4) in an attempt to decode h264. It seems this device only supports multi-planar buffers. With the v4l2_buf_type_V4L2_BUF_TYPE_VIDEO_OUTPUT buffer type you get Invalid Argument errors.

Oh I see. So you're not working with actual video I/O devices, but trying to use the hardware memory-to-memory codec on the RPi (which happens to be implemented as a v4l2 module).

I'm very new to this space and I'm not really even sure what multi-planar is. If I understand correctly it's buffers for a device that needs a discontiguous buffer? (going off https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/planar-apis.html#planar-apis)

Yes, that's about right. Decoding a video frame will usually give you "raw" frames, probably some YUV format instead of the more familiar RGB formats. YUV data captures luminance and chrominance. While YUV pixels can be packed, video codecs usually prefer them to be laid out in memory planes, e.g. one plane for the luminance (Y) and one for the chrominance components (UV).

raymanfx · 2020-11-13T22:30:41Z

FYI I'm planning to tackle this tomorrow. Are you available for testing this weekend?

FallingSnow · 2020-11-14T18:46:07Z

Yeah, I should be free sometime this weekend to test. Sorry, meant to reply sooner. Went out to buy a C7000, gonna try setting up virtual desktops, encode their screens, and decode on Pi's.

I got the PI back to having the v4l2 devices online.

Would also note that the PI seems to fail v4l2-compliance tests.

$ sudo v4l2-compliance -d /dev/video10  --stream-from output.h264 --streaming

v4l2-compliance SHA: not available
, 32 bits, 32-bit time_t

Compliance test for bcm2835-codec device /dev/video10:

Driver Info:
	Driver name      : bcm2835-codec
	Card type        : bcm2835-codec-decode
	Bus info         : platform:bcm2835-codec
	Driver version   : 5.4.51
	Capabilities     : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : bcm2835-codec
	Model            : bcm2835-codec
	Serial           : 0000
	Bus info         : platform:bcm2835-codec
	Media version    : 5.4.51
	Hardware revision: 0x00000001 (1)
	Driver version   : 5.4.51
Interface Info:
	ID               : 0x0300000c
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : bcm2835-codec-decode-source
	Function         : V4L2 I/O
	Pad 0x01000002   : 0: Source
	  Link 0x02000008: to remote pad 0x1000004 of entity 'bcm2835-codec-decode-proc': Data, Enabled, Immutable

Required ioctls:
	test MC information (see 'Media Driver Info' above): OK
		fail: v4l2-compliance.cpp(694): found_bit
	test VIDIOC_QUERYCAP: FAIL

Allow for multiple opens:
	test second /dev/video10 open: OK
		fail: v4l2-compliance.cpp(694): found_bit
	test VIDIOC_QUERYCAP: FAIL
	test VIDIOC_G/S_PRIORITY: OK
	test for unlimited opens: OK

	test invalid ioctls: OK
Debug ioctls:
	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
	test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
	test VIDIOC_ENUMAUDIO: OK (Not Supported)
	test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDIO: OK (Not Supported)
	Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
	Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
	test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
	test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
	test VIDIOC_QUERYCTRL: OK
	test VIDIOC_G/S_CTRL: OK
	test VIDIOC_G/S/TRY_EXT_CTRLS: OK
	test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
	test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
	Standard Controls: 2 Private Controls: 0

Format ioctls:
	test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
	test VIDIOC_G/S_PARM: OK (Not Supported)
	test VIDIOC_G_FBUF: OK (Not Supported)
	test VIDIOC_G_FMT: OK
	test VIDIOC_TRY_FMT: OK
	test VIDIOC_S_FMT: OK
	test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
	test Cropping: OK (Not Supported)
	test Composing: OK
	test Scaling: OK (Not Supported)

Codec ioctls:
	test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
	test VIDIOC_G_ENC_INDEX: OK (Not Supported)
		fail: v4l2-test-codecs.cpp(123): ret != 0
	test VIDIOC_(TRY_)DECODER_CMD: FAIL

Buffer ioctls:
	test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
	test VIDIOC_EXPBUF: OK
	test Requests: OK (Not Supported)

Test input 0:

Streaming ioctls:
	test read/write: OK (Not Supported)
	test blocking wait: OK
		fail: v4l2-test-buffers.cpp(1353): ret == 0
	test MMAP (select): FAIL
		fail: v4l2-test-buffers.cpp(1353): ret == 0
	test MMAP (epoll): FAIL
	test USERPTR (select): OK (Not Supported)
	test DMABUF: Cannot test, specify --expbuf-device

Total for bcm2835-codec device /dev/video10: 51, Succeeded: 46, Failed: 5, Warnings: 0

Anyway, shoot me any tests you want me to run.

raymanfx · 2020-11-14T20:15:53Z

It looks like getting your usecase working is going to be a lot more involved than I initially thought. I've started the work in the next branch. Basically, there's two (currently unsupported) features that are required:

Streaming output (MMAP in particular) - right now, only plain write() is supported for output devices.
Multi-planar buffer support

FallingSnow · 2020-11-14T21:04:00Z

Those 2 sound about right. I found some more interesting information at https://www.raspberrypi.org/forums/viewtopic.php?f=68&t=281296&p=1737303&hilit=v4l2+%2Fdev%2Fvideo10#p1737303 that might be useful.

I'm a beginner in rust but if there's anything I can do to help I'd love to.

raymanfx · 2020-11-15T16:06:32Z

Some good news - I was able to implement MMAP output streaming today (tested using v4l2loopback). My WIP stuff is on the next branch if you want to check it out.

raymanfx · 2020-11-15T21:05:42Z

I just pushed some preliminary multi-planar buffer support to the next branch. Could you please change this: https://github.com/raymanfx/libv4l-rs/blob/next/src/output/device.rs#L157 to return VideoOutputMplane and see if that makes it work for you? I'm still thinking about how to best incorporate multi-plane support into the API.

EDIT: the way to do this for your local project is to clone this repo (next branch) and point your project to its directory for dependency resolution in Cargo.toml.

FallingSnow · 2020-11-16T00:38:23Z

Cool. I'll give it a test in the next couple hours.

FallingSnow · 2020-11-16T06:41:02Z

Looks golden.

Source:

use std::io::Read;

use v4l::prelude::DeviceExt;
use v4l::output::device::Device as OutputDevice;

// use nix::sys::ioctl;

const DECODE_DEVICE: &str = "/dev/video10";

fn main() -> std::io::Result<()> {
    let mut file = std::fs::File::open("/home/alarm/short.h264")?;
    let mut contents = vec![];
    file.read_to_end(&mut contents)?;

    let dev = OutputDevice::with_path(DECODE_DEVICE)
        .expect(&format!("Unable to open decode device {}", DECODE_DEVICE));

    let capabilities = dev.query_caps().expect("could not get capabilities");

    dbg!(capabilities);

    let formats = dev.enum_formats()?;
    println!("Number of formats: {}", formats.len());
    for fmt in formats {
        print!("{}", fmt);
    }

    Ok(())
}

Output:

[alarm@alarmpi framebuffer_decode]$ ./target/debug/framebuffer_decode 
[src/main.rs:21] capabilities = Capabilities {
    driver: "bcm2835-codec",
    card: "bcm2835-codec-decode",
    bus: "platform:bcm2835-codec",
    version: (
        5,
        4,
        75,
    ),
    capabilities: VIDEO_M2M_MPLANE | EXT_PIX_FORMAT | STREAMING,
}
Number of formats: 2
index       : 0
type:       : 10
flags:      : COMPRESSED
description : H.264
fourcc      : H264
index       : 1
type:       : 10
flags:      : COMPRESSED
description : Motion-JPEG
fourcc      : MJPG

FallingSnow · 2020-11-16T06:43:58Z

Also, I hit up the raspberry pi forums about the compliance issue: https://www.raspberrypi.org/forums/viewtopic.php?f=67&t=291227&p=1760878#p1760878

Some interesting input on v4l2-ctl/v4l2-compliance in there.

raymanfx · 2020-11-16T07:46:52Z

Sounds good. Now we need to check whether the actual streaming I/O works with the new code. Did you try writing some H.264 or JPEG frames to that device (by creating a MmapStream and using the io::stream::Output trait)?

FallingSnow · 2020-11-16T15:32:43Z

Not sure if I'm doing this correctly, but output it Invalid Argument.

Source:

use std::io::Read;

use v4l::{Timestamp, buffer::{Buffer, Flags, Metadata}};
use v4l::io::stream::Output;
use v4l::output::device::Device as OutputDevice;
use v4l::prelude::{DeviceExt, MmapStream};

// use nix::sys::ioctl;

const DECODE_DEVICE: &str = "/dev/video10";
const VIDEO_PATH: &str = "/home/alarm/short.h264";

fn main() -> std::io::Result<()> {
    let mut dev = OutputDevice::with_path(DECODE_DEVICE)
        .expect(&format!("Unable to open decode device {}", DECODE_DEVICE));

    let capabilities = dev.query_caps().expect("could not get capabilities");

    dbg!(capabilities);

    let formats = dev.enum_formats()?;
    println!("Number of formats: {}", formats.len());
    for fmt in formats {
        print!("{}", fmt);
    }

    let mut stream = MmapStream::with_buffers(&mut dev, 1).expect("Failed to create buffer stream");

    let mut file = std::fs::File::open(VIDEO_PATH)?;
    let mut contents = vec![];
    file.read_to_end(&mut contents)?;

    let metadata = Metadata {
        bytesused: contents.len() as u32,
        flags: Flags::empty(),
        field: 0,
        timestamp: Timestamp::new(0, 0),
        sequence: 0
    };

    let buffer = Buffer {
        planes: vec![contents.as_ref()],
        meta: metadata
    };

    Output::next(&mut stream, buffer)?;

    Ok(())
}

Output:

[alarm@alarmpi ~]$ ./framebuffer_decode/target/debug/framebuffer_decode 
[src/main.rs:19] capabilities = Capabilities {
    driver: "bcm2835-codec",
    card: "bcm2835-codec-decode",
    bus: "platform:bcm2835-codec",
    version: (
        5,
        4,
        75,
    ),
    capabilities: VIDEO_M2M_MPLANE | EXT_PIX_FORMAT | STREAMING,
}
Number of formats: 2
index       : 0
type:       : 10
flags:      : COMPRESSED
description : H.264
fourcc      : H264
index       : 1
type:       : 10
flags:      : COMPRESSED
description : Motion-JPEG
fourcc      : MJPG
thread 'main' panicked at 'Failed to create buffer stream: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }', src/main.rs:27:60
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

raymanfx · 2020-11-17T18:21:57Z

Okay, so this will require some more debugging. Can you add some logs to this function: https://github.com/raymanfx/libv4l-rs/blob/next/src/io/mmap/arena.rs#L54 and see where exactly it fails?

I just looked at some Raspberry Pi 4 bundles today, but the 4GB models are still rather expensive, I don't think I can justify getting one just for this project. Anything less than 4GB is not worth paying for though.

Apart from that, I'm still wondering about how the H.264 decoder works here - you're basically just passing the entire file buffer in one go. From what I read in the link you provided (https://www.raspberrypi.org/forums/viewtopic.php?f=67&t=291227&p=1760878#p1760878), I think you're supposed to pass framed data - so you'd need to split the H.264 bytestream into frames before passing them to the decoder? Then again, they also they it should work with unframed data as well, disregarding a latency increase.

FallingSnow · 2020-11-17T19:54:08Z

I'll add some logs to that function and see where it's failing.

Yeah, from what the forum post explains, the decoder should work on unframed data (with performance degradation). But honestly I don't really know if passing in the file buffer is the right way to do it either.

Can I just give you ssh access to my PI? I leave it on 24/7 anyway. If that works just email me at ayrton AT sparling DOT us.

raymanfx · 2020-11-17T22:56:46Z

Oh .. I see it now. The H.264 decoder device on the Raspberry Pi 4 requires using the VIDEO_M2M_MPLANE API, but this crate right now only implements capture and output (not memory-to-memory codecs). That's going to involve more work. I think I want to restructure some things first before implementing M2M streaming.

EDIT: not entirely true - while your decoder supports M2M, you don't actually want to use it here, since you're feeding data from an external source. I'll continue with the refactor for now to ease implementing future usecases such as M2M.

patrickelectric · 2022-04-24T14:21:33Z

Hey @raymanfx, what is the status for it ?

raymanfx · 2022-04-26T14:49:49Z

Hi @patrickelectric, I prepared the multi-planar API support a while ago: https://github.com/raymanfx/libv4l-rs/tree/mplane but was not happy with the API at the time. Since it is not required for my projects, I did not finish the work. If you are interested in picking it up or collaborating, let me know.

wangxiaochuTHU · 2023-03-06T12:54:14Z

Hi @raymanfx , it looks that the mplane- branch has already been compatible with a VIDEO_CAPTURE_MPLANE device, just by looking into the data structures defined and capture procedures. But you said you did not finish the work. Would you mind pointing out the unfinished part or giving any guidelines on its accomplishment?

Hi @patrickelectric, I prepared the multi-planar API support a while ago: https://github.com/raymanfx/libv4l-rs/tree/mplane but was not happy with the API at the time. Since it is not required for my projects, I did not finish the work. If you are interested in picking it up or collaborating, let me know.

raymanfx added the enhancement New feature or request label Nov 12, 2020

D1plo1d mentioned this issue Feb 1, 2023

Hardware Encoding/Decoding #67

Open

msmarks mentioned this issue Jul 15, 2024

support mplane capture and mmap #105

Open

MarijnS95 mentioned this issue Dec 19, 2024

No support for VideoCaptureMPlane? #121

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raspberry Pi requires multi-planar buffer type #15

Raspberry Pi requires multi-planar buffer type #15

FallingSnow commented Nov 12, 2020

FallingSnow commented Nov 12, 2020

raymanfx commented Nov 12, 2020

FallingSnow commented Nov 12, 2020

raymanfx commented Nov 12, 2020

raymanfx commented Nov 13, 2020

FallingSnow commented Nov 14, 2020 •

edited

Loading

raymanfx commented Nov 14, 2020

FallingSnow commented Nov 14, 2020

raymanfx commented Nov 15, 2020

raymanfx commented Nov 15, 2020 •

edited

Loading

FallingSnow commented Nov 16, 2020

FallingSnow commented Nov 16, 2020

FallingSnow commented Nov 16, 2020

raymanfx commented Nov 16, 2020

FallingSnow commented Nov 16, 2020

raymanfx commented Nov 17, 2020

FallingSnow commented Nov 17, 2020

raymanfx commented Nov 17, 2020 •

edited

Loading

patrickelectric commented Apr 24, 2022

raymanfx commented Apr 26, 2022

wangxiaochuTHU commented Mar 6, 2023

Raspberry Pi requires multi-planar buffer type #15

Raspberry Pi requires multi-planar buffer type #15

Comments

FallingSnow commented Nov 12, 2020

FallingSnow commented Nov 12, 2020

raymanfx commented Nov 12, 2020

FallingSnow commented Nov 12, 2020

raymanfx commented Nov 12, 2020

raymanfx commented Nov 13, 2020

FallingSnow commented Nov 14, 2020 • edited Loading

raymanfx commented Nov 14, 2020

FallingSnow commented Nov 14, 2020

raymanfx commented Nov 15, 2020

raymanfx commented Nov 15, 2020 • edited Loading

FallingSnow commented Nov 16, 2020

FallingSnow commented Nov 16, 2020

FallingSnow commented Nov 16, 2020

raymanfx commented Nov 16, 2020

FallingSnow commented Nov 16, 2020

raymanfx commented Nov 17, 2020

FallingSnow commented Nov 17, 2020

raymanfx commented Nov 17, 2020 • edited Loading

patrickelectric commented Apr 24, 2022

raymanfx commented Apr 26, 2022

wangxiaochuTHU commented Mar 6, 2023

FallingSnow commented Nov 14, 2020 •

edited

Loading

raymanfx commented Nov 15, 2020 •

edited

Loading

raymanfx commented Nov 17, 2020 •

edited

Loading