staging: vc04_services: Add a V4L2 M2M codec driver

staging: vc04_services: Add a V4L2 M2M codec driver

This adds a V4L2 memory to memory device that wraps the MMAL
video decode and video_encode components for H264 and MJPEG encode
and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode
if the appropriate licence has been purchased).

This patch squashes all the work done in developing the driver
on the Raspberry Pi rpi-5.4.y kernel branch.
Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and
Marek Behún for their contributions. Please refer to the
rpi-5.4.y branch for the full history.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded

The firmware by default tries to ensure that decoded frame
timestamps always increment. This is counter to the V4L2 API
which wants exactly the OUTPUT queue timestamps passed to the
CAPTURE queue buffers.

Disable the firmware option.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE

Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder
is meant to emit the header bytes as a separate packet or with the
first encoded frame.
Add support for it.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/vc04_services/codec: Clear last buf dequeued flag on START

It appears that the V4L2 M2M framework requires the driver to manually
call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a
V4L2_DEC_CMD_START.
Add such a call.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/vc04-services/codec: Fix logical precedence issue

Two issues identified with operator precedence in logical
expressions. Fix them.

https://github.com/raspberrypi/linux/issues/4040

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
vc04_services: bcm2835-codec: Switch to s32fract

staging/bcm2835-codec: Add the unpacked (16bpp) raw formats

Now that the firmware supports the unpacked (16bpp) variants
of the MIPI raw formats, add the mappings.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Log the number of excess supported formats

When logging that the firmware has provided more supported formats
than we had allocated storage for, log the number allocated and
returned.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Add support for pixel aspect ratio

If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE
event is generated, then pass on the pixel aspect ratio as well.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Implement additional g_selection calls for decode

v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel
aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS
and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect
not to be implemented, it doesn't allow for either of the other two.

Add in support for the additional selection targets.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Add VC-1 support.

Providing the relevant licence has been purchased, then Pi0-3
can decode VC-1.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Fix support for levels 4.1 and 4.2

The driver said it supported H264 levels 4.1 and 4.2, but
was missing the V4L2 to MMAL mappings.

Add in those mappings.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Set the colourspace appropriately for RGB formats

Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M
or REC709 appropriately, whilst RGB should report SRGB.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Pass corrupt frame flag.

MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that
wasn't being passed through, so add it.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Do not update crop from S_FMT after res change

During decode, setting the CAPTURE queue format was setting the crop
rectangle to the requested height before aligning up the format to
cater for simple clients that weren't expecting to deal with cropping
and the SELECTION API.
This caused problems on some resolution change events if the client
didn't also then use the selection API.

Disable the crop update after a resolution change.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
bcm2835: Allow compressed frames to set sizeimage (#4386)

Allow the user to set sizeimage in TRY_FMT and S_FMT if the format
flags have V4L2_FMT_FLAG_COMPRESSED set

Signed-off-by: John Cox <jc@kynesim.co.uk>
staging/bcm2835-codec: Change the default codec res to 32x32

In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE
event occurs, adopt a default resolution of 32x32 so that it
is incredibly unlikely to be decoding a stream of that resolution
and therefore failing to note a "change" requiring the event.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Add support for decoding interlaced streams

The video decoder can support decoding interlaced streams, so add
the required plumbing to signal this correctly.

The encoder and ISP do NOT support interlaced data, so trying to
configure an interlaced format on those nodes will be rejected.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2

Being YUV420 formats, the step size is always 2 to avoid part
chroma subsampling.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Return buffers to QUEUED not ERROR state

Should start_streaming fail, or buffers be queued during
stop_streaming, they should be returned to the core as QUEUED
and not (as currently) as ERROR.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835_codec: Log MMAL flags in hex

The flags is a bitmask, so it's far easier to interpret as hex
data instead of decimal.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Allow custom specified strides/bytesperline.

If the client provides a bytesperline value in try_fmt/s_fmt then
validate it and correct if necessary.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835_codec: Add support for image_fx to deinterlace

Adds another /dev/video node wrapping image_fx doing deinterlace.

Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Dom Cobley <popcornmix@gmail.com>
staging/bcm2835-v4l2_codec: Fix for encode selection API

Matches correct behaviour from DECODE and DEINTERLACE

Signed-off-by: Dom Cobley <popcornmix@gmail.com>
staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE)

The V4L2 stateful video decoder API requires that you can STREAMON
on only the OUTPUT queue, feed in buffers, and wait for the
SOURCE_CHANGE event.
This requires that we enable the MMAL output port at the same time
as the input port, because the output port is the one that creates
the SOURCE_CHANGED event.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Do not send buffers to the VPU unless streaming

With video decode we now enable both input and output ports on
the component. This means that buffers will get passed to the VPU
earlier than desired if they are queued befoer STREAMON.

Check that the queue is streaming before sending buffers to the VPU.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Format changed should trigger drain

When a format changed event occurs, the spec says that it
triggers an implicit drain, and that needs to be signalled
via -EPIPE.

For BCM2835, the format changed event happens at the point
the format change occurs, so no further buffers exist from
before the resolution changed point. We therefore signal the
last buffer immediately.
We don't have a V4L2 available to us at this point, so set
the videobuf2 queue last_buffer_dequeued flag directly.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Signal the firmware to stop on all changes

The firmware defaults to not stopping video decode if only the
pixel aspect ratio or colourspace change. V4L2 requires us
to stop decoding on any change, therefore tell the firmware
of the desire for this alternate behaviour.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Queue flushed buffers instead of completing

When a buffer is returned on a port that is disabled, return it
to the videobuf2 QUEUED state instead of DONE which returns it
to the client.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835_codec: Correct flushing code for refcounting

Completions don't reference count, so setting the completion
on the first buffer returned and then not reinitialising it
means that the flush function doesn't behave as intended.

Signal the completion when the last buffer is returned.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Ensure all ctrls are set on streamon

Currently the code was only setting some controls from
bcm2835_codec_set_ctrls, but it's simpler to use
v4l2_ctrl_handler_setup to avoid forgetting to adding new
controls to the list.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Add support for H&V Flips to ISP

The ISP can do H & V flips whilst resizing or converting
the image, so expose that via V4L2_CID_[H|V]FLIP.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
bcm2835-v4l2-codec: Remove advertised support of VP8

The support for this format by firmware is very limited
and won't be faster than the arm.

Signed-off-by: Dom Cobley <popcornmix@gmail.com>
Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec

Following raspberrypi/linux#4704. This is necessary to set up
quantization for variable bitrate to avoid video flickering.

staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64

Matching https://github.com/raspberrypi/linux/pull/4419, the ISP
block (which is also used on the input of the encoder, and output
of the decoder) needs the base address of all planes to be aligned
to multiples of 32. This includes the chroma planes of YUV420 and
YVU420.
If the height is only a multiple of 2 (not 4), then you get an odd
number of lines in the second plane, which means the 3rd plane
starts at a multiple of bytesperline/2.

Set the minimum bytesperline alignment to 64 for those formats
so that the plane alignment is always right.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging/bcm2835-codec: Allow a different stride alignment per role

Deinterlace and decode aren't affected in the same way as encode
and ISP by the alignment requirement on 3 plane YUV420.
Decode would be affected, but it always aligns the height up to
a macroblock, and uses the selection API to reflect that.

Add in the facility to set the bytesperline alignment per role.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format

We already support V4L2_PIX_FMT_BGR32 which is the same thing with red
and blue swapped, so it makes sense to include this variant as well.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2

The encoder can skip frames totally should rate control overshoot
the target bitrate too far. In this situation it generates an
output buffer of length 0.
V4L2 treats a buffer of length 0 as an end of stream flag, which is
not appropriate in this case, therefore we can not return that buffer
to the client.

The driver was returning the buffer to videobuf2 in the QUEUED state,
however that buffer was then not dequeued again, so the number of
buffers was reduced each time this happened. In the pathological
case of using GStreamer's videotestsrc in mode 1 for noise, this happens
sufficiently frequently to totally stall the pipeline.

If the port is still enabled then return the buffer straight back to
the VPU rather than to videobuf2.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128

V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of
video_encode, output of video_decode, and both input and output
of the ISP.

Add in the plumbing to support the format on those ports.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
vc04_services: bcm2835-codec: Set crop_height for compressed formats

In particular for the encoder where the CAPTURE format dictates
the parameters given to the codec we need to be able to set the
value passed as the crop_height for the compressed format.
There's no crop available for cropped modes, so always set
crop_height to the requested height.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
vc04_services: bcm2835-codec: Set port format from s_selection

s_selection allows the crop region of an uncompressed pixel
format to be specified, but it wasn't passing the setting on to
the firmware. Depending on call order this would potentially
mean that the crop wasn't actioned.

Set the port format on s_selection if we have a component created.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder

Signed-off-by: Maxim Devaev <mdevaev@gmail.com>
bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels

vc04_services: bcm2835-codec:  Remove redundant role check

vidioc_try_encoder_cmd checks the role, but the ioctl is disabled
for any roles in which it is invalid.

Remove the redundant check.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace

ISP and deinterlace also need a mechanism for passing effectively
an EOS through the pipeline to signal when all buffers have been
processed.

VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same
function for ISP and deinterlace.
(VIDIOC_DECODER_CMD is slightly different in that it also passes
details of when and how to stop, so is not as relevant).

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
vc04_services: bcm2835_codec: Allow larger images through the ISP

Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP
isn't, but the limits advertised via V4L2 was 1920x1920 for all
roles.

Increase the limit to 16k x 16k for the ISP.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
media: bcm2835-v4l2-codec: Enable selection ioctl for ISP

The ISP cases do nothing. Remove the break that separates them from the
deinterlace case so they now do the same as deinterlace. This enables
simple width & height setting, but does not enable setting left and
top coordinates.

Signed-off-by: John Cox <jc@kynesim.co.uk>
media: bcm2835-v4l2-codec: Add profile & level ctrls to decode

In order to support discovery of what profile & levels are supported by
stateful decoders implement the profile and level controls where they
are defined by V4L2.

Signed-off-by: John Cox <jc@kynesim.co.uk>
vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl

In adding the MPEG2/MPEG4/H264 level and profile controls to
the decoder, they weren't declared as read-only, nor handlers
added to bcm2835_codec_s_ctrl. That resulted in an error message
"Invalid control" being logged every time v4l2_ctrl_handler_setup
was called from bcm2835_codec_create_component.

Define those controls as read only, and exit early from s_ctrl
on read only controls.

Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode"
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY

V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL  was missed from
"vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl"

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control

FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to
0, and generates an error should it fail.
As our encoder doesn't support B frames, add a stub handler for
it to silence FFmpeg.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE

For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra
I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR
frame period.
The firmware encoder doesn't produce I-frames that aren't IDR as well,
therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct
control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD.

Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder,
and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update
the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not
implemented).

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32

The patch adding image encode (JPEG) to the driver missed adding
the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant
it ended up giving a stride and size of 0.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
staging: bcm2835-codec: Downgrade the level for a debug message

The debug message from bcm2835_codec_buf_prepare when the buffer
size is incorrect can be a little spammy if the application isn't
careful on how it drives it, therefore drop the priority of the
message.

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>