drm/arm/malidp: Validate rotations for compressed/uncompressed framebuffers for each layer
Add support for compressed framebuffers that are described using
the framebuffer's modifier field. Mali DP uses the rotation memory for
the decompressor of the format, so we need to check for space when
the modifiers are present.
Signed-off-by: Ayan Kumar Halder <ayan.halder@arm.com>
Reviewed-by: Brian Starkey <brian.starkey@arm.com>
[re-worded commit, rebased, cleaned up duplicated checks for
RGB888 and BGR888 and removed additional parameter for
rotmem_required function hook]
Signed-off-by: Liviu Dudau <liviu.dudau@arm.com>