firmware/sysfb: Fix VESA format selection
Some legacy BIOSes report no reserved bits in their 32-bit rgb mode,
breaking the calculation of bits_per_pixel in commit
f35cd3fa7729
("firmware/sysfb: Fix EFI/VESA format selection"). However they report
lfb_depth correctly for those modes. Keep the computation but
set bits_per_pixel to lfb_depth if the latter is larger.
v2 fixes the warnings from a max3() macro with arguments of different
types; split the bits_per_pixel assignment to avoid uglyfing the code
with too many typecasts.
v3 fixes space and formatting blips pointed out by Javier, and change
the bit_per_pixel assignment back to a single statement using two casts.
v4 go back to v2 and use max_t()
Signed-off-by: Pierre Asselin <pa@panix.com>
Fixes:
f35cd3fa7729 ("firmware/sysfb: Fix EFI/VESA format selection")
Link: https://lore.kernel.org/r/4Psm6B6Lqkz1QXM@panix3.panix.com
Link: https://lore.kernel.org/r/20230412150225.3757223-1-javierm@redhat.com
Tested-by: Thomas Zimmermann <tzimmermann@suse.de>
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Link: https://patchwork.freedesktop.org/patch/msgid/20230419044834.10816-1-pa@panix.com