drm/amd/display: Fix UBSAN: shift-out-of-bounds warning
authorAnson Jacob <Anson.Jacob@amd.com>
Mon, 1 Mar 2021 19:25:44 +0000 (14:25 -0500)
committerAlex Deucher <alexander.deucher@amd.com>
Fri, 9 Apr 2021 20:41:21 +0000 (16:41 -0400)
commit54718747a6e1037317a8b3610c3be40621b2b75e
tree4373da5a31aa5a5140f1e7c9ddfa582beae2870b
parent3d223c5528a387f84ed8bdbed2ce05227be9afe0
drm/amd/display: Fix UBSAN: shift-out-of-bounds warning

[Why]
On NAVI14 CONFIG_UBSAN reported shift-out-of-bounds at
display_rq_dlg_calc_20v2.c:304:38

rq_param->misc.rq_c.blk256_height is 0 when chroma(*_c) is invalid.
dml_log2 returns -1023 for log2(0), although log2(0) is undefined.

Which ended up as:
rq_param->dlg.rq_c.swath_height = 1 << -1023

[How]
Fix applied on all dml versions.
1. Ensure dml_log2 is only called if the argument is greater than 0.
2. Subtract req128_l/req128_c from log2_swath_height_l/log2_swath_height_c
   only when it is greater than 0.

Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Anson Jacob <Anson.Jacob@amd.com>
Reviewed-by: Dmytro Laktyushkin <Dmytro.Laktyushkin@amd.com>
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Solomon Chiu <solomon.chiu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
drivers/gpu/drm/amd/display/dc/dml/dcn20/display_rq_dlg_calc_20.c
drivers/gpu/drm/amd/display/dc/dml/dcn20/display_rq_dlg_calc_20v2.c
drivers/gpu/drm/amd/display/dc/dml/dcn21/display_rq_dlg_calc_21.c
drivers/gpu/drm/amd/display/dc/dml/dcn30/display_rq_dlg_calc_30.c
drivers/gpu/drm/amd/display/dc/dml/dml1_display_rq_dlg_calc.c