drm/amd/display: Fix reg timeout in enc314_enable_fifo

[Why]
The link enablement sequence can end up resetting the encoder while
the PHY symclk isn't yet on.

This means that waiting for symclk on will timeout, along with the reset
bit never asserting high.

This causes unnecessary delay when enabling the link and produces a
warning affecting multiple IGT tests.

[How]
Don't wait for the symclk to be on here because firmware already does.

Don't wait for reset if we know the symclk isn't on.

Split the reset into a helper function that checks the bit and decides
whether or not a delay is sufficient.

Reviewed-by: Roman Li <Roman.Li@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Fix FCLK deviation and tool compile issues

[Why]
Recent backports from open source do not have header inclusion pattern
that is consistent with inclusion style in the rest of the file. This
breaks the internal tool builds as well. A recent commit erronously
modified the original DML formula for calculating
ActiveClockChangeLatencyHidingY. This resulted in a FCLK deviation
from the golden values.

[How]
Change the way in which display_mode_vba.h is included so that it is
consistent with the inclusion style in rest of the file which also fixes
the tool build. Restore the DML formula to its original state to fix the
FCLK deviation.

Reviewed-by: Aurabindo Pillai <Aurabindo.Pillai@amd.com>
Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Chaitanya Dhere <chaitanya.dhere@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Fix fallback issues for DP LL 1.4a tests

[WHY]
Unlike DP or USBC, the USB4 link does not get its own encoder and
has to share therefore verify_caps is skipped.

[HOW]
Fix the fallback logic for automated tests and take that
into consideration for LT and LS.

Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Mustapha Ghaddar <mustapha.ghaddar@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Disable DRR actions during state commit

[WHY]
Committing a state while performing DRR actions can cause underflow.

[HOW]
Disabled features performing DRR actions during state commit.
Need to follow-up on why DRR actions affect state commit.

Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Wesley Chalmers <Wesley.Chalmers@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Disable phantom OTG after enable for plane disable

[Description]
- Need to disable phantom OTG after it's enabled
  in order to restore it to it's original state.
- If it's enabled and then an MCLK switch comes in
  we may not prefetch the correct data since the phantom
  OTG could already be in the middle of the frame.

Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Alvin Lee <Alvin.Lee2@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Use min transition for SubVP into MPO

[Description]
- For SubVP transitioning into MPO, we want to
  use a minimal transition to prevent transient
  underflow
- Transitioning a phantom pipe directly into a
  "real" pipe can result in underflow due to the
  HUBP still having it's "phantom" programming
  when HUBP is unblanked (have to wait for next
  VUPDATE of the new OTG)
- Also ensure subvp pipe lock is acquired early
  enough for programming in dc_commit_state_no_check
- When disabling phantom planes, enable phantom OTG
  first so the disable gets the double buffer update

Reviewed-by: Aric Cyr <Aric.Cyr@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Alvin Lee <Alvin.Lee2@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Zeromem mypipe heap struct before using it

[Why&How]
Bug was caused when moving variable from stack to heap because it was reusable
and garbage was left over, so we need to zero mem.

Reviewed-by: Martin Leung <Martin.Leung@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Aurabindo Pillai <aurabindo.pillai@amd.com>
Signed-off-by: Martin Leung <Martin.Leung@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Allow tuning DCN314 bounding box

[Why]
We're missing the helpers from dcn20 that would allow
overriding these with DC debug options.

[How]
Use dcn20_patch_bounding_box to support overriding all the
relevant values.

Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Update SR watermarks for DCN314

[Why & How]
New values requested by hardware after fine-tuning.
Update for all memory types.

Reviewed-by: Jun Lei <Jun.Lei@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Nicholas Kazlauskas <nicholas.kazlauskas@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Waiting for 1 frame to fix the flash issue on PSR1

[Why]
Needs more frames waiting before the PSR_Exit sending for the specific
TCON.

[How]
Add relock_delay_frame_cnt to control how many frames waiting are needed
before the PSR_Exit sending. The default value is 0. The Driver side can
set this variable for specific TCONs.

Reviewed-by: Robin Chen <robin.chen@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Ryan Lin <tsung-hua.lin@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Consider dp cable id only when data is non zero

Cable ID is a DP2 feature to identify max certified link rate that
a cable can carry. The cable identification method requires both
cable and display hardware support. Since the specs comes late, it is
anticipated that the first round of DP2 cables and displays may not
be fully compatible to reliably return cable ID data. Therefore the
decision of our cable id policy is that if the cable can return non
zero cable id data, we will take cable's link rate capability into
account. However if we get zero data, the cable link rate capability
is considered inconclusive. In this case, we will not take cable's
capability into account to avoid of over limiting hardware capability
from users. The max overall link rate capability is still determined
after actual dp pre-training. Cable id is considered as an auxiliary
method of determining max link bandwidth capability.

Reviewed-by: George Shen <George.Shen@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Wenjing Liu <wenjing.liu@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Update 709 gamma to 2.222 as stated in the standerd

[WHY]
    Previously driver use gamma 2.2 for 709 color space,
    but the standard is to use gamma of 2.222

[HOW]
    Change it gamma to 2.222

Reviewed-by: Krunoslav Kovac <Krunoslav.Kovac@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Nawwar Ali <nawwar.ali@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Cursor update refactor: PSR-SU support condition

[Why]
PSR-SU requires extra conditions while cursor update.

Reviewed-by: Robin Chen <robin.chen@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Max Tseng <Max.Tseng@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Adding HDMI SCDC DEVICE_ID define

[Why && How]
We will need to differentiate vendor behavior in the future.

Reviewed-by: Chris Park <Chris.Park@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Leo Ma <hanghong.ma@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/display: Wait for VBLANK during pipe programming

[Description]
- Wait for vblank during front end programming
  for global sync to ensure all double buffer
  updates take.
- This prevents underflow in some cases.

Reviewed-by: Martin Leung <Martin.Leung@amd.com>
Acked-by: Alan Liu <HaoPing.Liu@amd.com>
Signed-off-by: Alvin Lee <Alvin.Lee2@amd.com>
Tested-by: Daniel Wheeler <daniel.wheeler@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amdgpu: workaround for TLB seq race

It can happen that we query the sequence value before the callback
had a chance to run.

Workaround that by grabbing the fence lock and releasing it again.
Should be replaced by hw handling soon.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org # 5.19+
Fixes: 5255e146c99a6 ("drm/amdgpu: rework TLB flushing")
Link: https://gitlab.freedesktop.org/drm/amd/-/issues/2113
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Acked-by: Philip Yang <Philip.Yang@amd.com>
Tested-by: Stefan Springer <stefanspr94@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amdkfd: Fix error handling in criu_checkpoint

Checkpoint BOs last. That way we don't need to close dmabuf FDs if
something else fails later. This avoids problematic access to user mode
memory in the error handling code path.

criu_checkpoint_bos has its own error handling and cleanup that does not
depend on access to user memory.

In the private data, keep BOs before the remaining objects. This is
necessary to restore things in the correct order as restoring events
depends on the events-page BO being restored first.

Fixes: be072b06c739 ("drm/amdkfd: CRIU export BOs as prime dmabuf objects")
Reported-by: Jann Horn <jannh@google.com>
CC: Rajneesh Bhardwaj <Rajneesh.Bhardwaj@amd.com>
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-and-tested-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amdkfd: Fix error handling in kfd_criu_restore_events

mutex_unlock before the exit label because all the error code paths that
jump there didn't take that lock. This fixes unbalanced locking errors
in case of restore errors.

Fixes: 40e8a766a761 ("drm/amdkfd: CRIU checkpoint and restore events")
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/amd/pm: update SMU IP v13.0.4 msg interface header

Some of the unused messages that were used earlier in development have
been freed up as spare messages, no intended functional changes.

Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Tim Huang <tim.huang@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Yifan Zhang <yifan1.zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Merge branch '00.06-gr-ampere' of https://gitlab.freedesktop.org/skeggsb/nouveau into drm-next

This is the pull request for a whole bunch of fixes and prep-work that
was done to support Ampere acceleration prior to GSP-RM being
available.  It uses the ACR firmware released by NVIDIA in
linux-firmware, as we do on earlier GPUs.  The work to support running
on top of GSP-RM also heavily depends on various pieces of this
series.

In addition to the new HW support, general stability of the driver
should be improved, especially around recovering HW from bugs that can
be generated by userspace driver components.

Signed-off-by: Dave Airlie <airlied@redhat.com>
From: Ben Skeggs <bskeggs@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/CABDvA==s+nZD0n7CuRWLPE=Pj+02CN13r+ZQJxoHQ_EmR+o=XQ@mail.gmail.com

drm/nouveau/gr/ga102: initial support

v2:
- whitespace

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Gourav Samaiya <gsamaiya@nvidia.com>

drm/nouveau/ltc/ga102: initial support

v2. fixup for ga103 early merge

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/acr/ga102: initial support

v2. fixup for ga103 early merge

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Gourav Samaiya <gsamaiya@nvidia.com>

drm/nouveau/fb/ga102: load and boot VPR scrubber FW

v2. fixup for ga103 early merge

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Gourav Samaiya <gsamaiya@nvidia.com>

drm/nouveau/gr/tu102: remove gv100_grctx_unkn88c

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/tu102: add gv100_gr_init_4188a4

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/tu102-: fix support for sw_bundle64_init

We weren't sending the high bits, though they're zero currently anyway.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/tu102-: use sw_veid_bundle_init from firmware

NVIDIA provided this on Turing, but we kept using the hardcoded version
from Volta (where they didn't).

Switch to the firmware version prior to Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gv100-: drop a write from init_shader_exceptions()

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gv100-: move init_419bd8() after sw_ctx load

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gv100-: add NV_PGRAPH_PRI_PD_AB_DIST_CONFIG_1 to patch list

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gv100-: fix number of tile map registers

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gv100-: port smid mapping code from nvgpu

Essentially ripped verbatim from NVGPU, comments and all, and adapted to
nvkm's structs and style.

- maybe fixes an nvgpu bug though, a small tweak was needed to match RM

v2:
- remove unnecessary WARN_ON

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gp100-: modify init_fecs_exceptions

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gk20a,gm20b,gp10b: split out netlist parsing from fw loading

We'll want to reuse the former for loading from proper netlist images.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gp100-: fix number of zcull tile regs

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf117-: make ppc_nr[gpc] accurate

We're going to be pulling in a chunk of code from NVGPU to fixup our
SMID mappings on Volta and above, which depends on ppc_nr[gpc]
reflecting the actual number of PPCs present, not the maximum number.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: switch to newer style interrupt handler

Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: move some init to init_exception2()

Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: move some init to init_rop_exceptions()

Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: move reset during golden ctx init to fecs_reset()

Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: wfi after register-bashing golden init

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: gpfifo_ctl zero before init

Match RM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: wait for FE_PWR_MODE_AUTO

This doesn't fix any known issue, but RM started doing it at some point,
so presumably it's needed for something.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: call FECS HALT_PIPE method before RC reset

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: call FECS WFI_GOLDEN_SAVE method

This won't work on Ampere, and, it's questionable whether we should have
been using our FW's method of storing the golden context image with NV's
firmware to begin with.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: stop using NV_PGRAPH_FECS_CTXSW_MAILBOX_CLEAR

This doesn't work on Ampere for some reason, switch to directly modifying
NV_PGRAPH_FECS_CTXSW_MAILBOX instead.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: make global attrib_cb actually global

This was thought to be per-channel initially - it's not.  The backing
pages for the VMM mappings are shared for all channels.

- switches to more straight-forward patch interfaces
- prepares for sub-context support
- this is saving a *sizeable* amount of vram

v2:
- whitespace

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: move misc context patching out of attrib_cb funcs

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: make global bundle_cb actually global

This was thought to be per-channel initially - it's not.  The backing
pages for the VMM mappings are shared for all channels.

- switches to more straight-forward patch interfaces
- prepares for sub-context support

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: make global pagepool actually global

This was thought to be per-channel initially - it's not.  The backing
pages for the VMM mappings are shared for all channels.

- switches to more straight-forward patch interfaces
- prepares for sub-context support

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: generate golden context during first object alloc

Needed for GV100 (and only GV100 for some reason) for WFI_GOLDEN_SAVE.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gr/gf100-: move some code around to make next commits nicer

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: expose function to read engine ctxsw status

Needed to support Ampere differences in gr/gf100-:

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/ltc: split color vs depth/stencil zbc counts

These differ on Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/engine: add HAL for engine-specific rc reset procedure

Will be used to improve gr reset on GF100 and newer.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/sec2: dump tracepc info on halt

- useful to distinguish between different issues.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/acr: use common falcon HS FW code for ACR FWs

Adds context binding and support for FWs with a bootloader to the code
that was added to load VPR scrubber HS binaries, and ports ACR over to
using all of it.

- gv100 split from gp108 to handle FW exit status differences

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fb/gp102-: unlock VPR right after devinit

Under memory load, instmem allocations could end up in the regions of
VRAM that are inaccessible right after boot, and be corrupted after a
suspend/resume cycle as a result of being restored before booting the
mem unlock firmware.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fb: handle sysmem flush page from common code

- also executes pre-DEVINIT, so early boot is able to DMA sysmem

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/flcn: new code to load+boot simple HS FWs (VPR scrubber)

Adds the start of common interfaces to load and boot the HS binaries
provided by NVIDIA that enable the usage of GR.

ACR already handles most of this, but it's very much tied into ACR's
init process, and there's other code that could benefit from reusing
a lot of this stuff too (ie. VBIOS DEVINIT/PreOS, VPR scrubber).

The VPR scrubber code is fairly independent, and a good first target.

- adds better debug output to fw loading process, to ease bring-up/debug

v2:
- whitespace, 0->false

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/flcn: rework falcon reset

Mostly preparation to fit in Ampere changes, but should result in reset
sequences a lot closer to RM's, and perhaps help out with the issues we
sometimes see reported in this area.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/sec2: switch to newer style interrupt handler

Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/sec2: unload RTOS before tearing down WPR

Reset regs won't be available on Ampere while SEC2 RTOS is running, and
we're apparently supposed to be doing this on earlier GPUs too.

v2:
- fixed some excessive indentation

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/pmu/gm20b,gp10b: boot RTOS from PMU init

Cleanup before falcon changes.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/pmu: move init() falcon reset to non-nvfw code

Cleanup before falcon changes.

- fixes (attempt at?) reset of pmu while rtos is running, on gm20b

v2:
- remove extra whitespace

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/pmu: move preinit() falcon reset to devinit

Cleanup before falcon changes.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/gsp: add funcs

Ampere.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo/ga100-: initial support

- replaces the hacked-up version that existed solely to support TTM

v2. remove earlier hack preventing use of non-stall intr for fences

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/ce/ga100-: initial support

- replaces the hacked-up version that existed solely to support TTM
- noop until the next commit, adding proper support for ampere host

v2. fixup for ga103 early merge

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: add new channel classes

Exposes a bunch of the new features that became possible as a result
of the earlier commits.  DRM will build on this in the future to add
support for features such as SCG ("async compute") and multi-device
rendering, as part of the work necessary to be able to write a half-
decent vulkan driver - finally.

For the moment, this just crudely ports DRM to the API changes.

- channel class interfaces now the same for all HW classes
- channel group class exposed (SCG)
- channel runqueue selector exposed (SCG)
- channel sub-device id control exposed (multi-device rendering)
- channel names in logging will reflect creating process, not fd owner
- explicit USERD allocation required by VOLTA_CHANNEL_GPFIFO_A and newer
- drm is smarter about determining the appropriate channel class to use

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add new engine object handling

Simplifies the GPU-specific code, completing the switch to newer HALs.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add new engine context handling

Builds on the context tracking that was added earlier.

- marks engine context PTEs as 'priv' where possible

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add RAMFC info to nvkm_chan_func

- adds support for specifying SUBDEVICE_ID for channel
- rounds non-power-of-two GPFIFO sizes down, rather than up

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add USERD info to nvkm_chan_func

And use it to cleanup multiple implementations of almost the same thing.

- prepares for non-polled / client-provided USERD
- only zeroes relevant "registers", rather than entire USERD

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add RAMIN info to nvkm_chan_func

Currently provided by {chan,dma,gpfifo}*.c, and those are going away.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add common runlist control

- less dependence on waiting for runlist updates, on GPUs that allow it
- supports runqueue selector in RAMRL entries
- completes switch to common runl/cgrp/chan topology info

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add common channel recovery

That sure was fun to untangle.

- handled per-runlist, rather than globally
- more straight-forward process in general
- various potential SW/HW races have been fixed
- fixes lockdep issues that were present in >=gk104's prior implementation
- volta recovery now actually stands a chance of working
- volta/turing waiting for PBDMA idle before engine reset
- turing using hw-provided TSG info for CTXSW_TIMEOUT

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: kill channel on NV_PPBDMA_INTR_1_CTXNOTVALID

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: kill channel on a selection of PBDMA errors

A bunch of these can be handled in such a way that the channel can
continue, however, any of these are a pretty decent sign something
has gone horribly wrong, and the safest option is to disable the
channel.

This is a bit of a hack, we will want to handle these individually
and dump relevant debug info for each at some point.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chan/cgrp preempt()

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chan start()/stop()

- nvkm_chan_error() built on top, stops channel and sends 'killed' event
- removes an odd double-bashing of channel enable regs on kepler and up
- pokes doorbell on turing and up, after enabling channel

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chan bind()/unbind()

- stops programming (non-existent) runl id field on bind(), from maxwell

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add runlist block()/allow()

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add runlist wait()

- adds g8x/turing registers, which were missing before
- switches fermi to polled wait, like later hw (see: 4f2fc25c0f8bc...)

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add new engine context tracking

Channel groups have somewhat more complicated requirements than what we
currently support.  An engine context is shared between all channels in
a channel group, VEID/subctx support (later) brings per-VEID components,
and we need to track an individual channel's engine context pointers.

This commit adds the structures and refcounting to support the above,
wrapping the prior implementation for the moment.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add new channel lookup interfaces

- supports per-runlist CHIDs
- channel group lock held across reference, rather than global lock

v2:
- remove unnecessary parenthesis

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: merge mmu fault handlers together

After updating GF100 implementation from the GK104/TU102 ones, and using
the new runlist/engine topology info, all three handlers become (almost)
identical.

- there's a temporary kludge to call through to the HW-specific recovery
- engine fault mapping info determined at load time, not on every fault

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: move PBDMA intr to runq

- merges gf100/gk104- NV_PFIFO_INTR_0_PBDMA and NV_PPBDMA_INTR_0 code

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: move PBDMA init to runq

- bumps pbdma timeout to value RM uses on newer HW
- bumps fb timeout to max from boot default
- one/both of these greatly improves stability on // piglit runs

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: program NV_PFIFO_FB_TIMEOUT on init

NVGPU and RM both program this value.

Fixes a bunch of random hangs running parallel piglit.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: tidy global PBDMA init

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: tidy up non-stall intr handling

- removes a layer of indirection in the intr handling
- prevents non-stall ctrl racing with unknown intrs

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: use explicit intr interfaces

More control, and shallower call-chain to get to the point.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: use runlist engine info to lookup engine classes

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add cgrp, have all channels be part of one

Engine context tracking will move to nvkm_cgrp in later commits, so we
create SW-only channel groups on HW without support for them.

- switches to nvkm_chid for TSG/channel ID allocation

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: expose per-runlist CHID information

DRM uses this to setup fence-related items.

- nouveau_chan.runlist will always be "0" for the moment, not an issue
  as GPUs prior to ampere have system-wide channel IDs,

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: expose runlist topology info on all chipsets

Previously only available from Kepler onwards.

- also fixes the info() queries causing fifo init()/fini() unnecessarily

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add common runlist/engine topology

Creates an nvkm_runl for each runlist on the GPU, and an nvkm_engn for
each engine that is reachable from a runlist.

- basically what gk104- already does, but extended to all chips
- adds per-runlist CHID allocators (Ampere)
- splits g98/gt2xx out from g84 (different target engines)

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add runq

Creates an nvkm_runq for each PBDMA, these will be associated with the
relevant runlist(s) later.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>