habanalabs: increase timeout during reset
authorOded Gabbay <oded.gabbay@gmail.com>
Fri, 27 Mar 2020 13:38:37 +0000 (16:38 +0300)
committerOded Gabbay <oded.gabbay@gmail.com>
Sun, 17 May 2020 09:06:22 +0000 (12:06 +0300)
commit7a65ee046b2238e053f6ebb610e1a082cfc49490
tree6a70b06695d9ac9b28ad83df81263c6c546321df
parent49aba0bbab20a581dc3e32a6ee636c07a542eb9e
habanalabs: increase timeout during reset

When doing training, the DL framework (e.g. tensorflow) performs hundreds
of thousands of memory allocations and mappings. In case the driver needs
to perform hard-reset during training, the driver kills the application and
unmaps all those memory allocations. Unfortunately, because of that large
amount of mappings, the driver isn't able to do that in the current timeout
(5 seconds). Therefore, increase the timeout significantly to 30 seconds
to avoid situation where the driver resets the device with active mappings,
which sometime can cause a kernel bug.

BTW, it doesn't mean we will spend all the 30 seconds because the reset
thread checks every one second if the unmap operation is done.

Reviewed-by: Omer Shpigelman <oshpigelman@habana.ai>
Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>
drivers/misc/habanalabs/habanalabs.h