gpu: nvgpu: support per-channel wdt timeouts

Replace the padding in nvgpu_channel_wdt_args with a timeout value in
milliseconds, and add NVGPU_IOCTL_CHANNEL_WDT_FLAG_SET_TIMEOUT to
signify the existence of this new field. When the new flag is included
in the value of wdt_status, the field is used to set a per-channel
timeout to override the per-GPU default.

Add NVGPU_IOCTL_CHANNEL_WDT_FLAG_DISABLE_DUMP to disable the long debug
dump when a timed out channel gets recovered by the watchdog. Printing
the dump to serial console takes easily several seconds. (Note that
there is NVGPU_TIMEOUT_FLAG_DISABLE_DUMP about ctxsw timeout separately
for NVGPU_IOCTL_CHANNEL_SET_TIMEOUT_EX as well.)

The behaviour of NVGPU_IOCTL_CHANNEL_WDT is changed so that either
NVGPU_IOCTL_CHANNEL_ENABLE_WDT or NVGPU_IOCTL_CHANNEL_DISABLE_WDT has to
be set. The old behaviour was that other values were silently ignored.

The usage of the global default debugfs-controlled ch_wdt_timeout_ms is
changed so that its value takes effect only for newly opened channels
instead of in realtime. Also, zero value no longer means that the
watchdog is disabled; there is a separate flag for that after all.

gk20a_fifo_recover_tsg used to ignore the value of "verbose" when no
engines were found. Correct this.

Bug 1982826
Bug 1985845
Jira NVGPU-73

Change-Id: Iea6213a646a66cb7c631ed7d7c91d8c2ba8a92a4
Signed-off-by: Konsta Holtta <kholtta@nvidia.com>
Reviewed-on: https://git-master.nvidia.com/r/1510898
Reviewed-by: mobile promotions <svcmobile_promotions@nvidia.com>
Tested-by: mobile promotions <svcmobile_promotions@nvidia.com>

1 files changed, 8 insertions, 1 deletions

diff --git a/drivers/gpu/nvgpu/gk20a/channel_gk20a.h b/drivers/gpu/nvgpu/gk20a/channel_gk20a.h
index edb645b5..947b8913 100644
--- a/drivers/gpu/nvgpu/gk20a/channel_gk20a.h
+++ b/drivers/gpu/nvgpu/gk20a/channel_gk20a.h
@@ -96,11 +96,17 @@ struct channel_gk20a_joblist {
 };
 
 struct channel_gk20a_timeout {
+        /* lock protects the running timer state */
         struct nvgpu_raw_spinlock lock;
         struct nvgpu_timeout timer;
         bool running;
         u32 gp_get;
         u64 pb_get;
+
+        /* lock not needed */
+        u32 limit_ms;
+        bool enabled;
+        bool debug_dump;
 };
 
 /*
@@ -167,7 +173,6 @@ struct channel_gk20a {
         struct nvgpu_semaphore_int *hw_sema;
 
         int chid;
-        bool wdt_enabled;
         nvgpu_atomic_t bound;
         bool vpr;
         bool deterministic;
@@ -203,7 +208,9 @@ struct channel_gk20a {
         u32 timeout_accumulated_ms;
         u32 timeout_gpfifo_get;
 
+        /* kernel watchdog to kill stuck jobs */
         struct channel_gk20a_timeout timeout;
+
         /* for job cleanup handling in the background worker */
         struct nvgpu_list_node worker_item;