drm/amdgpu: Track pending retry faults in IH and VM (v2)

IH tracks pending retry faults in a hash table for fast lookup in
interrupt context. Each VM has a short FIFO of pending VM faults for
processing in a bottom half.

The IH prescreening stage adds retry faults and filters out repeated
retry interrupts to minimize the impact of interrupt storms.

It's the VM's responsibility remove pending faults once they are
handled. For now this is only done when the VM is destroyed.

v2:
- Made the hash table smaller and the FIFO longer. I never want the
  FIFO to fill up, because that would make prescreen take longer.
  128 pending page faults should be enough to keep migrations busy.

Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Acked-by: Christian König <christian.koenig@amd.com> (v1)
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

1 files changed, 76 insertions, 0 deletions

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c
index c834a40cfad6..f5f27e4f0f7f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c
@@ -196,3 +196,79 @@ restart_ih:
 
         return IRQ_HANDLED;
 }
+
+/**
+ * amdgpu_ih_add_fault - Add a page fault record
+ *
+ * @adev: amdgpu device pointer
+ * @key: 64-bit encoding of PASID and address
+ *
+ * This should be called when a retry page fault interrupt is
+ * received. If this is a new page fault, it will be added to a hash
+ * table. The return value indicates whether this is a new fault, or
+ * a fault that was already known and is already being handled.
+ *
+ * If there are too many pending page faults, this will fail. Retry
+ * interrupts should be ignored in this case until there is enough
+ * free space.
+ *
+ * Returns 0 if the fault was added, 1 if the fault was already known,
+ * -ENOSPC if there are too many pending faults.
+ */
+int amdgpu_ih_add_fault(struct amdgpu_device *adev, u64 key)
+{
+        unsigned long flags;
+        int r = -ENOSPC;
+
+        if (WARN_ON_ONCE(!adev->irq.ih.faults))
+                /* Should be allocated in <IP>_ih_sw_init on GPUs that
+                 * support retry faults and require retry filtering.
+                 */
+                return r;
+
+        spin_lock_irqsave(&adev->irq.ih.faults->lock, flags);
+
+        /* Only let the hash table fill up to 50% for best performance */
+        if (adev->irq.ih.faults->count >= (1 << (AMDGPU_PAGEFAULT_HASH_BITS-1)))
+                goto unlock_out;
+
+        r = chash_table_copy_in(&adev->irq.ih.faults->hash, key, NULL);
+        if (!r)
+                adev->irq.ih.faults->count++;
+
+        /* chash_table_copy_in should never fail unless we're losing count */
+        WARN_ON_ONCE(r < 0);
+
+unlock_out:
+        spin_unlock_irqrestore(&adev->irq.ih.faults->lock, flags);
+        return r;
+}
+
+/**
+ * amdgpu_ih_clear_fault - Remove a page fault record
+ *
+ * @adev: amdgpu device pointer
+ * @key: 64-bit encoding of PASID and address
+ *
+ * This should be called when a page fault has been handled. Any
+ * future interrupt with this key will be processed as a new
+ * page fault.
+ */
+void amdgpu_ih_clear_fault(struct amdgpu_device *adev, u64 key)
+{
+        unsigned long flags;
+        int r;
+
+        if (!adev->irq.ih.faults)
+                return;
+
+        spin_lock_irqsave(&adev->irq.ih.faults->lock, flags);
+
+        r = chash_table_remove(&adev->irq.ih.faults->hash, key, NULL);
+        if (!WARN_ON_ONCE(r < 0)) {
+                adev->irq.ih.faults->count--;
+                WARN_ON_ONCE(adev->irq.ih.faults->count < 0);
+        }
+
+        spin_unlock_irqrestore(&adev->irq.ih.faults->lock, flags);
+}