powerpc/eeh: No hotplug on permanently removed dev

The issue was detected in a bit complicated test case where
we have multiple hierarchical PEs shown as following figure:

                +-----------------+
                | PE#3     p2p#0  |
                |          p2p#1  |
                +-----------------+
                        |
                +-----------------+
                | PE#4     pdev#0 |
                |          pdev#1 |
                +-----------------+

PE#4 (have 2 PCI devices) is the child of PE#3, which has 2 p2p
bridges. We accidentally had less-known scenario: PE#4 was removed
permanently from the system because of permanent failure (e.g.
exceeding the max allowd failure times in last hour), then we detects
EEH errors on PE#3 and tried to recover it. However, eeh_dev instances
for pdev#0/1 were not detached from PE#4, which was still connected to
PE#3. All of that was because of the fact that we rely on count-based
pcibios_release_device(), which isn't reliable enough. When doing
recovery for PE#3, we still apply hotplug on PE#4 and pdev#0/1, which
are not valid any more. Eventually, we run into kernel crash.

The patch fixes above issue from two aspects. For unplug, we simply
skip those permanently removed PE, whose state is (EEH_PE_STATE_ISOLATED
&& !EEH_PE_STATE_RECOVERING) and its frozen count should be greater
than EEH_MAX_ALLOWED_FREEZES. For plug, we marked all permanently
removed EEH devices with EEH_DEV_REMOVED and return 0xFF's on read
its PCI config so that PCI core will omit them.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>

1 files changed, 40 insertions, 7 deletions

diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c
index f0c353fa655a..995c2a284630 100644
--- a/arch/powerpc/kernel/eeh_pe.c
+++ b/arch/powerpc/kernel/eeh_pe.c
@@ -503,13 +503,17 @@ static void *__eeh_pe_state_mark(void *data, void *flag)
         struct eeh_dev *edev, *tmp;
         struct pci_dev *pdev;
 
-        /*
-         * Mark the PE with the indicated state. Also,
-         * the associated PCI device will be put into
-         * I/O frozen state to avoid I/O accesses from
-         * the PCI device driver.
-         */
+        /* Keep the state of permanently removed PE intact */
+        if ((pe->freeze_count > EEH_MAX_ALLOWED_FREEZES) &&
+            (state & (EEH_PE_ISOLATED | EEH_PE_RECOVERING)))
+                return NULL;
+
         pe->state |= state;
+
+        /* Offline PCI devices if applicable */
+        if (state != EEH_PE_ISOLATED)
+                return NULL;
+
         eeh_pe_for_each_dev(pe, edev, tmp) {
                 pdev = eeh_dev_to_pci_dev(edev);
                 if (pdev)
@@ -532,6 +536,27 @@ void eeh_pe_state_mark(struct eeh_pe *pe, int state)
         eeh_pe_traverse(pe, __eeh_pe_state_mark, &state);
 }
 
+static void *__eeh_pe_dev_mode_mark(void *data, void *flag)
+{
+        struct eeh_dev *edev = data;
+        int mode = *((int *)flag);
+
+        edev->mode |= mode;
+
+        return NULL;
+}
+
+/**
+ * eeh_pe_dev_state_mark - Mark state for all device under the PE
+ * @pe: EEH PE
+ *
+ * Mark specific state for all child devices of the PE.
+ */
+void eeh_pe_dev_mode_mark(struct eeh_pe *pe, int mode)
+{
+        eeh_pe_dev_traverse(pe, __eeh_pe_dev_mode_mark, &mode);
+}
+
 /**
  * __eeh_pe_state_clear - Clear state for the PE
  * @data: EEH PE
@@ -546,8 +571,16 @@ static void *__eeh_pe_state_clear(void *data, void *flag)
         struct eeh_pe *pe = (struct eeh_pe *)data;
         int state = *((int *)flag);
 
+        /* Keep the state of permanently removed PE intact */
+        if ((pe->freeze_count > EEH_MAX_ALLOWED_FREEZES) &&
+            (state & EEH_PE_ISOLATED))
+                return NULL;
+
         pe->state &= ~state;
-        pe->check_count = 0;
+
+        /* Clear check count since last isolation */
+        if (state & EEH_PE_ISOLATED)
+                pe->check_count = 0;
 
         return NULL;
 }