aboutsummaryrefslogtreecommitdiffstats
path: root/Documentation/pci-error-recovery.txt
diff options
context:
space:
mode:
Diffstat (limited to 'Documentation/pci-error-recovery.txt')
-rw-r--r--Documentation/pci-error-recovery.txt246
1 files changed, 246 insertions, 0 deletions
diff --git a/Documentation/pci-error-recovery.txt b/Documentation/pci-error-recovery.txt
new file mode 100644
index 000000000000..d089967e4948
--- /dev/null
+++ b/Documentation/pci-error-recovery.txt
@@ -0,0 +1,246 @@
1
2 PCI Error Recovery
3 ------------------
4 May 31, 2005
5
6 Current document maintainer:
7 Linas Vepstas <linas@austin.ibm.com>
8
9
10Some PCI bus controllers are able to detect certain "hard" PCI errors
11on the bus, such as parity errors on the data and address busses, as
12well as SERR and PERR errors. These chipsets are then able to disable
13I/O to/from the affected device, so that, for example, a bad DMA
14address doesn't end up corrupting system memory. These same chipsets
15are also able to reset the affected PCI device, and return it to
16working condition. This document describes a generic API form
17performing error recovery.
18
19The core idea is that after a PCI error has been detected, there must
20be a way for the kernel to coordinate with all affected device drivers
21so that the pci card can be made operational again, possibly after
22performing a full electrical #RST of the PCI card. The API below
23provides a generic API for device drivers to be notified of PCI
24errors, and to be notified of, and respond to, a reset sequence.
25
26Preliminary sketch of API, cut-n-pasted-n-modified email from
27Ben Herrenschmidt, circa 5 april 2005
28
29The error recovery API support is exposed to the driver in the form of
30a structure of function pointers pointed to by a new field in struct
31pci_driver. The absence of this pointer in pci_driver denotes an
32"non-aware" driver, behaviour on these is platform dependant.
33Platforms like ppc64 can try to simulate pci hotplug remove/add.
34
35The definition of "pci_error_token" is not covered here. It is based on
36Seto's work on the synchronous error detection. We still need to define
37functions for extracting infos out of an opaque error token. This is
38separate from this API.
39
40This structure has the form:
41
42struct pci_error_handlers
43{
44 int (*error_detected)(struct pci_dev *dev, pci_error_token error);
45 int (*mmio_enabled)(struct pci_dev *dev);
46 int (*resume)(struct pci_dev *dev);
47 int (*link_reset)(struct pci_dev *dev);
48 int (*slot_reset)(struct pci_dev *dev);
49};
50
51A driver doesn't have to implement all of these callbacks. The
52only mandatory one is error_detected(). If a callback is not
53implemented, the corresponding feature is considered unsupported.
54For example, if mmio_enabled() and resume() aren't there, then the
55driver is assumed as not doing any direct recovery and requires
56a reset. If link_reset() is not implemented, the card is assumed as
57not caring about link resets, in which case, if recover is supported,
58the core can try recover (but not slot_reset() unless it really did
59reset the slot). If slot_reset() is not supported, link_reset() can
60be called instead on a slot reset.
61
62At first, the call will always be :
63
64 1) error_detected()
65
66 Error detected. This is sent once after an error has been detected. At
67this point, the device might not be accessible anymore depending on the
68platform (the slot will be isolated on ppc64). The driver may already
69have "noticed" the error because of a failing IO, but this is the proper
70"synchronisation point", that is, it gives a chance to the driver to
71cleanup, waiting for pending stuff (timers, whatever, etc...) to
72complete; it can take semaphores, schedule, etc... everything but touch
73the device. Within this function and after it returns, the driver
74shouldn't do any new IOs. Called in task context. This is sort of a
75"quiesce" point. See note about interrupts at the end of this doc.
76
77 Result codes:
78 - PCIERR_RESULT_CAN_RECOVER:
79 Driever returns this if it thinks it might be able to recover
80 the HW by just banging IOs or if it wants to be given
81 a chance to extract some diagnostic informations (see
82 below).
83 - PCIERR_RESULT_NEED_RESET:
84 Driver returns this if it thinks it can't recover unless the
85 slot is reset.
86 - PCIERR_RESULT_DISCONNECT:
87 Return this if driver thinks it won't recover at all,
88 (this will detach the driver ? or just leave it
89 dangling ? to be decided)
90
91So at this point, we have called error_detected() for all drivers
92on the segment that had the error. On ppc64, the slot is isolated. What
93happens now typically depends on the result from the drivers. If all
94drivers on the segment/slot return PCIERR_RESULT_CAN_RECOVER, we would
95re-enable IOs on the slot (or do nothing special if the platform doesn't
96isolate slots) and call 2). If not and we can reset slots, we go to 4),
97if neither, we have a dead slot. If it's an hotplug slot, we might
98"simulate" reset by triggering HW unplug/replug though.
99
100>>> Current ppc64 implementation assumes that a device driver will
101>>> *not* schedule or semaphore in this routine; the current ppc64
102>>> implementation uses one kernel thread to notify all devices;
103>>> thus, of one device sleeps/schedules, all devices are affected.
104>>> Doing better requires complex multi-threaded logic in the error
105>>> recovery implementation (e.g. waiting for all notification threads
106>>> to "join" before proceeding with recovery.) This seems excessively
107>>> complex and not worth implementing.
108
109>>> The current ppc64 implementation doesn't much care if the device
110>>> attempts i/o at this point, or not. I/O's will fail, returning
111>>> a value of 0xff on read, and writes will be dropped. If the device
112>>> driver attempts more than 10K I/O's to a frozen adapter, it will
113>>> assume that the device driver has gone into an infinite loop, and
114>>> it will panic the the kernel.
115
116 2) mmio_enabled()
117
118 This is the "early recovery" call. IOs are allowed again, but DMA is
119not (hrm... to be discussed, I prefer not), with some restrictions. This
120is NOT a callback for the driver to start operations again, only to
121peek/poke at the device, extract diagnostic information, if any, and
122eventually do things like trigger a device local reset or some such,
123but not restart operations. This is sent if all drivers on a segment
124agree that they can try to recover and no automatic link reset was
125performed by the HW. If the platform can't just re-enable IOs without
126a slot reset or a link reset, it doesn't call this callback and goes
127directly to 3) or 4). All IOs should be done _synchronously_ from
128within this callback, errors triggered by them will be returned via
129the normal pci_check_whatever() api, no new error_detected() callback
130will be issued due to an error happening here. However, such an error
131might cause IOs to be re-blocked for the whole segment, and thus
132invalidate the recovery that other devices on the same segment might
133have done, forcing the whole segment into one of the next states,
134that is link reset or slot reset.
135
136 Result codes:
137 - PCIERR_RESULT_RECOVERED
138 Driver returns this if it thinks the device is fully
139 functionnal and thinks it is ready to start
140 normal driver operations again. There is no
141 guarantee that the driver will actually be
142 allowed to proceed, as another driver on the
143 same segment might have failed and thus triggered a
144 slot reset on platforms that support it.
145
146 - PCIERR_RESULT_NEED_RESET
147 Driver returns this if it thinks the device is not
148 recoverable in it's current state and it needs a slot
149 reset to proceed.
150
151 - PCIERR_RESULT_DISCONNECT
152 Same as above. Total failure, no recovery even after
153 reset driver dead. (To be defined more precisely)
154
155>>> The current ppc64 implementation does not implement this callback.
156
157 3) link_reset()
158
159 This is called after the link has been reset. This is typically
160a PCI Express specific state at this point and is done whenever a
161non-fatal error has been detected that can be "solved" by resetting
162the link. This call informs the driver of the reset and the driver
163should check if the device appears to be in working condition.
164This function acts a bit like 2) mmio_enabled(), in that the driver
165is not supposed to restart normal driver I/O operations right away.
166Instead, it should just "probe" the device to check it's recoverability
167status. If all is right, then the core will call resume() once all
168drivers have ack'd link_reset().
169
170 Result codes:
171 (identical to mmio_enabled)
172
173>>> The current ppc64 implementation does not implement this callback.
174
175 4) slot_reset()
176
177 This is called after the slot has been soft or hard reset by the
178platform. A soft reset consists of asserting the adapter #RST line
179and then restoring the PCI BARs and PCI configuration header. If the
180platform supports PCI hotplug, then it might instead perform a hard
181reset by toggling power on the slot off/on. This call gives drivers
182the chance to re-initialize the hardware (re-download firmware, etc.),
183but drivers shouldn't restart normal I/O processing operations at
184this point. (See note about interrupts; interrupts aren't guaranteed
185to be delivered until the resume() callback has been called). If all
186device drivers report success on this callback, the patform will call
187resume() to complete the error handling and let the driver restart
188normal I/O processing.
189
190A driver can still return a critical failure for this function if
191it can't get the device operational after reset. If the platform
192previously tried a soft reset, it migh now try a hard reset (power
193cycle) and then call slot_reset() again. It the device still can't
194be recovered, there is nothing more that can be done; the platform
195will typically report a "permanent failure" in such a case. The
196device will be considered "dead" in this case.
197
198 Result codes:
199 - PCIERR_RESULT_DISCONNECT
200 Same as above.
201
202>>> The current ppc64 implementation does not try a power-cycle reset
203>>> if the driver returned PCIERR_RESULT_DISCONNECT. However, it should.
204
205 5) resume()
206
207 This is called if all drivers on the segment have returned
208PCIERR_RESULT_RECOVERED from one of the 3 prevous callbacks.
209That basically tells the driver to restart activity, tht everything
210is back and running. No result code is taken into account here. If
211a new error happens, it will restart a new error handling process.
212
213That's it. I think this covers all the possibilities. The way those
214callbacks are called is platform policy. A platform with no slot reset
215capability for example may want to just "ignore" drivers that can't
216recover (disconnect them) and try to let other cards on the same segment
217recover. Keep in mind that in most real life cases, though, there will
218be only one driver per segment.
219
220Now, there is a note about interrupts. If you get an interrupt and your
221device is dead or has been isolated, there is a problem :)
222
223After much thinking, I decided to leave that to the platform. That is,
224the recovery API only precies that:
225
226 - There is no guarantee that interrupt delivery can proceed from any
227device on the segment starting from the error detection and until the
228restart callback is sent, at which point interrupts are expected to be
229fully operational.
230
231 - There is no guarantee that interrupt delivery is stopped, that is, ad
232river that gets an interrupts after detecting an error, or that detects
233and error within the interrupt handler such that it prevents proper
234ack'ing of the interrupt (and thus removal of the source) should just
235return IRQ_NOTHANDLED. It's up to the platform to deal with taht
236condition, typically by masking the irq source during the duration of
237the error handling. It is expected that the platform "knows" which
238interrupts are routed to error-management capable slots and can deal
239with temporarily disabling that irq number during error processing (this
240isn't terribly complex). That means some IRQ latency for other devices
241sharing the interrupt, but there is simply no other way. High end
242platforms aren't supposed to share interrupts between many devices
243anyway :)
244
245
246Revised: 31 May 2005 Linas Vepstas <linas@austin.ibm.com>