diff options
author | Linus Torvalds <torvalds@linux-foundation.org> | 2009-04-01 12:47:12 -0400 |
---|---|---|
committer | Linus Torvalds <torvalds@linux-foundation.org> | 2009-04-01 12:47:12 -0400 |
commit | e76e5b2c663ac74ae6a542ac20795c625e36a5cd (patch) | |
tree | 2e7271be1f3a26832f4b121839fc4044fbbf27a6 /Documentation | |
parent | 32527bc0e4b4fa7711ad1c923cf64ae72a7ffd9d (diff) | |
parent | eeafda70bf2807544e96fa4e52b2433cd470ff46 (diff) |
Merge branch 'linux-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jbarnes/pci-2.6
* 'linux-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jbarnes/pci-2.6: (88 commits)
PCI: fix HT MSI mapping fix
PCI: don't enable too much HT MSI mapping
x86/PCI: make pci=lastbus=255 work when acpi is on
PCI: save and restore PCIe 2.0 registers
PCI: update fakephp for bus_id removal
PCI: fix kernel oops on bridge removal
PCI: fix conflict between SR-IOV and config space sizing
powerpc/PCI: include pci.h in powerpc MSI implementation
PCI Hotplug: schedule fakephp for feature removal
PCI Hotplug: rename legacy_fakephp to fakephp
PCI Hotplug: restore fakephp interface with complete reimplementation
PCI: Introduce /sys/bus/pci/devices/.../rescan
PCI: Introduce /sys/bus/pci/devices/.../remove
PCI: Introduce /sys/bus/pci/rescan
PCI: Introduce pci_rescan_bus()
PCI: do not enable bridges more than once
PCI: do not initialize bridges more than once
PCI: always scan child buses
PCI: pci_scan_slot() returns newly found devices
PCI: don't scan existing devices
...
Fix trivial append-only conflict in Documentation/feature-removal-schedule.txt
Diffstat (limited to 'Documentation')
-rw-r--r-- | Documentation/ABI/testing/sysfs-bus-pci | 70 | ||||
-rw-r--r-- | Documentation/DocBook/kernel-api.tmpl | 1 | ||||
-rw-r--r-- | Documentation/PCI/MSI-HOWTO.txt | 814 | ||||
-rw-r--r-- | Documentation/PCI/pci-iov-howto.txt | 99 | ||||
-rw-r--r-- | Documentation/feature-removal-schedule.txt | 32 | ||||
-rw-r--r-- | Documentation/filesystems/sysfs-pci.txt | 10 | ||||
-rw-r--r-- | Documentation/kernel-parameters.txt | 11 |
7 files changed, 555 insertions, 482 deletions
diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci index e638e15a8895..97ad190e13af 100644 --- a/Documentation/ABI/testing/sysfs-bus-pci +++ b/Documentation/ABI/testing/sysfs-bus-pci | |||
@@ -41,6 +41,49 @@ Description: | |||
41 | for the device and attempt to bind to it. For example: | 41 | for the device and attempt to bind to it. For example: |
42 | # echo "8086 10f5" > /sys/bus/pci/drivers/foo/new_id | 42 | # echo "8086 10f5" > /sys/bus/pci/drivers/foo/new_id |
43 | 43 | ||
44 | What: /sys/bus/pci/drivers/.../remove_id | ||
45 | Date: February 2009 | ||
46 | Contact: Chris Wright <chrisw@sous-sol.org> | ||
47 | Description: | ||
48 | Writing a device ID to this file will remove an ID | ||
49 | that was dynamically added via the new_id sysfs entry. | ||
50 | The format for the device ID is: | ||
51 | VVVV DDDD SVVV SDDD CCCC MMMM. That is Vendor ID, Device | ||
52 | ID, Subsystem Vendor ID, Subsystem Device ID, Class, | ||
53 | and Class Mask. The Vendor ID and Device ID fields are | ||
54 | required, the rest are optional. After successfully | ||
55 | removing an ID, the driver will no longer support the | ||
56 | device. This is useful to ensure auto probing won't | ||
57 | match the driver to the device. For example: | ||
58 | # echo "8086 10f5" > /sys/bus/pci/drivers/foo/remove_id | ||
59 | |||
60 | What: /sys/bus/pci/rescan | ||
61 | Date: January 2009 | ||
62 | Contact: Linux PCI developers <linux-pci@vger.kernel.org> | ||
63 | Description: | ||
64 | Writing a non-zero value to this attribute will | ||
65 | force a rescan of all PCI buses in the system, and | ||
66 | re-discover previously removed devices. | ||
67 | Depends on CONFIG_HOTPLUG. | ||
68 | |||
69 | What: /sys/bus/pci/devices/.../remove | ||
70 | Date: January 2009 | ||
71 | Contact: Linux PCI developers <linux-pci@vger.kernel.org> | ||
72 | Description: | ||
73 | Writing a non-zero value to this attribute will | ||
74 | hot-remove the PCI device and any of its children. | ||
75 | Depends on CONFIG_HOTPLUG. | ||
76 | |||
77 | What: /sys/bus/pci/devices/.../rescan | ||
78 | Date: January 2009 | ||
79 | Contact: Linux PCI developers <linux-pci@vger.kernel.org> | ||
80 | Description: | ||
81 | Writing a non-zero value to this attribute will | ||
82 | force a rescan of the device's parent bus and all | ||
83 | child buses, and re-discover devices removed earlier | ||
84 | from this part of the device tree. | ||
85 | Depends on CONFIG_HOTPLUG. | ||
86 | |||
44 | What: /sys/bus/pci/devices/.../vpd | 87 | What: /sys/bus/pci/devices/.../vpd |
45 | Date: February 2008 | 88 | Date: February 2008 |
46 | Contact: Ben Hutchings <bhutchings@solarflare.com> | 89 | Contact: Ben Hutchings <bhutchings@solarflare.com> |
@@ -52,3 +95,30 @@ Description: | |||
52 | that some devices may have malformatted data. If the | 95 | that some devices may have malformatted data. If the |
53 | underlying VPD has a writable section then the | 96 | underlying VPD has a writable section then the |
54 | corresponding section of this file will be writable. | 97 | corresponding section of this file will be writable. |
98 | |||
99 | What: /sys/bus/pci/devices/.../virtfnN | ||
100 | Date: March 2009 | ||
101 | Contact: Yu Zhao <yu.zhao@intel.com> | ||
102 | Description: | ||
103 | This symbolic link appears when hardware supports the SR-IOV | ||
104 | capability and the Physical Function driver has enabled it. | ||
105 | The symbolic link points to the PCI device sysfs entry of the | ||
106 | Virtual Function whose index is N (0...MaxVFs-1). | ||
107 | |||
108 | What: /sys/bus/pci/devices/.../dep_link | ||
109 | Date: March 2009 | ||
110 | Contact: Yu Zhao <yu.zhao@intel.com> | ||
111 | Description: | ||
112 | This symbolic link appears when hardware supports the SR-IOV | ||
113 | capability and the Physical Function driver has enabled it, | ||
114 | and this device has vendor specific dependencies with others. | ||
115 | The symbolic link points to the PCI device sysfs entry of | ||
116 | Physical Function this device depends on. | ||
117 | |||
118 | What: /sys/bus/pci/devices/.../physfn | ||
119 | Date: March 2009 | ||
120 | Contact: Yu Zhao <yu.zhao@intel.com> | ||
121 | Description: | ||
122 | This symbolic link appears when a device is a Virtual Function. | ||
123 | The symbolic link points to the PCI device sysfs entry of the | ||
124 | Physical Function this device associates with. | ||
diff --git a/Documentation/DocBook/kernel-api.tmpl b/Documentation/DocBook/kernel-api.tmpl index bc962cda6504..58c194572c76 100644 --- a/Documentation/DocBook/kernel-api.tmpl +++ b/Documentation/DocBook/kernel-api.tmpl | |||
@@ -199,6 +199,7 @@ X!Edrivers/pci/hotplug.c | |||
199 | --> | 199 | --> |
200 | !Edrivers/pci/probe.c | 200 | !Edrivers/pci/probe.c |
201 | !Edrivers/pci/rom.c | 201 | !Edrivers/pci/rom.c |
202 | !Edrivers/pci/iov.c | ||
202 | </sect1> | 203 | </sect1> |
203 | <sect1><title>PCI Hotplug Support Library</title> | 204 | <sect1><title>PCI Hotplug Support Library</title> |
204 | !Edrivers/pci/hotplug/pci_hotplug_core.c | 205 | !Edrivers/pci/hotplug/pci_hotplug_core.c |
diff --git a/Documentation/PCI/MSI-HOWTO.txt b/Documentation/PCI/MSI-HOWTO.txt index 256defd7e174..dcf7acc720e1 100644 --- a/Documentation/PCI/MSI-HOWTO.txt +++ b/Documentation/PCI/MSI-HOWTO.txt | |||
@@ -4,506 +4,356 @@ | |||
4 | Revised Feb 12, 2004 by Martine Silbermann | 4 | Revised Feb 12, 2004 by Martine Silbermann |
5 | email: Martine.Silbermann@hp.com | 5 | email: Martine.Silbermann@hp.com |
6 | Revised Jun 25, 2004 by Tom L Nguyen | 6 | Revised Jun 25, 2004 by Tom L Nguyen |
7 | Revised Jul 9, 2008 by Matthew Wilcox <willy@linux.intel.com> | ||
8 | Copyright 2003, 2008 Intel Corporation | ||
7 | 9 | ||
8 | 1. About this guide | 10 | 1. About this guide |
9 | 11 | ||
10 | This guide describes the basics of Message Signaled Interrupts (MSI), | 12 | This guide describes the basics of Message Signaled Interrupts (MSIs), |
11 | the advantages of using MSI over traditional interrupt mechanisms, | 13 | the advantages of using MSI over traditional interrupt mechanisms, how |
12 | and how to enable your driver to use MSI or MSI-X. Also included is | 14 | to change your driver to use MSI or MSI-X and some basic diagnostics to |
13 | a Frequently Asked Questions (FAQ) section. | 15 | try if a device doesn't support MSIs. |
14 | |||
15 | 1.1 Terminology | ||
16 | |||
17 | PCI devices can be single-function or multi-function. In either case, | ||
18 | when this text talks about enabling or disabling MSI on a "device | ||
19 | function," it is referring to one specific PCI device and function and | ||
20 | not to all functions on a PCI device (unless the PCI device has only | ||
21 | one function). | ||
22 | |||
23 | 2. Copyright 2003 Intel Corporation | ||
24 | |||
25 | 3. What is MSI/MSI-X? | ||
26 | |||
27 | Message Signaled Interrupt (MSI), as described in the PCI Local Bus | ||
28 | Specification Revision 2.3 or later, is an optional feature, and a | ||
29 | required feature for PCI Express devices. MSI enables a device function | ||
30 | to request service by sending an Inbound Memory Write on its PCI bus to | ||
31 | the FSB as a Message Signal Interrupt transaction. Because MSI is | ||
32 | generated in the form of a Memory Write, all transaction conditions, | ||
33 | such as a Retry, Master-Abort, Target-Abort or normal completion, are | ||
34 | supported. | ||
35 | |||
36 | A PCI device that supports MSI must also support pin IRQ assertion | ||
37 | interrupt mechanism to provide backward compatibility for systems that | ||
38 | do not support MSI. In systems which support MSI, the bus driver is | ||
39 | responsible for initializing the message address and message data of | ||
40 | the device function's MSI/MSI-X capability structure during device | ||
41 | initial configuration. | ||
42 | |||
43 | An MSI capable device function indicates MSI support by implementing | ||
44 | the MSI/MSI-X capability structure in its PCI capability list. The | ||
45 | device function may implement both the MSI capability structure and | ||
46 | the MSI-X capability structure; however, the bus driver should not | ||
47 | enable both. | ||
48 | |||
49 | The MSI capability structure contains Message Control register, | ||
50 | Message Address register and Message Data register. These registers | ||
51 | provide the bus driver control over MSI. The Message Control register | ||
52 | indicates the MSI capability supported by the device. The Message | ||
53 | Address register specifies the target address and the Message Data | ||
54 | register specifies the characteristics of the message. To request | ||
55 | service, the device function writes the content of the Message Data | ||
56 | register to the target address. The device and its software driver | ||
57 | are prohibited from writing to these registers. | ||
58 | |||
59 | The MSI-X capability structure is an optional extension to MSI. It | ||
60 | uses an independent and separate capability structure. There are | ||
61 | some key advantages to implementing the MSI-X capability structure | ||
62 | over the MSI capability structure as described below. | ||
63 | |||
64 | - Support a larger maximum number of vectors per function. | ||
65 | |||
66 | - Provide the ability for system software to configure | ||
67 | each vector with an independent message address and message | ||
68 | data, specified by a table that resides in Memory Space. | ||
69 | |||
70 | - MSI and MSI-X both support per-vector masking. Per-vector | ||
71 | masking is an optional extension of MSI but a required | ||
72 | feature for MSI-X. Per-vector masking provides the kernel the | ||
73 | ability to mask/unmask a single MSI while running its | ||
74 | interrupt service routine. If per-vector masking is | ||
75 | not supported, then the device driver should provide the | ||
76 | hardware/software synchronization to ensure that the device | ||
77 | generates MSI when the driver wants it to do so. | ||
78 | |||
79 | 4. Why use MSI? | ||
80 | |||
81 | As a benefit to the simplification of board design, MSI allows board | ||
82 | designers to remove out-of-band interrupt routing. MSI is another | ||
83 | step towards a legacy-free environment. | ||
84 | |||
85 | Due to increasing pressure on chipset and processor packages to | ||
86 | reduce pin count, the need for interrupt pins is expected to | ||
87 | diminish over time. Devices, due to pin constraints, may implement | ||
88 | messages to increase performance. | ||
89 | |||
90 | PCI Express endpoints uses INTx emulation (in-band messages) instead | ||
91 | of IRQ pin assertion. Using INTx emulation requires interrupt | ||
92 | sharing among devices connected to the same node (PCI bridge) while | ||
93 | MSI is unique (non-shared) and does not require BIOS configuration | ||
94 | support. As a result, the PCI Express technology requires MSI | ||
95 | support for better interrupt performance. | ||
96 | |||
97 | Using MSI enables the device functions to support two or more | ||
98 | vectors, which can be configured to target different CPUs to | ||
99 | increase scalability. | ||
100 | |||
101 | 5. Configuring a driver to use MSI/MSI-X | ||
102 | |||
103 | By default, the kernel will not enable MSI/MSI-X on all devices that | ||
104 | support this capability. The CONFIG_PCI_MSI kernel option | ||
105 | must be selected to enable MSI/MSI-X support. | ||
106 | |||
107 | 5.1 Including MSI/MSI-X support into the kernel | ||
108 | |||
109 | To allow MSI/MSI-X capable device drivers to selectively enable | ||
110 | MSI/MSI-X (using pci_enable_msi()/pci_enable_msix() as described | ||
111 | below), the VECTOR based scheme needs to be enabled by setting | ||
112 | CONFIG_PCI_MSI during kernel config. | ||
113 | |||
114 | Since the target of the inbound message is the local APIC, providing | ||
115 | CONFIG_X86_LOCAL_APIC must be enabled as well as CONFIG_PCI_MSI. | ||
116 | |||
117 | 5.2 Configuring for MSI support | ||
118 | |||
119 | Due to the non-contiguous fashion in vector assignment of the | ||
120 | existing Linux kernel, this version does not support multiple | ||
121 | messages regardless of a device function is capable of supporting | ||
122 | more than one vector. To enable MSI on a device function's MSI | ||
123 | capability structure requires a device driver to call the function | ||
124 | pci_enable_msi() explicitly. | ||
125 | |||
126 | 5.2.1 API pci_enable_msi | ||
127 | 16 | ||
128 | int pci_enable_msi(struct pci_dev *dev) | ||
129 | 17 | ||
130 | With this new API, a device driver that wants to have MSI | 18 | 2. What are MSIs? |
131 | enabled on its device function must call this API to enable MSI. | ||
132 | A successful call will initialize the MSI capability structure | ||
133 | with ONE vector, regardless of whether a device function is | ||
134 | capable of supporting multiple messages. This vector replaces the | ||
135 | pre-assigned dev->irq with a new MSI vector. To avoid a conflict | ||
136 | of the new assigned vector with existing pre-assigned vector requires | ||
137 | a device driver to call this API before calling request_irq(). | ||
138 | 19 | ||
139 | 5.2.2 API pci_disable_msi | 20 | A Message Signaled Interrupt is a write from the device to a special |
21 | address which causes an interrupt to be received by the CPU. | ||
140 | 22 | ||
141 | void pci_disable_msi(struct pci_dev *dev) | 23 | The MSI capability was first specified in PCI 2.2 and was later enhanced |
24 | in PCI 3.0 to allow each interrupt to be masked individually. The MSI-X | ||
25 | capability was also introduced with PCI 3.0. It supports more interrupts | ||
26 | per device than MSI and allows interrupts to be independently configured. | ||
142 | 27 | ||
143 | This API should always be used to undo the effect of pci_enable_msi() | 28 | Devices may support both MSI and MSI-X, but only one can be enabled at |
144 | when a device driver is unloading. This API restores dev->irq with | 29 | a time. |
145 | the pre-assigned IOAPIC vector and switches a device's interrupt | ||
146 | mode to PCI pin-irq assertion/INTx emulation mode. | ||
147 | |||
148 | Note that a device driver should always call free_irq() on the MSI vector | ||
149 | that it has done request_irq() on before calling this API. Failure to do | ||
150 | so results in a BUG_ON() and a device will be left with MSI enabled and | ||
151 | leaks its vector. | ||
152 | |||
153 | 5.2.3 MSI mode vs. legacy mode diagram | ||
154 | |||
155 | The below diagram shows the events which switch the interrupt | ||
156 | mode on the MSI-capable device function between MSI mode and | ||
157 | PIN-IRQ assertion mode. | ||
158 | |||
159 | ------------ pci_enable_msi ------------------------ | ||
160 | | | <=============== | | | ||
161 | | MSI MODE | | PIN-IRQ ASSERTION MODE | | ||
162 | | | ===============> | | | ||
163 | ------------ pci_disable_msi ------------------------ | ||
164 | |||
165 | |||
166 | Figure 1. MSI Mode vs. Legacy Mode | ||
167 | |||
168 | In Figure 1, a device operates by default in legacy mode. Legacy | ||
169 | in this context means PCI pin-irq assertion or PCI-Express INTx | ||
170 | emulation. A successful MSI request (using pci_enable_msi()) switches | ||
171 | a device's interrupt mode to MSI mode. A pre-assigned IOAPIC vector | ||
172 | stored in dev->irq will be saved by the PCI subsystem and a new | ||
173 | assigned MSI vector will replace dev->irq. | ||
174 | |||
175 | To return back to its default mode, a device driver should always call | ||
176 | pci_disable_msi() to undo the effect of pci_enable_msi(). Note that a | ||
177 | device driver should always call free_irq() on the MSI vector it has | ||
178 | done request_irq() on before calling pci_disable_msi(). Failure to do | ||
179 | so results in a BUG_ON() and a device will be left with MSI enabled and | ||
180 | leaks its vector. Otherwise, the PCI subsystem restores a device's | ||
181 | dev->irq with a pre-assigned IOAPIC vector and marks the released | ||
182 | MSI vector as unused. | ||
183 | |||
184 | Once being marked as unused, there is no guarantee that the PCI | ||
185 | subsystem will reserve this MSI vector for a device. Depending on | ||
186 | the availability of current PCI vector resources and the number of | ||
187 | MSI/MSI-X requests from other drivers, this MSI may be re-assigned. | ||
188 | |||
189 | For the case where the PCI subsystem re-assigns this MSI vector to | ||
190 | another driver, a request to switch back to MSI mode may result | ||
191 | in being assigned a different MSI vector or a failure if no more | ||
192 | vectors are available. | ||
193 | |||
194 | 5.3 Configuring for MSI-X support | ||
195 | |||
196 | Due to the ability of the system software to configure each vector of | ||
197 | the MSI-X capability structure with an independent message address | ||
198 | and message data, the non-contiguous fashion in vector assignment of | ||
199 | the existing Linux kernel has no impact on supporting multiple | ||
200 | messages on an MSI-X capable device functions. To enable MSI-X on | ||
201 | a device function's MSI-X capability structure requires its device | ||
202 | driver to call the function pci_enable_msix() explicitly. | ||
203 | |||
204 | The function pci_enable_msix(), once invoked, enables either | ||
205 | all or nothing, depending on the current availability of PCI vector | ||
206 | resources. If the PCI vector resources are available for the number | ||
207 | of vectors requested by a device driver, this function will configure | ||
208 | the MSI-X table of the MSI-X capability structure of a device with | ||
209 | requested messages. To emphasize this reason, for example, a device | ||
210 | may be capable for supporting the maximum of 32 vectors while its | ||
211 | software driver usually may request 4 vectors. It is recommended | ||
212 | that the device driver should call this function once during the | ||
213 | initialization phase of the device driver. | ||
214 | |||
215 | Unlike the function pci_enable_msi(), the function pci_enable_msix() | ||
216 | does not replace the pre-assigned IOAPIC dev->irq with a new MSI | ||
217 | vector because the PCI subsystem writes the 1:1 vector-to-entry mapping | ||
218 | into the field vector of each element contained in a second argument. | ||
219 | Note that the pre-assigned IOAPIC dev->irq is valid only if the device | ||
220 | operates in PIN-IRQ assertion mode. In MSI-X mode, any attempt at | ||
221 | using dev->irq by the device driver to request for interrupt service | ||
222 | may result in unpredictable behavior. | ||
223 | |||
224 | For each MSI-X vector granted, a device driver is responsible for calling | ||
225 | other functions like request_irq(), enable_irq(), etc. to enable | ||
226 | this vector with its corresponding interrupt service handler. It is | ||
227 | a device driver's choice to assign all vectors with the same | ||
228 | interrupt service handler or each vector with a unique interrupt | ||
229 | service handler. | ||
230 | |||
231 | 5.3.1 Handling MMIO address space of MSI-X Table | ||
232 | |||
233 | The PCI 3.0 specification has implementation notes that MMIO address | ||
234 | space for a device's MSI-X structure should be isolated so that the | ||
235 | software system can set different pages for controlling accesses to the | ||
236 | MSI-X structure. The implementation of MSI support requires the PCI | ||
237 | subsystem, not a device driver, to maintain full control of the MSI-X | ||
238 | table/MSI-X PBA (Pending Bit Array) and MMIO address space of the MSI-X | ||
239 | table/MSI-X PBA. A device driver should not access the MMIO address | ||
240 | space of the MSI-X table/MSI-X PBA. | ||
241 | |||
242 | 5.3.2 API pci_enable_msix | ||
243 | 30 | ||
244 | int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec) | ||
245 | 31 | ||
246 | This API enables a device driver to request the PCI subsystem | 32 | 3. Why use MSIs? |
247 | to enable MSI-X messages on its hardware device. Depending on | 33 | |
248 | the availability of PCI vectors resources, the PCI subsystem enables | 34 | There are three reasons why using MSIs can give an advantage over |
249 | either all or none of the requested vectors. | 35 | traditional pin-based interrupts. |
36 | |||
37 | Pin-based PCI interrupts are often shared amongst several devices. | ||
38 | To support this, the kernel must call each interrupt handler associated | ||
39 | with an interrupt, which leads to reduced performance for the system as | ||
40 | a whole. MSIs are never shared, so this problem cannot arise. | ||
41 | |||
42 | When a device writes data to memory, then raises a pin-based interrupt, | ||
43 | it is possible that the interrupt may arrive before all the data has | ||
44 | arrived in memory (this becomes more likely with devices behind PCI-PCI | ||
45 | bridges). In order to ensure that all the data has arrived in memory, | ||
46 | the interrupt handler must read a register on the device which raised | ||
47 | the interrupt. PCI transaction ordering rules require that all the data | ||
48 | arrives in memory before the value can be returned from the register. | ||
49 | Using MSIs avoids this problem as the interrupt-generating write cannot | ||
50 | pass the data writes, so by the time the interrupt is raised, the driver | ||
51 | knows that all the data has arrived in memory. | ||
52 | |||
53 | PCI devices can only support a single pin-based interrupt per function. | ||
54 | Often drivers have to query the device to find out what event has | ||
55 | occurred, slowing down interrupt handling for the common case. With | ||
56 | MSIs, a device can support more interrupts, allowing each interrupt | ||
57 | to be specialised to a different purpose. One possible design gives | ||
58 | infrequent conditions (such as errors) their own interrupt which allows | ||
59 | the driver to handle the normal interrupt handling path more efficiently. | ||
60 | Other possible designs include giving one interrupt to each packet queue | ||
61 | in a network card or each port in a storage controller. | ||
62 | |||
63 | |||
64 | 4. How to use MSIs | ||
65 | |||
66 | PCI devices are initialised to use pin-based interrupts. The device | ||
67 | driver has to set up the device to use MSI or MSI-X. Not all machines | ||
68 | support MSIs correctly, and for those machines, the APIs described below | ||
69 | will simply fail and the device will continue to use pin-based interrupts. | ||
70 | |||
71 | 4.1 Include kernel support for MSIs | ||
72 | |||
73 | To support MSI or MSI-X, the kernel must be built with the CONFIG_PCI_MSI | ||
74 | option enabled. This option is only available on some architectures, | ||
75 | and it may depend on some other options also being set. For example, | ||
76 | on x86, you must also enable X86_UP_APIC or SMP in order to see the | ||
77 | CONFIG_PCI_MSI option. | ||
78 | |||
79 | 4.2 Using MSI | ||
80 | |||
81 | Most of the hard work is done for the driver in the PCI layer. It simply | ||
82 | has to request that the PCI layer set up the MSI capability for this | ||
83 | device. | ||
84 | |||
85 | 4.2.1 pci_enable_msi | ||
86 | |||
87 | int pci_enable_msi(struct pci_dev *dev) | ||
88 | |||
89 | A successful call will allocate ONE interrupt to the device, regardless | ||
90 | of how many MSIs the device supports. The device will be switched from | ||
91 | pin-based interrupt mode to MSI mode. The dev->irq number is changed | ||
92 | to a new number which represents the message signaled interrupt. | ||
93 | This function should be called before the driver calls request_irq() | ||
94 | since enabling MSIs disables the pin-based IRQ and the driver will not | ||
95 | receive interrupts on the old interrupt. | ||
96 | |||
97 | 4.2.2 pci_enable_msi_block | ||
98 | |||
99 | int pci_enable_msi_block(struct pci_dev *dev, int count) | ||
100 | |||
101 | This variation on the above call allows a device driver to request multiple | ||
102 | MSIs. The MSI specification only allows interrupts to be allocated in | ||
103 | powers of two, up to a maximum of 2^5 (32). | ||
104 | |||
105 | If this function returns 0, it has succeeded in allocating at least as many | ||
106 | interrupts as the driver requested (it may have allocated more in order | ||
107 | to satisfy the power-of-two requirement). In this case, the function | ||
108 | enables MSI on this device and updates dev->irq to be the lowest of | ||
109 | the new interrupts assigned to it. The other interrupts assigned to | ||
110 | the device are in the range dev->irq to dev->irq + count - 1. | ||
111 | |||
112 | If this function returns a negative number, it indicates an error and | ||
113 | the driver should not attempt to request any more MSI interrupts for | ||
114 | this device. If this function returns a positive number, it will be | ||
115 | less than 'count' and indicate the number of interrupts that could have | ||
116 | been allocated. In neither case will the irq value have been | ||
117 | updated, nor will the device have been switched into MSI mode. | ||
118 | |||
119 | The device driver must decide what action to take if | ||
120 | pci_enable_msi_block() returns a value less than the number asked for. | ||
121 | Some devices can make use of fewer interrupts than the maximum they | ||
122 | request; in this case the driver should call pci_enable_msi_block() | ||
123 | again. Note that it is not guaranteed to succeed, even when the | ||
124 | 'count' has been reduced to the value returned from a previous call to | ||
125 | pci_enable_msi_block(). This is because there are multiple constraints | ||
126 | on the number of vectors that can be allocated; pci_enable_msi_block() | ||
127 | will return as soon as it finds any constraint that doesn't allow the | ||
128 | call to succeed. | ||
129 | |||
130 | 4.2.3 pci_disable_msi | ||
131 | |||
132 | void pci_disable_msi(struct pci_dev *dev) | ||
250 | 133 | ||
251 | Argument 'dev' points to the device (pci_dev) structure. | 134 | This function should be used to undo the effect of pci_enable_msi() or |
135 | pci_enable_msi_block(). Calling it restores dev->irq to the pin-based | ||
136 | interrupt number and frees the previously allocated message signaled | ||
137 | interrupt(s). The interrupt may subsequently be assigned to another | ||
138 | device, so drivers should not cache the value of dev->irq. | ||
252 | 139 | ||
253 | Argument 'entries' is a pointer to an array of msix_entry structs. | 140 | A device driver must always call free_irq() on the interrupt(s) |
254 | The number of entries is indicated in argument 'nvec'. | 141 | for which it has called request_irq() before calling this function. |
255 | struct msix_entry is defined in /driver/pci/msi.h: | 142 | Failure to do so will result in a BUG_ON(), the device will be left with |
143 | MSI enabled and will leak its vector. | ||
144 | |||
145 | 4.3 Using MSI-X | ||
146 | |||
147 | The MSI-X capability is much more flexible than the MSI capability. | ||
148 | It supports up to 2048 interrupts, each of which can be controlled | ||
149 | independently. To support this flexibility, drivers must use an array of | ||
150 | `struct msix_entry': | ||
256 | 151 | ||
257 | struct msix_entry { | 152 | struct msix_entry { |
258 | u16 vector; /* kernel uses to write alloc vector */ | 153 | u16 vector; /* kernel uses to write alloc vector */ |
259 | u16 entry; /* driver uses to specify entry */ | 154 | u16 entry; /* driver uses to specify entry */ |
260 | }; | 155 | }; |
261 | 156 | ||
262 | A device driver is responsible for initializing the field 'entry' of | 157 | This allows for the device to use these interrupts in a sparse fashion; |
263 | each element with a unique entry supported by MSI-X table. Otherwise, | 158 | for example it could use interrupts 3 and 1027 and allocate only a |
264 | -EINVAL will be returned as a result. A successful return of zero | 159 | two-element array. The driver is expected to fill in the 'entry' value |
265 | indicates the PCI subsystem completed initializing each of the requested | 160 | in each element of the array to indicate which entries it wants the kernel |
266 | entries of the MSI-X table with message address and message data. | 161 | to assign interrupts for. It is invalid to fill in two entries with the |
267 | Last but not least, the PCI subsystem will write the 1:1 | 162 | same number. |
268 | vector-to-entry mapping into the field 'vector' of each element. A | 163 | |
269 | device driver is responsible for keeping track of allocated MSI-X | 164 | 4.3.1 pci_enable_msix |
270 | vectors in its internal data structure. | 165 | |
271 | 166 | int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec) | |
272 | A return of zero indicates that the number of MSI-X vectors was | 167 | |
273 | successfully allocated. A return of greater than zero indicates | 168 | Calling this function asks the PCI subsystem to allocate 'nvec' MSIs. |
274 | MSI-X vector shortage. Or a return of less than zero indicates | 169 | The 'entries' argument is a pointer to an array of msix_entry structs |
275 | a failure. This failure may be a result of duplicate entries | 170 | which should be at least 'nvec' entries in size. On success, the |
276 | specified in second argument, or a result of no available vector, | 171 | function will return 0 and the device will have been switched into |
277 | or a result of failing to initialize MSI-X table entries. | 172 | MSI-X interrupt mode. The 'vector' elements in each entry will have |
278 | 173 | been filled in with the interrupt number. The driver should then call | |
279 | 5.3.3 API pci_disable_msix | 174 | request_irq() for each 'vector' that it decides to use. |
175 | |||
176 | If this function returns a negative number, it indicates an error and | ||
177 | the driver should not attempt to allocate any more MSI-X interrupts for | ||
178 | this device. If it returns a positive number, it indicates the maximum | ||
179 | number of interrupt vectors that could have been allocated. See example | ||
180 | below. | ||
181 | |||
182 | This function, in contrast with pci_enable_msi(), does not adjust | ||
183 | dev->irq. The device will not generate interrupts for this interrupt | ||
184 | number once MSI-X is enabled. The device driver is responsible for | ||
185 | keeping track of the interrupts assigned to the MSI-X vectors so it can | ||
186 | free them again later. | ||
187 | |||
188 | Device drivers should normally call this function once per device | ||
189 | during the initialization phase. | ||
190 | |||
191 | It is ideal if drivers can cope with a variable number of MSI-X interrupts, | ||
192 | there are many reasons why the platform may not be able to provide the | ||
193 | exact number a driver asks for. | ||
194 | |||
195 | A request loop to achieve that might look like: | ||
196 | |||
197 | static int foo_driver_enable_msix(struct foo_adapter *adapter, int nvec) | ||
198 | { | ||
199 | while (nvec >= FOO_DRIVER_MINIMUM_NVEC) { | ||
200 | rc = pci_enable_msix(adapter->pdev, | ||
201 | adapter->msix_entries, nvec); | ||
202 | if (rc > 0) | ||
203 | nvec = rc; | ||
204 | else | ||
205 | return rc; | ||
206 | } | ||
207 | |||
208 | return -ENOSPC; | ||
209 | } | ||
210 | |||
211 | 4.3.2 pci_disable_msix | ||
280 | 212 | ||
281 | void pci_disable_msix(struct pci_dev *dev) | 213 | void pci_disable_msix(struct pci_dev *dev) |
282 | 214 | ||
283 | This API should always be used to undo the effect of pci_enable_msix() | 215 | This API should be used to undo the effect of pci_enable_msix(). It frees |
284 | when a device driver is unloading. Note that a device driver should | 216 | the previously allocated message signaled interrupts. The interrupts may |
285 | always call free_irq() on all MSI-X vectors it has done request_irq() | 217 | subsequently be assigned to another device, so drivers should not cache |
286 | on before calling this API. Failure to do so results in a BUG_ON() and | 218 | the value of the 'vector' elements over a call to pci_disable_msix(). |
287 | a device will be left with MSI-X enabled and leaks its vectors. | 219 | |
288 | 220 | A device driver must always call free_irq() on the interrupt(s) | |
289 | 5.3.4 MSI-X mode vs. legacy mode diagram | 221 | for which it has called request_irq() before calling this function. |
290 | 222 | Failure to do so will result in a BUG_ON(), the device will be left with | |
291 | The below diagram shows the events which switch the interrupt | 223 | MSI enabled and will leak its vector. |
292 | mode on the MSI-X capable device function between MSI-X mode and | 224 | |
293 | PIN-IRQ assertion mode (legacy). | 225 | 4.3.3 The MSI-X Table |
294 | 226 | ||
295 | ------------ pci_enable_msix(,,n) ------------------------ | 227 | The MSI-X capability specifies a BAR and offset within that BAR for the |
296 | | | <=============== | | | 228 | MSI-X Table. This address is mapped by the PCI subsystem, and should not |
297 | | MSI-X MODE | | PIN-IRQ ASSERTION MODE | | 229 | be accessed directly by the device driver. If the driver wishes to |
298 | | | ===============> | | | 230 | mask or unmask an interrupt, it should call disable_irq() / enable_irq(). |
299 | ------------ pci_disable_msix ------------------------ | 231 | |
300 | 232 | 4.4 Handling devices implementing both MSI and MSI-X capabilities | |
301 | Figure 2. MSI-X Mode vs. Legacy Mode | 233 | |
302 | 234 | If a device implements both MSI and MSI-X capabilities, it can | |
303 | In Figure 2, a device operates by default in legacy mode. A | 235 | run in either MSI mode or MSI-X mode but not both simultaneously. |
304 | successful MSI-X request (using pci_enable_msix()) switches a | 236 | This is a requirement of the PCI spec, and it is enforced by the |
305 | device's interrupt mode to MSI-X mode. A pre-assigned IOAPIC vector | 237 | PCI layer. Calling pci_enable_msi() when MSI-X is already enabled or |
306 | stored in dev->irq will be saved by the PCI subsystem; however, | 238 | pci_enable_msix() when MSI is already enabled will result in an error. |
307 | unlike MSI mode, the PCI subsystem will not replace dev->irq with | 239 | If a device driver wishes to switch between MSI and MSI-X at runtime, |
308 | assigned MSI-X vector because the PCI subsystem already writes the 1:1 | 240 | it must first quiesce the device, then switch it back to pin-interrupt |
309 | vector-to-entry mapping into the field 'vector' of each element | 241 | mode, before calling pci_enable_msi() or pci_enable_msix() and resuming |
310 | specified in second argument. | 242 | operation. This is not expected to be a common operation but may be |
311 | 243 | useful for debugging or testing during development. | |
312 | To return back to its default mode, a device driver should always call | 244 | |
313 | pci_disable_msix() to undo the effect of pci_enable_msix(). Note that | 245 | 4.5 Considerations when using MSIs |
314 | a device driver should always call free_irq() on all MSI-X vectors it | 246 | |
315 | has done request_irq() on before calling pci_disable_msix(). Failure | 247 | 4.5.1 Choosing between MSI-X and MSI |
316 | to do so results in a BUG_ON() and a device will be left with MSI-X | 248 | |
317 | enabled and leaks its vectors. Otherwise, the PCI subsystem switches a | 249 | If your device supports both MSI-X and MSI capabilities, you should use |
318 | device function's interrupt mode from MSI-X mode to legacy mode and | 250 | the MSI-X facilities in preference to the MSI facilities. As mentioned |
319 | marks all allocated MSI-X vectors as unused. | 251 | above, MSI-X supports any number of interrupts between 1 and 2048. |
320 | 252 | In constrast, MSI is restricted to a maximum of 32 interrupts (and | |
321 | Once being marked as unused, there is no guarantee that the PCI | 253 | must be a power of two). In addition, the MSI interrupt vectors must |
322 | subsystem will reserve these MSI-X vectors for a device. Depending on | 254 | be allocated consecutively, so the system may not be able to allocate |
323 | the availability of current PCI vector resources and the number of | 255 | as many vectors for MSI as it could for MSI-X. On some platforms, MSI |
324 | MSI/MSI-X requests from other drivers, these MSI-X vectors may be | 256 | interrupts must all be targetted at the same set of CPUs whereas MSI-X |
325 | re-assigned. | 257 | interrupts can all be targetted at different CPUs. |
326 | 258 | ||
327 | For the case where the PCI subsystem re-assigned these MSI-X vectors | 259 | 4.5.2 Spinlocks |
328 | to other drivers, a request to switch back to MSI-X mode may result | 260 | |
329 | being assigned with another set of MSI-X vectors or a failure if no | 261 | Most device drivers have a per-device spinlock which is taken in the |
330 | more vectors are available. | 262 | interrupt handler. With pin-based interrupts or a single MSI, it is not |
331 | 263 | necessary to disable interrupts (Linux guarantees the same interrupt will | |
332 | 5.4 Handling function implementing both MSI and MSI-X capabilities | 264 | not be re-entered). If a device uses multiple interrupts, the driver |
333 | 265 | must disable interrupts while the lock is held. If the device sends | |
334 | For the case where a function implements both MSI and MSI-X | 266 | a different interrupt, the driver will deadlock trying to recursively |
335 | capabilities, the PCI subsystem enables a device to run either in MSI | 267 | acquire the spinlock. |
336 | mode or MSI-X mode but not both. A device driver determines whether it | 268 | |
337 | wants MSI or MSI-X enabled on its hardware device. Once a device | 269 | There are two solutions. The first is to take the lock with |
338 | driver requests for MSI, for example, it is prohibited from requesting | 270 | spin_lock_irqsave() or spin_lock_irq() (see |
339 | MSI-X; in other words, a device driver is not permitted to ping-pong | 271 | Documentation/DocBook/kernel-locking). The second is to specify |
340 | between MSI mod MSI-X mode during a run-time. | 272 | IRQF_DISABLED to request_irq() so that the kernel runs the entire |
341 | 273 | interrupt routine with interrupts disabled. | |
342 | 5.5 Hardware requirements for MSI/MSI-X support | 274 | |
343 | 275 | If your MSI interrupt routine does not hold the lock for the whole time | |
344 | MSI/MSI-X support requires support from both system hardware and | 276 | it is running, the first solution may be best. The second solution is |
345 | individual hardware device functions. | 277 | normally preferred as it avoids making two transitions from interrupt |
346 | 278 | disabled to enabled and back again. | |
347 | 5.5.1 Required x86 hardware support | 279 | |
348 | 280 | 4.6 How to tell whether MSI/MSI-X is enabled on a device | |
349 | Since the target of MSI address is the local APIC CPU, enabling | 281 | |
350 | MSI/MSI-X support in the Linux kernel is dependent on whether existing | 282 | Using 'lspci -v' (as root) may show some devices with "MSI", "Message |
351 | system hardware supports local APIC. Users should verify that their | 283 | Signalled Interrupts" or "MSI-X" capabilities. Each of these capabilities |
352 | system supports local APIC operation by testing that it runs when | 284 | has an 'Enable' flag which will be followed with either "+" (enabled) |
353 | CONFIG_X86_LOCAL_APIC=y. | 285 | or "-" (disabled). |
354 | 286 | ||
355 | In SMP environment, CONFIG_X86_LOCAL_APIC is automatically set; | 287 | |
356 | however, in UP environment, users must manually set | 288 | 5. MSI quirks |
357 | CONFIG_X86_LOCAL_APIC. Once CONFIG_X86_LOCAL_APIC=y, setting | 289 | |
358 | CONFIG_PCI_MSI enables the VECTOR based scheme and the option for | 290 | Several PCI chipsets or devices are known not to support MSIs. |
359 | MSI-capable device drivers to selectively enable MSI/MSI-X. | 291 | The PCI stack provides three ways to disable MSIs: |
360 | 292 | ||
361 | Note that CONFIG_X86_IO_APIC setting is irrelevant because MSI/MSI-X | 293 | 1. globally |
362 | vector is allocated new during runtime and MSI/MSI-X support does not | 294 | 2. on all devices behind a specific bridge |
363 | depend on BIOS support. This key independency enables MSI/MSI-X | 295 | 3. on a single device |
364 | support on future IOxAPIC free platforms. | 296 | |
365 | 297 | 5.1. Disabling MSIs globally | |
366 | 5.5.2 Device hardware support | 298 | |
367 | 299 | Some host chipsets simply don't support MSIs properly. If we're | |
368 | The hardware device function supports MSI by indicating the | 300 | lucky, the manufacturer knows this and has indicated it in the ACPI |
369 | MSI/MSI-X capability structure on its PCI capability list. By | 301 | FADT table. In this case, Linux will automatically disable MSIs. |
370 | default, this capability structure will not be initialized by | 302 | Some boards don't include this information in the table and so we have |
371 | the kernel to enable MSI during the system boot. In other words, | 303 | to detect them ourselves. The complete list of these is found near the |
372 | the device function is running on its default pin assertion mode. | 304 | quirk_disable_all_msi() function in drivers/pci/quirks.c. |
373 | Note that in many cases the hardware supporting MSI have bugs, | 305 | |
374 | which may result in system hangs. The software driver of specific | 306 | If you have a board which has problems with MSIs, you can pass pci=nomsi |
375 | MSI-capable hardware is responsible for deciding whether to call | 307 | on the kernel command line to disable MSIs on all devices. It would be |
376 | pci_enable_msi or not. A return of zero indicates the kernel | 308 | in your best interests to report the problem to linux-pci@vger.kernel.org |
377 | successfully initialized the MSI/MSI-X capability structure of the | 309 | including a full 'lspci -v' so we can add the quirks to the kernel. |
378 | device function. The device function is now running on MSI/MSI-X mode. | 310 | |
379 | 311 | 5.2. Disabling MSIs below a bridge | |
380 | 5.6 How to tell whether MSI/MSI-X is enabled on device function | 312 | |
381 | 313 | Some PCI bridges are not able to route MSIs between busses properly. | |
382 | At the driver level, a return of zero from the function call of | 314 | In this case, MSIs must be disabled on all devices behind the bridge. |
383 | pci_enable_msi()/pci_enable_msix() indicates to a device driver that | 315 | |
384 | its device function is initialized successfully and ready to run in | 316 | Some bridges allow you to enable MSIs by changing some bits in their |
385 | MSI/MSI-X mode. | 317 | PCI configuration space (especially the Hypertransport chipsets such |
386 | 318 | as the nVidia nForce and Serverworks HT2000). As with host chipsets, | |
387 | At the user level, users can use the command 'cat /proc/interrupts' | 319 | Linux mostly knows about them and automatically enables MSIs if it can. |
388 | to display the vectors allocated for devices and their interrupt | 320 | If you have a bridge which Linux doesn't yet know about, you can enable |
389 | MSI/MSI-X modes ("PCI-MSI"/"PCI-MSI-X"). Below shows MSI mode is | 321 | MSIs in configuration space using whatever method you know works, then |
390 | enabled on a SCSI Adaptec 39320D Ultra320 controller. | 322 | enable MSIs on that bridge by doing: |
391 | 323 | ||
392 | CPU0 CPU1 | 324 | echo 1 > /sys/bus/pci/devices/$bridge/msi_bus |
393 | 0: 324639 0 IO-APIC-edge timer | 325 | |
394 | 1: 1186 0 IO-APIC-edge i8042 | 326 | where $bridge is the PCI address of the bridge you've enabled (eg |
395 | 2: 0 0 XT-PIC cascade | 327 | 0000:00:0e.0). |
396 | 12: 2797 0 IO-APIC-edge i8042 | 328 | |
397 | 14: 6543 0 IO-APIC-edge ide0 | 329 | To disable MSIs, echo 0 instead of 1. Changing this value should be |
398 | 15: 1 0 IO-APIC-edge ide1 | 330 | done with caution as it can break interrupt handling for all devices |
399 | 169: 0 0 IO-APIC-level uhci-hcd | 331 | below this bridge. |
400 | 185: 0 0 IO-APIC-level uhci-hcd | 332 | |
401 | 193: 138 10 PCI-MSI aic79xx | 333 | Again, please notify linux-pci@vger.kernel.org of any bridges that need |
402 | 201: 30 0 PCI-MSI aic79xx | 334 | special handling. |
403 | 225: 30 0 IO-APIC-level aic7xxx | 335 | |
404 | 233: 30 0 IO-APIC-level aic7xxx | 336 | 5.3. Disabling MSIs on a single device |
405 | NMI: 0 0 | 337 | |
406 | LOC: 324553 325068 | 338 | Some devices are known to have faulty MSI implementations. Usually this |
407 | ERR: 0 | 339 | is handled in the individual device driver but occasionally it's necessary |
408 | MIS: 0 | 340 | to handle this with a quirk. Some drivers have an option to disable use |
409 | 341 | of MSI. While this is a convenient workaround for the driver author, | |
410 | 6. MSI quirks | 342 | it is not good practise, and should not be emulated. |
411 | 343 | ||
412 | Several PCI chipsets or devices are known to not support MSI. | 344 | 5.4. Finding why MSIs are disabled on a device |
413 | The PCI stack provides 3 possible levels of MSI disabling: | 345 | |
414 | * on a single device | 346 | From the above three sections, you can see that there are many reasons |
415 | * on all devices behind a specific bridge | 347 | why MSIs may not be enabled for a given device. Your first step should |
416 | * globally | 348 | be to examine your dmesg carefully to determine whether MSIs are enabled |
417 | 349 | for your machine. You should also check your .config to be sure you | |
418 | 6.1. Disabling MSI on a single device | 350 | have enabled CONFIG_PCI_MSI. |
419 | 351 | ||
420 | Under some circumstances it might be required to disable MSI on a | 352 | Then, 'lspci -t' gives the list of bridges above a device. Reading |
421 | single device. This may be achieved by either not calling pci_enable_msi() | 353 | /sys/bus/pci/devices/*/msi_bus will tell you whether MSI are enabled (1) |
422 | or all, or setting the pci_dev->no_msi flag before (most of the time | 354 | or disabled (0). If 0 is found in any of the msi_bus files belonging |
423 | in a quirk). | 355 | to bridges between the PCI root and the device, MSIs are disabled. |
424 | 356 | ||
425 | 6.2. Disabling MSI below a bridge | 357 | It is also worth checking the device driver to see whether it supports MSIs. |
426 | 358 | For example, it may contain calls to pci_enable_msi(), pci_enable_msix() or | |
427 | The vast majority of MSI quirks are required by PCI bridges not | 359 | pci_enable_msi_block(). |
428 | being able to route MSI between busses. In this case, MSI have to be | ||
429 | disabled on all devices behind this bridge. It is achieves by setting | ||
430 | the PCI_BUS_FLAGS_NO_MSI flag in the pci_bus->bus_flags of the bridge | ||
431 | subordinate bus. There is no need to set the same flag on bridges that | ||
432 | are below the broken bridge. When pci_enable_msi() is called to enable | ||
433 | MSI on a device, pci_msi_supported() takes care of checking the NO_MSI | ||
434 | flag in all parent busses of the device. | ||
435 | |||
436 | Some bridges actually support dynamic MSI support enabling/disabling | ||
437 | by changing some bits in their PCI configuration space (especially | ||
438 | the Hypertransport chipsets such as the nVidia nForce and Serverworks | ||
439 | HT2000). It may then be required to update the NO_MSI flag on the | ||
440 | corresponding devices in the sysfs hierarchy. To enable MSI support | ||
441 | on device "0000:00:0e", do: | ||
442 | |||
443 | echo 1 > /sys/bus/pci/devices/0000:00:0e/msi_bus | ||
444 | |||
445 | To disable MSI support, echo 0 instead of 1. Note that it should be | ||
446 | used with caution since changing this value might break interrupts. | ||
447 | |||
448 | 6.3. Disabling MSI globally | ||
449 | |||
450 | Some extreme cases may require to disable MSI globally on the system. | ||
451 | For now, the only known case is a Serverworks PCI-X chipsets (MSI are | ||
452 | not supported on several busses that are not all connected to the | ||
453 | chipset in the Linux PCI hierarchy). In the vast majority of other | ||
454 | cases, disabling only behind a specific bridge is enough. | ||
455 | |||
456 | For debugging purpose, the user may also pass pci=nomsi on the kernel | ||
457 | command-line to explicitly disable MSI globally. But, once the appro- | ||
458 | priate quirks are added to the kernel, this option should not be | ||
459 | required anymore. | ||
460 | |||
461 | 6.4. Finding why MSI cannot be enabled on a device | ||
462 | |||
463 | Assuming that MSI are not enabled on a device, you should look at | ||
464 | dmesg to find messages that quirks may output when disabling MSI | ||
465 | on some devices, some bridges or even globally. | ||
466 | Then, lspci -t gives the list of bridges above a device. Reading | ||
467 | /sys/bus/pci/devices/0000:00:0e/msi_bus will tell you whether MSI | ||
468 | are enabled (1) or disabled (0). In 0 is found in a single bridge | ||
469 | msi_bus file above the device, MSI cannot be enabled. | ||
470 | |||
471 | 7. FAQ | ||
472 | |||
473 | Q1. Are there any limitations on using the MSI? | ||
474 | |||
475 | A1. If the PCI device supports MSI and conforms to the | ||
476 | specification and the platform supports the APIC local bus, | ||
477 | then using MSI should work. | ||
478 | |||
479 | Q2. Will it work on all the Pentium processors (P3, P4, Xeon, | ||
480 | AMD processors)? In P3 IPI's are transmitted on the APIC local | ||
481 | bus and in P4 and Xeon they are transmitted on the system | ||
482 | bus. Are there any implications with this? | ||
483 | |||
484 | A2. MSI support enables a PCI device sending an inbound | ||
485 | memory write (0xfeexxxxx as target address) on its PCI bus | ||
486 | directly to the FSB. Since the message address has a | ||
487 | redirection hint bit cleared, it should work. | ||
488 | |||
489 | Q3. The target address 0xfeexxxxx will be translated by the | ||
490 | Host Bridge into an interrupt message. Are there any | ||
491 | limitations on the chipsets such as Intel 8xx, Intel e7xxx, | ||
492 | or VIA? | ||
493 | |||
494 | A3. If these chipsets support an inbound memory write with | ||
495 | target address set as 0xfeexxxxx, as conformed to PCI | ||
496 | specification 2.3 or latest, then it should work. | ||
497 | |||
498 | Q4. From the driver point of view, if the MSI is lost because | ||
499 | of errors occurring during inbound memory write, then it may | ||
500 | wait forever. Is there a mechanism for it to recover? | ||
501 | |||
502 | A4. Since the target of the transaction is an inbound memory | ||
503 | write, all transaction termination conditions (Retry, | ||
504 | Master-Abort, Target-Abort, or normal completion) are | ||
505 | supported. A device sending an MSI must abide by all the PCI | ||
506 | rules and conditions regarding that inbound memory write. So, | ||
507 | if a retry is signaled it must retry, etc... We believe that | ||
508 | the recommendation for Abort is also a retry (refer to PCI | ||
509 | specification 2.3 or latest). | ||
diff --git a/Documentation/PCI/pci-iov-howto.txt b/Documentation/PCI/pci-iov-howto.txt new file mode 100644 index 000000000000..fc73ef5d65b8 --- /dev/null +++ b/Documentation/PCI/pci-iov-howto.txt | |||
@@ -0,0 +1,99 @@ | |||
1 | PCI Express I/O Virtualization Howto | ||
2 | Copyright (C) 2009 Intel Corporation | ||
3 | Yu Zhao <yu.zhao@intel.com> | ||
4 | |||
5 | |||
6 | 1. Overview | ||
7 | |||
8 | 1.1 What is SR-IOV | ||
9 | |||
10 | Single Root I/O Virtualization (SR-IOV) is a PCI Express Extended | ||
11 | capability which makes one physical device appear as multiple virtual | ||
12 | devices. The physical device is referred to as Physical Function (PF) | ||
13 | while the virtual devices are referred to as Virtual Functions (VF). | ||
14 | Allocation of the VF can be dynamically controlled by the PF via | ||
15 | registers encapsulated in the capability. By default, this feature is | ||
16 | not enabled and the PF behaves as traditional PCIe device. Once it's | ||
17 | turned on, each VF's PCI configuration space can be accessed by its own | ||
18 | Bus, Device and Function Number (Routing ID). And each VF also has PCI | ||
19 | Memory Space, which is used to map its register set. VF device driver | ||
20 | operates on the register set so it can be functional and appear as a | ||
21 | real existing PCI device. | ||
22 | |||
23 | 2. User Guide | ||
24 | |||
25 | 2.1 How can I enable SR-IOV capability | ||
26 | |||
27 | The device driver (PF driver) will control the enabling and disabling | ||
28 | of the capability via API provided by SR-IOV core. If the hardware | ||
29 | has SR-IOV capability, loading its PF driver would enable it and all | ||
30 | VFs associated with the PF. | ||
31 | |||
32 | 2.2 How can I use the Virtual Functions | ||
33 | |||
34 | The VF is treated as hot-plugged PCI devices in the kernel, so they | ||
35 | should be able to work in the same way as real PCI devices. The VF | ||
36 | requires device driver that is same as a normal PCI device's. | ||
37 | |||
38 | 3. Developer Guide | ||
39 | |||
40 | 3.1 SR-IOV API | ||
41 | |||
42 | To enable SR-IOV capability: | ||
43 | int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn); | ||
44 | 'nr_virtfn' is number of VFs to be enabled. | ||
45 | |||
46 | To disable SR-IOV capability: | ||
47 | void pci_disable_sriov(struct pci_dev *dev); | ||
48 | |||
49 | To notify SR-IOV core of Virtual Function Migration: | ||
50 | irqreturn_t pci_sriov_migration(struct pci_dev *dev); | ||
51 | |||
52 | 3.2 Usage example | ||
53 | |||
54 | Following piece of code illustrates the usage of the SR-IOV API. | ||
55 | |||
56 | static int __devinit dev_probe(struct pci_dev *dev, const struct pci_device_id *id) | ||
57 | { | ||
58 | pci_enable_sriov(dev, NR_VIRTFN); | ||
59 | |||
60 | ... | ||
61 | |||
62 | return 0; | ||
63 | } | ||
64 | |||
65 | static void __devexit dev_remove(struct pci_dev *dev) | ||
66 | { | ||
67 | pci_disable_sriov(dev); | ||
68 | |||
69 | ... | ||
70 | } | ||
71 | |||
72 | static int dev_suspend(struct pci_dev *dev, pm_message_t state) | ||
73 | { | ||
74 | ... | ||
75 | |||
76 | return 0; | ||
77 | } | ||
78 | |||
79 | static int dev_resume(struct pci_dev *dev) | ||
80 | { | ||
81 | ... | ||
82 | |||
83 | return 0; | ||
84 | } | ||
85 | |||
86 | static void dev_shutdown(struct pci_dev *dev) | ||
87 | { | ||
88 | ... | ||
89 | } | ||
90 | |||
91 | static struct pci_driver dev_driver = { | ||
92 | .name = "SR-IOV Physical Function driver", | ||
93 | .id_table = dev_id_table, | ||
94 | .probe = dev_probe, | ||
95 | .remove = __devexit_p(dev_remove), | ||
96 | .suspend = dev_suspend, | ||
97 | .resume = dev_resume, | ||
98 | .shutdown = dev_shutdown, | ||
99 | }; | ||
diff --git a/Documentation/feature-removal-schedule.txt b/Documentation/feature-removal-schedule.txt index ea7d1bdad34d..d0f354670646 100644 --- a/Documentation/feature-removal-schedule.txt +++ b/Documentation/feature-removal-schedule.txt | |||
@@ -392,3 +392,35 @@ Why: The defines and typedefs (hw_interrupt_type, no_irq_type, irq_desc_t) | |||
392 | have been kept around for migration reasons. After more than two years | 392 | have been kept around for migration reasons. After more than two years |
393 | it's time to remove them finally | 393 | it's time to remove them finally |
394 | Who: Thomas Gleixner <tglx@linutronix.de> | 394 | Who: Thomas Gleixner <tglx@linutronix.de> |
395 | |||
396 | --------------------------- | ||
397 | |||
398 | What: fakephp and associated sysfs files in /sys/bus/pci/slots/ | ||
399 | When: 2011 | ||
400 | Why: In 2.6.27, the semantics of /sys/bus/pci/slots was redefined to | ||
401 | represent a machine's physical PCI slots. The change in semantics | ||
402 | had userspace implications, as the hotplug core no longer allowed | ||
403 | drivers to create multiple sysfs files per physical slot (required | ||
404 | for multi-function devices, e.g.). fakephp was seen as a developer's | ||
405 | tool only, and its interface changed. Too late, we learned that | ||
406 | there were some users of the fakephp interface. | ||
407 | |||
408 | In 2.6.30, the original fakephp interface was restored. At the same | ||
409 | time, the PCI core gained the ability that fakephp provided, namely | ||
410 | function-level hot-remove and hot-add. | ||
411 | |||
412 | Since the PCI core now provides the same functionality, exposed in: | ||
413 | |||
414 | /sys/bus/pci/rescan | ||
415 | /sys/bus/pci/devices/.../remove | ||
416 | /sys/bus/pci/devices/.../rescan | ||
417 | |||
418 | there is no functional reason to maintain fakephp as well. | ||
419 | |||
420 | We will keep the existing module so that 'modprobe fakephp' will | ||
421 | present the old /sys/bus/pci/slots/... interface for compatibility, | ||
422 | but users are urged to migrate their applications to the API above. | ||
423 | |||
424 | After a reasonable transition period, we will remove the legacy | ||
425 | fakephp interface. | ||
426 | Who: Alex Chiang <achiang@hp.com> | ||
diff --git a/Documentation/filesystems/sysfs-pci.txt b/Documentation/filesystems/sysfs-pci.txt index 9f8740ca3f3b..26e4b8bc53ee 100644 --- a/Documentation/filesystems/sysfs-pci.txt +++ b/Documentation/filesystems/sysfs-pci.txt | |||
@@ -12,6 +12,7 @@ that support it. For example, a given bus might look like this: | |||
12 | | |-- enable | 12 | | |-- enable |
13 | | |-- irq | 13 | | |-- irq |
14 | | |-- local_cpus | 14 | | |-- local_cpus |
15 | | |-- remove | ||
15 | | |-- resource | 16 | | |-- resource |
16 | | |-- resource0 | 17 | | |-- resource0 |
17 | | |-- resource1 | 18 | | |-- resource1 |
@@ -36,6 +37,7 @@ files, each with their own function. | |||
36 | enable Whether the device is enabled (ascii, rw) | 37 | enable Whether the device is enabled (ascii, rw) |
37 | irq IRQ number (ascii, ro) | 38 | irq IRQ number (ascii, ro) |
38 | local_cpus nearby CPU mask (cpumask, ro) | 39 | local_cpus nearby CPU mask (cpumask, ro) |
40 | remove remove device from kernel's list (ascii, wo) | ||
39 | resource PCI resource host addresses (ascii, ro) | 41 | resource PCI resource host addresses (ascii, ro) |
40 | resource0..N PCI resource N, if present (binary, mmap) | 42 | resource0..N PCI resource N, if present (binary, mmap) |
41 | resource0_wc..N_wc PCI WC map resource N, if prefetchable (binary, mmap) | 43 | resource0_wc..N_wc PCI WC map resource N, if prefetchable (binary, mmap) |
@@ -46,6 +48,7 @@ files, each with their own function. | |||
46 | 48 | ||
47 | ro - read only file | 49 | ro - read only file |
48 | rw - file is readable and writable | 50 | rw - file is readable and writable |
51 | wo - write only file | ||
49 | mmap - file is mmapable | 52 | mmap - file is mmapable |
50 | ascii - file contains ascii text | 53 | ascii - file contains ascii text |
51 | binary - file contains binary data | 54 | binary - file contains binary data |
@@ -73,6 +76,13 @@ that the device must be enabled for a rom read to return data succesfully. | |||
73 | In the event a driver is not bound to the device, it can be enabled using the | 76 | In the event a driver is not bound to the device, it can be enabled using the |
74 | 'enable' file, documented above. | 77 | 'enable' file, documented above. |
75 | 78 | ||
79 | The 'remove' file is used to remove the PCI device, by writing a non-zero | ||
80 | integer to the file. This does not involve any kind of hot-plug functionality, | ||
81 | e.g. powering off the device. The device is removed from the kernel's list of | ||
82 | PCI devices, the sysfs directory for it is removed, and the device will be | ||
83 | removed from any drivers attached to it. Removal of PCI root buses is | ||
84 | disallowed. | ||
85 | |||
76 | Accessing legacy resources through sysfs | 86 | Accessing legacy resources through sysfs |
77 | ---------------------------------------- | 87 | ---------------------------------------- |
78 | 88 | ||
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index aeedb89a307a..240257dd4238 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt | |||
@@ -1695,6 +1695,8 @@ and is between 256 and 4096 characters. It is defined in the file | |||
1695 | See also Documentation/blockdev/paride.txt. | 1695 | See also Documentation/blockdev/paride.txt. |
1696 | 1696 | ||
1697 | pci=option[,option...] [PCI] various PCI subsystem options: | 1697 | pci=option[,option...] [PCI] various PCI subsystem options: |
1698 | earlydump [X86] dump PCI config space before the kernel | ||
1699 | changes anything | ||
1698 | off [X86] don't probe for the PCI bus | 1700 | off [X86] don't probe for the PCI bus |
1699 | bios [X86-32] force use of PCI BIOS, don't access | 1701 | bios [X86-32] force use of PCI BIOS, don't access |
1700 | the hardware directly. Use this if your machine | 1702 | the hardware directly. Use this if your machine |
@@ -1794,6 +1796,15 @@ and is between 256 and 4096 characters. It is defined in the file | |||
1794 | cbmemsize=nn[KMG] The fixed amount of bus space which is | 1796 | cbmemsize=nn[KMG] The fixed amount of bus space which is |
1795 | reserved for the CardBus bridge's memory | 1797 | reserved for the CardBus bridge's memory |
1796 | window. The default value is 64 megabytes. | 1798 | window. The default value is 64 megabytes. |
1799 | resource_alignment= | ||
1800 | Format: | ||
1801 | [<order of align>@][<domain>:]<bus>:<slot>.<func>[; ...] | ||
1802 | Specifies alignment and device to reassign | ||
1803 | aligned memory resources. | ||
1804 | If <order of align> is not specified, | ||
1805 | PAGE_SIZE is used as alignment. | ||
1806 | PCI-PCI bridge can be specified, if resource | ||
1807 | windows need to be expanded. | ||
1797 | 1808 | ||
1798 | pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power | 1809 | pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power |
1799 | Management. | 1810 | Management. |