1 files changed, 766 insertions, 0 deletions

diff --git a/Documentation/networking/NAPI_HOWTO.txt b/Documentation/networking/NAPI_HOWTO.txt
new file mode 100644
index 000000000000..54376e8249c1
--- /dev/null
+++ b/Documentation/networking/NAPI_HOWTO.txt
@@ -0,0 +1,766 @@
+HISTORY:
+February 16/2002 -- revision 0.2.1:
+COR typo corrected
+February 10/2002 -- revision 0.2:
+some spell checking ;->
+January 12/2002 -- revision 0.1
+This is still work in progress so may change.
+To keep up to date please watch this space.
+
+Introduction to NAPI
+====================
+
+NAPI is a proven (www.cyberus.ca/~hadi/usenix-paper.tgz) technique
+to improve network performance on Linux. For more details please
+read that paper.
+NAPI provides a "inherent mitigation" which is bound by system capacity
+as can be seen from the following data collected by Robert on Gigabit 
+ethernet (e1000):
+
+ Psize    Ipps       Tput     Rxint     Txint    Done     Ndone
+ ---------------------------------------------------------------
+   60    890000     409362        17     27622        7     6823
+  128    758150     464364        21      9301       10     7738
+  256    445632     774646        42     15507       21    12906
+  512    232666     994445    241292     19147   241192     1062
+ 1024    119061    1000003    872519     19258   872511        0
+ 1440     85193    1000003    946576     19505   946569        0
+ 
+
+Legend:
+"Ipps" stands for input packets per second. 
+"Tput" == packets out of total 1M that made it out.
+"txint" == transmit completion interrupts seen
+"Done" == The number of times that the poll() managed to pull all
+packets out of the rx ring. Note from this that the lower the
+load the more we could clean up the rxring
+"Ndone" == is the converse of "Done". Note again, that the higher
+the load the more times we couldnt clean up the rxring.
+
+Observe that:
+when the NIC receives 890Kpackets/sec only 17 rx interrupts are generated. 
+The system cant handle the processing at 1 interrupt/packet at that load level. 
+At lower rates on the other hand, rx interrupts go up and therefore the
+interrupt/packet ratio goes up (as observable from that table). So there is
+possibility that under low enough input, you get one poll call for each
+input packet caused by a single interrupt each time. And if the system 
+cant handle interrupt per packet ratio of 1, then it will just have to 
+chug along ....
+
+
+0) Prerequisites:
+==================
+A driver MAY continue using the old 2.4 technique for interfacing
+to the network stack and not benefit from the NAPI changes.
+NAPI additions to the kernel do not break backward compatibility.
+NAPI, however, requires the following features to be available:
+
+A) DMA ring or enough RAM to store packets in software devices.
+
+B) Ability to turn off interrupts or maybe events that send packets up 
+the stack.
+
+NAPI processes packet events in what is known as dev->poll() method.
+Typically, only packet receive events are processed in dev->poll(). 
+The rest of the events MAY be processed by the regular interrupt handler 
+to reduce processing latency (justified also because there are not that 
+many of them).
+Note, however, NAPI does not enforce that dev->poll() only processes 
+receive events. 
+Tests with the tulip driver indicated slightly increased latency if
+all of the interrupt handler is moved to dev->poll(). Also MII handling
+gets a little trickier.
+The example used in this document is to move the receive processing only
+to dev->poll(); this is shown with the patch for the tulip driver.
+For an example of code that moves all the interrupt driver to 
+dev->poll() look at the ported e1000 code.
+
+There are caveats that might force you to go with moving everything to 
+dev->poll(). Different NICs work differently depending on their status/event 
+acknowledgement setup. 
+There are two types of event register ACK mechanisms.
+        I)  what is known as Clear-on-read (COR).
+        when you read the status/event register, it clears everything!
+        The natsemi and sunbmac NICs are known to do this.
+        In this case your only choice is to move all to dev->poll()
+
+        II) Clear-on-write (COW)
+         i) you clear the status by writing a 1 in the bit-location you want.
+                These are the majority of the NICs and work the best with NAPI.
+                Put only receive events in dev->poll(); leave the rest in
+                the old interrupt handler.
+         ii) whatever you write in the status register clears every thing ;->
+                Cant seem to find any supported by Linux which do this. If
+                someone knows such a chip email us please.
+                Move all to dev->poll()
+
+C) Ability to detect new work correctly.
+NAPI works by shutting down event interrupts when theres work and
+turning them on when theres none. 
+New packets might show up in the small window while interrupts were being 
+re-enabled (refer to appendix 2).  A packet might sneak in during the period 
+we are enabling interrupts. We only get to know about such a packet when the 
+next new packet arrives and generates an interrupt. 
+Essentially, there is a small window of opportunity for a race condition
+which for clarity we'll refer to as the "rotting packet".
+
+This is a very important topic and appendix 2 is dedicated for more 
+discussion.
+
+Locking rules and environmental guarantees
+==========================================
+
+-Guarantee: Only one CPU at any time can call dev->poll(); this is because
+only one CPU can pick the initial interrupt and hence the initial
+netif_rx_schedule(dev);
+- The core layer invokes devices to send packets in a round robin format.
+This implies receive is totaly lockless because of the guarantee only that 
+one CPU is executing it.
+-  contention can only be the result of some other CPU accessing the rx
+ring. This happens only in close() and suspend() (when these methods
+try to clean the rx ring); 
+****guarantee: driver authors need not worry about this; synchronization 
+is taken care for them by the top net layer.
+-local interrupts are enabled (if you dont move all to dev->poll()). For 
+example link/MII and txcomplete continue functioning just same old way. 
+This improves the latency of processing these events. It is also assumed that 
+the receive interrupt is the largest cause of noise. Note this might not 
+always be true. 
+[according to Manfred Spraul, the winbond insists on sending one 
+txmitcomplete interrupt for each packet (although this can be mitigated)].
+For these broken drivers, move all to dev->poll().
+
+For the rest of this text, we'll assume that dev->poll() only
+processes receive events.
+
+new methods introduce by NAPI
+=============================
+
+a) netif_rx_schedule(dev)
+Called by an IRQ handler to schedule a poll for device
+
+b) netif_rx_schedule_prep(dev)
+puts the device in a state which allows for it to be added to the
+CPU polling list if it is up and running. You can look at this as
+the first half of  netif_rx_schedule(dev) above; the second half
+being c) below.
+
+c) __netif_rx_schedule(dev)
+Add device to the poll list for this CPU; assuming that _prep above
+has already been called and returned 1.
+
+d) netif_rx_reschedule(dev, undo)
+Called to reschedule polling for device specifically for some
+deficient hardware. Read Appendix 2 for more details.
+
+e) netif_rx_complete(dev)
+
+Remove interface from the CPU poll list: it must be in the poll list
+on current cpu. This primitive is called by dev->poll(), when
+it completes its work. The device cannot be out of poll list at this
+call, if it is then clearly it is a BUG(). You'll know ;->
+
+All these above nethods are used below. So keep reading for clarity.
+
+Device driver changes to be made when porting NAPI
+==================================================
+
+Below we describe what kind of changes are required for NAPI to work.
+
+1) introduction of dev->poll() method 
+=====================================
+
+This is the method that is invoked by the network core when it requests
+for new packets from the driver. A driver is allowed to send upto
+dev->quota packets by the current CPU before yielding to the network
+subsystem (so other devices can also get opportunity to send to the stack).
+
+dev->poll() prototype looks as follows:
+int my_poll(struct net_device *dev, int *budget)
+
+budget is the remaining number of packets the network subsystem on the
+current CPU can send up the stack before yielding to other system tasks.
+*Each driver is responsible for decrementing budget by the total number of
+packets sent.
+        Total number of packets cannot exceed dev->quota.
+
+dev->poll() method is invoked by the top layer, the driver just sends if it 
+can to the stack the packet quantity requested.
+
+more on dev->poll() below after the interrupt changes are explained.
+
+2) registering dev->poll() method
+===================================
+
+dev->poll should be set in the dev->probe() method. 
+e.g:
+dev->open = my_open;
+.
+.
+/* two new additions */
+/* first register my poll method */
+dev->poll = my_poll;
+/* next register my weight/quanta; can be overridden in /proc */
+dev->weight = 16;
+.
+.
+dev->stop = my_close;
+
+
+
+3) scheduling dev->poll()
+=============================
+This involves modifying the interrupt handler and the code
+path which takes the packet off the NIC and sends them to the 
+stack.
+
+it's important at this point to introduce the classical D Becker 
+interrupt processor:
+
+------------------
+static irqreturn_t
+netdevice_interrupt(int irq, void *dev_id, struct pt_regs *regs)
+{
+
+        struct net_device *dev = (struct net_device *)dev_instance;
+        struct my_private *tp = (struct my_private *)dev->priv;
+
+        int work_count = my_work_count;
+        status = read_interrupt_status_reg();
+        if (status == 0)
+                return IRQ_NONE; /* Shared IRQ: not us */
+        if (status == 0xffff)
+                return IRQ_HANDLED;      /* Hot unplug */
+        if (status & error)
+                do_some_error_handling()
+        
+        do {
+                acknowledge_ints_ASAP();
+
+                if (status & link_interrupt) {
+                        spin_lock(&tp->link_lock);
+                        do_some_link_stat_stuff();
+                        spin_lock(&tp->link_lock);
+                }
+                
+                if (status & rx_interrupt) {
+                        receive_packets(dev);
+                }
+
+                if (status & rx_nobufs) {
+                        make_rx_buffs_avail();
+                }
+                        
+                if (status & tx_related) {
+                        spin_lock(&tp->lock);
+                        tx_ring_free(dev);
+                        if (tx_died)
+                                restart_tx();
+                        spin_unlock(&tp->lock);
+                }
+
+                status = read_interrupt_status_reg();
+
+        } while (!(status & error) || more_work_to_be_done);
+        return IRQ_HANDLED;
+}
+
+----------------------------------------------------------------------
+
+We now change this to what is shown below to NAPI-enable it:
+
+----------------------------------------------------------------------
+static irqreturn_t
+netdevice_interrupt(int irq, void *dev_id, struct pt_regs *regs)
+{
+        struct net_device *dev = (struct net_device *)dev_instance;
+        struct my_private *tp = (struct my_private *)dev->priv;
+
+        status = read_interrupt_status_reg();
+        if (status == 0)
+                return IRQ_NONE;         /* Shared IRQ: not us */
+        if (status == 0xffff)
+                return IRQ_HANDLED;         /* Hot unplug */
+        if (status & error)
+                do_some_error_handling();
+        
+        do {
+/************************ start note *********************************/         
+                acknowledge_ints_ASAP();  // dont ack rx and rxnobuff here
+/************************ end note *********************************/           
+
+                if (status & link_interrupt) {
+                        spin_lock(&tp->link_lock);
+                        do_some_link_stat_stuff();
+                        spin_unlock(&tp->link_lock);
+                }
+/************************ start note *********************************/         
+                if (status & rx_interrupt || (status & rx_nobuffs)) {
+                        if (netif_rx_schedule_prep(dev)) {
+
+                                /* disable interrupts caused 
+                                 *      by arriving packets */
+                                disable_rx_and_rxnobuff_ints();
+                                /* tell system we have work to be done. */
+                                __netif_rx_schedule(dev);
+                        } else {
+                                printk("driver bug! interrupt while in poll\n");
+                                /* FIX by disabling interrupts  */
+                                disable_rx_and_rxnobuff_ints();
+                        }
+                }
+/************************ end note note *********************************/              
+                        
+                if (status & tx_related) {
+                        spin_lock(&tp->lock);
+                        tx_ring_free(dev);
+
+                        if (tx_died)
+                                restart_tx();
+                        spin_unlock(&tp->lock);
+                }
+
+                status = read_interrupt_status_reg();
+
+/************************ start note *********************************/         
+        } while (!(status & error) || more_work_to_be_done(status));
+/************************ end note note *********************************/              
+        return IRQ_HANDLED;
+}
+
+---------------------------------------------------------------------
+
+
+We note several things from above:
+
+I) Any interrupt source which is caused by arriving packets is now
+turned off when it occurs. Depending on the hardware, there could be
+several reasons that arriving packets would cause interrupts; these are the
+interrupt sources we wish to avoid. The two common ones are a) a packet 
+arriving (rxint) b) a packet arriving and finding no DMA buffers available
+(rxnobuff) .
+This means also acknowledge_ints_ASAP() will not clear the status
+register for those two items above; clearing is done in the place where 
+proper work is done within NAPI; at the poll() and refill_rx_ring() 
+discussed further below.
+netif_rx_schedule_prep() returns 1 if device is in running state and
+gets successfully added to the core poll list. If we get a zero value
+we can _almost_ assume are already added to the list (instead of not running. 
+Logic based on the fact that you shouldn't get interrupt if not running)
+We rectify this by disabling rx and rxnobuf interrupts.
+
+II) that receive_packets(dev) and make_rx_buffs_avail() may have disappeared.
+These functionalities are still around actually......
+
+infact, receive_packets(dev) is very close to my_poll() and 
+make_rx_buffs_avail() is invoked from my_poll()
+
+4) converting receive_packets() to dev->poll()
+===============================================
+
+We need to convert the classical D Becker receive_packets(dev) to my_poll()
+
+First the typical receive_packets() below:
+-------------------------------------------------------------------
+
+/* this is called by interrupt handler */
+static void receive_packets (struct net_device *dev)
+{
+
+        struct my_private *tp = (struct my_private *)dev->priv;
+        rx_ring = tp->rx_ring;
+        cur_rx = tp->cur_rx;
+        int entry = cur_rx % RX_RING_SIZE;
+        int received = 0;
+        int rx_work_limit = tp->dirty_rx + RX_RING_SIZE - tp->cur_rx;
+
+        while (rx_ring_not_empty) {
+                u32 rx_status;
+                unsigned int rx_size;
+                unsigned int pkt_size;
+                struct sk_buff *skb;
+                /* read size+status of next frame from DMA ring buffer */
+                /* the number 16 and 4 are just examples */
+                rx_status = le32_to_cpu (*(u32 *) (rx_ring + ring_offset));
+                rx_size = rx_status >> 16;
+                pkt_size = rx_size - 4;
+
+                /* process errors */
+                if ((rx_size > (MAX_ETH_FRAME_SIZE+4)) ||
+                    (!(rx_status & RxStatusOK))) {
+                        netdrv_rx_err (rx_status, dev, tp, ioaddr);
+                        return;
+                }
+
+                if (--rx_work_limit < 0)
+                        break;
+
+                /* grab a skb */
+                skb = dev_alloc_skb (pkt_size + 2);
+                if (skb) {
+                        .
+                        .
+                        netif_rx (skb);
+                        .
+                        .
+                } else {  /* OOM */
+                        /*seems very driver specific ... some just pass
+                        whatever is on the ring already. */
+                }
+
+                /* move to the next skb on the ring */
+                entry = (++tp->cur_rx) % RX_RING_SIZE;
+                received++ ;
+
+        }
+
+        /* store current ring pointer state */
+        tp->cur_rx = cur_rx;
+
+        /* Refill the Rx ring buffers if they are needed */
+        refill_rx_ring();
+        .
+        .
+
+}
+-------------------------------------------------------------------
+We change it to a new one below; note the additional parameter in
+the call.
+
+-------------------------------------------------------------------
+
+/* this is called by the network core */
+static int my_poll (struct net_device *dev, int *budget)
+{
+
+        struct my_private *tp = (struct my_private *)dev->priv;
+        rx_ring = tp->rx_ring;
+        cur_rx = tp->cur_rx;
+        int entry = cur_rx % RX_BUF_LEN;
+        /* maximum packets to send to the stack */
+/************************ note note *********************************/          
+        int rx_work_limit = dev->quota;
+
+/************************ end note note *********************************/              
+    do {  // outer beginning loop starts here
+
+        clear_rx_status_register_bit();
+
+        while (rx_ring_not_empty) {
+                u32 rx_status;
+                unsigned int rx_size;
+                unsigned int pkt_size;
+                struct sk_buff *skb;
+                /* read size+status of next frame from DMA ring buffer */
+                /* the number 16 and 4 are just examples */
+                rx_status = le32_to_cpu (*(u32 *) (rx_ring + ring_offset));
+                rx_size = rx_status >> 16;
+                pkt_size = rx_size - 4;
+
+                /* process errors */
+                if ((rx_size > (MAX_ETH_FRAME_SIZE+4)) ||
+                    (!(rx_status & RxStatusOK))) {
+                        netdrv_rx_err (rx_status, dev, tp, ioaddr);
+                        return 1;
+                }
+
+/************************ note note *********************************/          
+                if (--rx_work_limit < 0) { /* we got packets, but no quota */
+                        /* store current ring pointer state */
+                        tp->cur_rx = cur_rx;
+
+                        /* Refill the Rx ring buffers if they are needed */
+                        refill_rx_ring(dev);
+                        goto not_done;
+                }
+/**********************  end note **********************************/
+
+                /* grab a skb */
+                skb = dev_alloc_skb (pkt_size + 2);
+                if (skb) {
+                        .
+                        .
+/************************ note note *********************************/          
+                        netif_receive_skb (skb);
+/**********************  end note **********************************/
+                        .
+                        .
+                } else {  /* OOM */
+                        /*seems very driver specific ... common is just pass
+                        whatever is on the ring already. */
+                }
+
+                /* move to the next skb on the ring */
+                entry = (++tp->cur_rx) % RX_RING_SIZE;
+                received++ ;
+
+        }
+
+        /* store current ring pointer state */
+        tp->cur_rx = cur_rx;
+
+        /* Refill the Rx ring buffers if they are needed */
+        refill_rx_ring(dev);
+        
+        /* no packets on ring; but new ones can arrive since we last 
+           checked  */
+        status = read_interrupt_status_reg();
+        if (rx status is not set) {
+                        /* If something arrives in this narrow window,
+                        an interrupt will be generated */
+                        goto done;
+        }
+        /* done! at least thats what it looks like ;->
+        if new packets came in after our last check on status bits
+        they'll be caught by the while check and we go back and clear them 
+        since we havent exceeded our quota */
+    } while (rx_status_is_set); 
+
+done:
+
+/************************ note note *********************************/          
+        dev->quota -= received;
+        *budget -= received;
+
+        /* If RX ring is not full we are out of memory. */
+        if (tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL)
+                goto oom;
+
+        /* we are happy/done, no more packets on ring; put us back
+        to where we can start processing interrupts again */
+        netif_rx_complete(dev);
+        enable_rx_and_rxnobuf_ints();
+
+       /* The last op happens after poll completion. Which means the following:
+        * 1. it can race with disabling irqs in irq handler (which are done to 
+        * schedule polls)
+        * 2. it can race with dis/enabling irqs in other poll threads
+        * 3. if an irq raised after the begining of the outer  beginning 
+        * loop(marked in the code above), it will be immediately
+        * triggered here.
+        *
+        * Summarizing: the logic may results in some redundant irqs both
+        * due to races in masking and due to too late acking of already
+        * processed irqs. The good news: no events are ever lost.
+        */
+
+        return 0;   /* done */
+
+not_done:
+        if (tp->cur_rx - tp->dirty_rx > RX_RING_SIZE/2 ||
+            tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL)
+                refill_rx_ring(dev);
+
+        if (!received) {
+                printk("received==0\n");
+                received = 1;
+        }
+        dev->quota -= received;
+        *budget -= received;
+        return 1;  /* not_done */
+
+oom:
+        /* Start timer, stop polling, but do not enable rx interrupts. */
+        start_poll_timer(dev);
+        return 0;  /* we'll take it from here so tell core "done"*/
+
+/************************ End note note *********************************/              
+}
+-------------------------------------------------------------------
+
+From above we note that:
+0) rx_work_limit = dev->quota 
+1) refill_rx_ring() is in charge of clearing the bit for rxnobuff when
+it does the work.
+2) We have a done and not_done state.
+3) instead of netif_rx() we call netif_receive_skb() to pass the skb.
+4) we have a new way of handling oom condition
+5) A new outer for (;;) loop has been added. This serves the purpose of
+ensuring that if a new packet has come in, after we are all set and done,
+and we have not exceeded our quota that we continue sending packets up.
+ 
+
+-----------------------------------------------------------
+Poll timer code will need to do the following:
+
+a) 
+
+        if (tp->cur_rx - tp->dirty_rx > RX_RING_SIZE/2 ||
+            tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL) 
+                refill_rx_ring(dev);
+
+        /* If RX ring is not full we are still out of memory.
+           Restart the timer again. Else we re-add ourselves 
+           to the master poll list.
+         */
+
+        if (tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL)
+                restart_timer();
+
+        else netif_rx_schedule(dev);  /* we are back on the poll list */
+        
+5) dev->close() and dev->suspend() issues
+==========================================
+The driver writter neednt worry about this. The top net layer takes
+care of it.
+
+6) Adding new Stats to /proc 
+=============================
+In order to debug some of the new features, we introduce new stats
+that need to be collected.
+TODO: Fill this later.
+
+APPENDIX 1: discussion on using ethernet HW FC
+==============================================
+Most chips with FC only send a pause packet when they run out of Rx buffers.
+Since packets are pulled off the DMA ring by a softirq in NAPI,
+if the system is slow in grabbing them and we have a high input
+rate (faster than the system's capacity to remove packets), then theoretically
+there will only be one rx interrupt for all packets during a given packetstorm.
+Under low load, we might have a single interrupt per packet.
+FC should be programmed to apply in the case when the system cant pull out
+packets fast enough i.e send a pause only when you run out of rx buffers.
+Note FC in itself is a good solution but we have found it to not be
+much of a commodity feature (both in NICs and switches) and hence falls
+under the same category as using NIC based mitigation. Also experiments
+indicate that its much harder to resolve the resource allocation
+issue (aka lazy receiving that NAPI offers) and hence quantify its usefullness
+proved harder. In any case, FC works even better with NAPI but is not
+necessary.
+
+
+APPENDIX 2: the "rotting packet" race-window avoidance scheme 
+=============================================================
+
+There are two types of associations seen here
+
+1) status/int which honors level triggered IRQ
+
+If a status bit for receive or rxnobuff is set and the corresponding 
+interrupt-enable bit is not on, then no interrupts will be generated. However, 
+as soon as the "interrupt-enable" bit is unmasked, an immediate interrupt is 
+generated.  [assuming the status bit was not turned off].
+Generally the concept of level triggered IRQs in association with a status and
+interrupt-enable CSR register set is used to avoid the race.
+
+If we take the example of the tulip:
+"pending work" is indicated by the status bit(CSR5 in tulip).
+the corresponding interrupt bit (CSR7 in tulip) might be turned off (but
+the CSR5 will continue to be turned on with new packet arrivals even if
+we clear it the first time)
+Very important is the fact that if we turn on the interrupt bit on when
+status is set that an immediate irq is triggered.
+ 
+If we cleared the rx ring and proclaimed there was "no more work
+to be done" and then went on to do a few other things;  then when we enable
+interrupts, there is a possibility that a new packet might sneak in during
+this phase. It helps to look at the pseudo code for the tulip poll
+routine:
+
+--------------------------
+        do {
+                ACK;
+                while (ring_is_not_empty()) {
+                        work-work-work
+                        if quota is exceeded: exit, no touching irq status/mask
+                }
+                /* No packets, but new can arrive while we are doing this*/
+                CSR5 := read
+                if (CSR5 is not set) {
+                        /* If something arrives in this narrow window here,
+                        *  where the comments are ;-> irq will be generated */
+                        unmask irqs;
+                        exit poll;
+                }
+        } while (rx_status_is_set);
+------------------------
+
+CSR5 bit of interest is only the rx status. 
+If you look at the last if statement: 
+you just finished grabbing all the packets from the rx ring .. you check if
+status bit says theres more packets just in ... it says none; you then
+enable rx interrupts again; if a new packet just came in during this check,
+we are counting that CSR5 will be set in that small window of opportunity
+and that by re-enabling interrupts, we would actually triger an interrupt
+to register the new packet for processing.
+
+[The above description nay be very verbose, if you have better wording 
+that will make this more understandable, please suggest it.]
+
+2) non-capable hardware
+
+These do not generally respect level triggered IRQs. Normally,
+irqs may be lost while being masked and the only way to leave poll is to do
+a double check for new input after netif_rx_complete() is invoked
+and re-enable polling (after seeing this new input).
+
+Sample code:
+
+---------
+        .
+        .
+restart_poll:
+        while (ring_is_not_empty()) {
+                work-work-work
+                if quota is exceeded: exit, not touching irq status/mask
+        }
+        .
+        .
+        .
+        enable_rx_interrupts()
+        netif_rx_complete(dev);
+        if (ring_has_new_packet() && netif_rx_reschedule(dev, received)) {
+                disable_rx_and_rxnobufs()
+                goto restart_poll
+        } while (rx_status_is_set);
+---------
+                
+Basically netif_rx_complete() removes us from the poll list, but because a
+new packet which will never be caught due to the possibility of a race
+might come in, we attempt to re-add ourselves to the poll list. 
+
+
+
+
+APPENDIX 3: Scheduling issues.
+==============================
+As seen NAPI moves processing to softirq level. Linux uses the ksoftirqd as the 
+general solution to schedule softirq's to run before next interrupt and by putting 
+them under scheduler control. Also this prevents consecutive softirq's from 
+monopolize the CPU. This also have the effect that the priority of ksoftirq needs 
+to be considered when running very CPU-intensive applications and networking to
+get the proper balance of softirq/user balance. Increasing ksoftirq priority to 0 
+(eventually more) is reported cure problems with low network performance at high 
+CPU load.
+
+Most used processes in a GIGE router:
+USER       PID %CPU %MEM  SIZE   RSS TTY STAT START   TIME COMMAND
+root         3  0.2  0.0     0     0  ?  RWN Aug 15 602:00 (ksoftirqd_CPU0)
+root       232  0.0  7.9 41400 40884  ?  S   Aug 15  74:12 gated 
+
+--------------------------------------------------------------------
+
+relevant sites:
+==================
+ftp://robur.slu.se/pub/Linux/net-development/NAPI/
+
+
+--------------------------------------------------------------------
+TODO: Write net-skeleton.c driver.
+-------------------------------------------------------------
+
+Authors:
+========
+Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
+Jamal Hadi Salim <hadi@cyberus.ca>
+Robert Olsson <Robert.Olsson@data.slu.se>
+
+Acknowledgements:
+================
+People who made this document better:
+
+Lennert Buytenhek <buytenh@gnu.org>
+Andrew Morton  <akpm@zip.com.au>
+Manfred Spraul <manfred@colorfullife.com>
+Donald Becker <becker@scyld.com>
+Jeff Garzik <jgarzik@pobox.com>