/drivers/ssb/

p-d10-hz1000'>wip-d10-hz1000 The LITMUS^RT kernel.Bjoern Brandenburg
aboutsummaryrefslogblamecommitdiffstats
path: root/net/ipv4/fib_trie.c
blob: 9ac481a10d37dbff04910af538ab0a624fd2808c (plain) (tree)
1
2
3
4
5
6
7
8
9








                                                                  
                                                            
                             
  

                                                         
                                                                       
  

                                                                      
                                                     




                                                                               
















                                                                             






                                                       

   
                       
 

                        
                         

                         






                          
                             


                            
                           



                          
                       
                              







                         
                         
 
                                   
 




                             

                                                         

                                           
 
                     
                             
                  


             
                             
                  
                               
                            



                                
                            




                              
                             
                  

                                                                         

                                                                   


                                        
                                         
          
                                            








                                           
                                         








                                  
                              
                                               
  

             
                                        


                                    

  

                                                                                       
                                               
                                                                     

                                                               

                                                      







                                                                                
 
                                                      
                                                       
 



                                                                        
 




                                                                                   

 



                                                                            
 



                                                                                   
 
                                                          

 


                                                    
                                                                                
 

                                                            
 
 



                                                                                          

                                    
 
                                              

 



                                                                                              
 
                                    
 
                                                  

 
                                                            
 
                             

 
                                                     



                                                                  
                                                                                      
 
                               
                                                                    
            




                                               
                      



                                                                         

                                             

                                                              
 





                                                             


                                                 



                    
  

                                                                          



                                              




                                                                             

                   




                                                                             

                                             
                                                                         

                               
          


                                                                   
                                                                 








                                                                   
             
 

                                                                           


                                                                            
                                                                         




                                                                             
                                                                          

                                                              
                                                                             
                                                                  
 
 




                                                                              
                                                      
 
                                             

 

                                        
                                           
                                             

                                                   
 

                                                                         

 
                                                           
 

                                             
 

                                                  

                                                              
 
 




                                              
                                                       
 
                                                         

 
                                                         
 
                                                   

 
                                             
 
                              
                                                 
            
                                     
 
 



                                                                 

 
                                                   
 
                                                                 
                                            
                                                                  


                              



                                                    

 

                                               


                                              
                                                     

 


                                             

                                         
                                                 
                                                                       










                                                 




                                                        

 

                                  
                                                                      
















                                                                              
                                                            
 
                                                                                   
                                           
 
                 
                                     





                                             
 
                                                               
                                                      


                  




                                                                   
                                                                                  
 
                                    




                                                               
                                                                     
                                                    



                                            
   



                                                         
                                                                                  
                                              
 
                                                                  

                   

                                 




                                          
 
                                 
                          


                                              
                               
                                    
                                    
                                    
 
              
                                       
 
                                            

 
                   
                                                                    

              
                             

                                  
                     
 
                

                            

                                                                          


                                                           
                                    



                                                             
                               
          




                                                               

                                                                             
                                      
                                                                       

                                                              




                                                                          
                                             



                                                                            
                                                                    
                                                                           
                                                  
          
                                                
          
                                             
                                                                            



                                                         
                                                                          

                                                    


                                                                


                                                                            
          


                                                         
          
                                                                   
                                                               
                                                                        
          
                                   
                                                               
                                   
                                                              
          
                         
                                                             
                                                        
                                    
          


                        
 

                                    
                                                      

                                                               
                
                                                          

                                                      
 

                                                       


                                                                     
 

                                    
 

                                    




                                                       



                        
                                                   
                                 
                                                  
 



                                                                
 

                                             
                                                                    
                                                              
 



                                    





                                                       
 
 
                                    

                                                               
                                                              
                                               
 
                                                           
                               
                                         


                                                
                                                 
                                            
                                 
                 
         
                                          

 













                                                                        
                                                              
 



                                          
                                 


                                                                         
                
                                        

          


                                                                       

                               

                                    
                                    
 
                                                                      




                                                                   
                                                                       
 

                                                                         

                                           
 


                                                                       
                                     

                                                 
                         
 

                                                                               


                 
                                    
                                    
                                                                         

                                           
 





                                                                  
                                                                   
                                            


                                                                             









                                                              

                                                                                   
 
                                               
                                 

                 

















                                                                      
 


                                                               
 

                                                                 
 
                              
 

                                                                    
 
                               
 

                                                

                                                                                         
                 


                                                          
                                       
         
                                  
                  
      

                                

 
                                                            

                                    
                                          


                                          
                               

                                                                         
 

                                        

          


                                                                    


                               
                                       

                                                       
 
                                           
                                    
                                           
 
                                                                           

                                  
                                           
 
                                                                           
                 
 
         
 


                                         

                                                       
 




                                                                  
                                 
                 

                                    
                                                    

                                 
 
                                           

                                                                       


                                                             
         
                                  
                  
      

                                

 
                                                          

                            
                                                                 
 
                                           


                                
                                                       
                                     
                                  
 


                    
                                                                     
 
                                                       
 

                            
 
                         



                                                                            
















                                                                       

 

                                                            




                                      
                               

                
                                          


                                                       
 
                                
 
                                                                      
                                                 



                                                                             
                      



                                                         


                                                                


                    
                                                            
 
                    
                          
                         
 

                      
                                                                                     

                                                                      


                                                                    
                                                                          
 
                                                             
                        
                                                                               
 
                                   
                        
                              
                        
         
 
                                     
                         
                                                                   
 
                                                               
                           

 

                                 
                                                                           


                                            
                               

                       
                                         



                             
                                      
 

                                                                             
                                                       

                                                                           

                                            
                                                                          



                                                                      


                                                                           




                                                               
 
                                
 
                                                                      
                                
                                                 



                                                                         
 
                                                          
                      





                                       
                                                 

           
                                  


                                                   
                                                                  
                                      
                                         
 

                                    




                                               

                       

                            



                                 
                  
                             
                            
         



                                       
                                   
                                                                        
 
                                                              
 
                                                                   
                                                                                   

                                                                               

                                        



                                                          
                                               
                    

                                
                        

                                                                 
                        
                                   
                                                                         
                 
 
                          
                                           
                                     
                                    

                 
                                                               
 
                                                            
                                                                    

                                               
                         
                                                                           
                                                                
                                                             
                        
                                                                               


                                

                                          


                                                                       
 
                                
 
                              
     


                       


                         
                                                                  


                                                     
                                         
                            

                                   






                               
                                 
 
                                                                    
 
                                           
 
                        



                               


                                  
                         
         

                                  
                  
 
                














                                                                        


                                                            

                              
                                                 

                                 













                                                                              





                                                
                                                      


                                                 



                                                   
                                         
                         
                                       
                                                                             

                                           

                                              

                                                    
                                                       
                                             
                                                                  
 

                                                                         


                                                  
                                                                          

                                                                           
 
                                       




                                                         

                                 
 
                                                      
                                      

                      
                                              


                         
                                                             




                             
                                       
                             



                                        
                       


                                                        
                                             
                 
         
 


                                     

                                                         
 
                                                  
                                                                    
                                      

                 


                                               

                             
    


                   
                                         
                                                                           
                                                           
                                                            
 


                                            
 
                                                          
                                     


                                                   
                                                  

                                 


                                                                 
 
                                                                        
                                         
                                                                       



                                                           
                               
                                                                 
      
                                           







                                                                             
                                                                                     







                                                                 
                                                                    










                                                                          
      
         
 
                 

 
                                                                    
                                                           

                                                     
                
                               
                         
                               
                                      
                                 
                         
                                                       
                         
                            
 
                        
 
                                     
               







                               
                                                                                    
                           
         
 

                                
 
                    


                                
                                 

                                                                                        
 
                                                    







                                                 
                                 
                                                                                            
                                    
                                               
                                   

                 
                                       
 














                                                                           
 








                                                                            
 

                                                        
 

                                                                             

                                                                                

                                               
 








                                                                            
 







                                                                            
 










                                                                     

                   
                                                                 
 



                                                                      

                                    
                                                                
 
                                                                              



                                                             
                 
 



                                                     



                                                                 

                                                           
                                      


                                                                             

                                                                  
 
                  
                                                                           


                                                                            
                                              
                                                          
                        
                                                                                           
                                    
                                            
 
                                               

                                                                                       





                                             
                 

         
                
      
                          


                   



                                                            
 
                                                                  
 
                                                       
 
                 
                                                                            
                                                               
                                      
              
                                                  
 
                     

 


                         
                                                                  


                                                     

                                   


                                            

                             
                      

                               
                                 
                                           
 
                        




                                  
               







                                             
                                                                      

                            

                                                                     




                                                  

                                                                     
                                                                

                                                           


                                                             




                                          

                              
 
                          
                                                                
                                      

                                  
                                     
 
                                   
 


                                     
                                  
                                          
                                   
         
 
                                  
                                       
 
                                         
                                                          
 

                                      
                 

 
                                                  





                                                              
 



                                                          





                                
                                          






                                                              
                                                    

                                            
                                                  





                                           



                                                              
                                                                          
 

                          
 
                      
                                                                             
                    
                                
 

                                                          
                               

                                         
                                         
                                                                              
                                                         
                         



                                                               
                 

                                                        
                                              
                                                   



                                       

                                                  
                                                                        











                                                          
                                                           
                                             




                                                             

 



                                                             
                                
                                     
 



                 


                         
                                         

                                                     
                                   
                      
 
                                                              
                                            

                                                 
                                                



                                         
                                        
 
                                                 


                     




                                         

                                                                      



                                                                            
                                 
 
                          

              


                                                   



                                 





                                                               
                                       

                                             
                                                                  
                                        
                                  
                 

                    
                        


                        

                                                                         
 


                                
 
                          
              
 



                                                             
                                 
                 
 
                            
                                        
 
                                          

                                 
                                                                                    
                                        

                                  
                    
         
 
                        


                        

                                                             
 
                       
                                                     
                                
                                
 
                        


                                                     
                       

                                      


                                                                           
                                          

                                                     
         
 

                                     
                                                            
                                            
                                          
                                  
                 
 
                        
                                     

                                                                 
         
                            
                          
 
                        

 
                               
 

                                                                   





                                                                         
 
 
 
                                        



                             





                                                                    
                            
                               

                                        
                                 
 
                                 
                                                                         



                  


                                    
                                 
                             
                            

                           
  
 
                                                                         
 
                                       
                                          
                        
 



                                          



                                                               
                                                                         
 











                                                                 
 

                         
 
                                                 
                                                       




                                                                       
         


                       

 
                                                                          
                                                      
 
                               
 
               


                                     
               
                            
 







                                                 
         

                 
 
 

                                                                   
                               
                                  
 
                                 
 
                        
                                                                                 
                                 



                                                          



                                                         


                                                                          




                                                                          
                                                      

                                                         


                                                           
                 
         
                          

 



                                                                         
 
                                                      
 



                                                            
 

                                                      
                                                                  
 
                                                                
                                                   



                                                                  
                                                                  
                                                     
 

                                                      
                      
 


                                              
                                                                            


                                                                
                                                      
 
                                                          

                                                                       
 
 
                               



                                                               








                                                                     
 

                                      
                                                                       
 





                                                       
 
 
 

                                                               
                                                     
                       
 
                       

                                                             

                                                              







                                                                       
 











                                                        
 
                 

 
                                                                        
 
                                                                   

 
                                                         



                                        
                                      

  
                                                                              
 

                                                  
                       
                       
 



                                                                       
 
                                                                    
                                               








                                                                                 
         
 


                    
                                                                  
                       
 
                        
                                           

 
                                                                          
 
                                                  
                                            


                                        
                               
 
               



                                     
 

                                               
                                                                            




                                                                          
 








                                                                                  
                    



                      
 
 
                                                            
                       
 

                          
 

                                                   

                                     
 
 
                                                                             
 
                    





                                                  
                                                  


                           
 
                                                      












                                          
 
                                                                         
 

                                               
                                         
                   

 

                                                           
 
                                                        
                                   
 

                                               
 

                                                      
                                                               
 
                                               

                                                                      
                                               
 

                                                   

                                        
                                           

                                             
                                                      









                                                                         
                                                                             


                                                                       
                                                                               
                                                    

                         
         
 


                 
                                                       



                                     

  
                                                                    
 

                                                           

 
                                                     



                                    
                                   

  







































                                                                                   
                                                             




































                                                                           
                                                                                    
 
                               
 

                                                            

                                     
                                      


                                  

 


                                                            
                                                                 


                                                            
 
                           

                                
 





                                                                          
 
                                                             
                                     
                                    
 

                                                
 
                                                                 
                                                                
                                                                                   
                                
 


                                                            
 
                               


                                                                         




                                                                               

                                                                   
                                                        
                                                                 
                            


                                                                        
                                                                   
                                                              
 
                                                                
                 




                 
                                                        


                                      
                                     

  
                                                                     
 
                                                            
                                                           

 
                                                      



                                     
                                   

  
                                             
 
                                                                            

                          

                                                               

                          
                                                                          

                          
                 

     
                                             
     
                                         

                       

 
                                              
 


                                             


                           
/*
 *   This program is free software; you can redistribute it and/or
 *   modify it under the terms of the GNU General Public License
 *   as published by the Free Software Foundation; either version
 *   2 of the License, or (at your option) any later version.
 *
 *   Robert Olsson <robert.olsson@its.uu.se> Uppsala Universitet
 *     & Swedish University of Agricultural Sciences.
 *
 *   Jens Laas <jens.laas@data.slu.se> Swedish University of
 *     Agricultural Sciences.
 *
 *   Hans Liss <hans.liss@its.uu.se>  Uppsala Universitet
 *
 * This work is based on the LPC-trie which is originally described in:
 *
 * An experimental study of compression methods for dynamic tries
 * Stefan Nilsson and Matti Tikkanen. Algorithmica, 33(1):19-33, 2002.
 * http://www.csc.kth.se/~snilsson/software/dyntrie2/
 *
 *
 * IP-address lookup using LC-tries. Stefan Nilsson and Gunnar Karlsson
 * IEEE Journal on Selected Areas in Communications, 17(6):1083-1092, June 1999
 *
 *
 * Code from fib_hash has been reused which includes the following header:
 *
 *
 * INET		An implementation of the TCP/IP protocol suite for the LINUX
 *		operating system.  INET is implemented using the  BSD Socket
 *		interface as the means of communication with the user level.
 *
 *		IPv4 FIB: lookup engine and maintenance routines.
 *
 *
 * Authors:	Alexey Kuznetsov, <kuznet@ms2.inr.ac.ru>
 *
 *		This program is free software; you can redistribute it and/or
 *		modify it under the terms of the GNU General Public License
 *		as published by the Free Software Foundation; either version
 *		2 of the License, or (at your option) any later version.
 *
 * Substantial contributions to this work comes from:
 *
 *		David S. Miller, <davem@davemloft.net>
 *		Stephen Hemminger <shemminger@osdl.org>
 *		Paul E. McKenney <paulmck@us.ibm.com>
 *		Patrick McHardy <kaber@trash.net>
 */

#define VERSION "0.409"

#include <asm/uaccess.h>
#include <asm/system.h>
#include <linux/bitops.h>
#include <linux/types.h>
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/string.h>
#include <linux/socket.h>
#include <linux/sockios.h>
#include <linux/errno.h>
#include <linux/in.h>
#include <linux/inet.h>
#include <linux/inetdevice.h>
#include <linux/netdevice.h>
#include <linux/if_arp.h>
#include <linux/proc_fs.h>
#include <linux/rcupdate.h>
#include <linux/skbuff.h>
#include <linux/netlink.h>
#include <linux/init.h>
#include <linux/list.h>
#include <linux/slab.h>
#include <net/net_namespace.h>
#include <net/ip.h>
#include <net/protocol.h>
#include <net/route.h>
#include <net/tcp.h>
#include <net/sock.h>
#include <net/ip_fib.h>
#include "fib_lookup.h"

#define MAX_STAT_DEPTH 32

#define KEYLENGTH (8*sizeof(t_key))

typedef unsigned int t_key;

#define T_TNODE 0
#define T_LEAF  1
#define NODE_TYPE_MASK	0x1UL
#define NODE_TYPE(node) ((node)->parent & NODE_TYPE_MASK)

#define IS_TNODE(n) (!(n->parent & T_LEAF))
#define IS_LEAF(n) (n->parent & T_LEAF)

struct rt_trie_node {
	unsigned long parent;
	t_key key;
};

struct leaf {
	unsigned long parent;
	t_key key;
	struct hlist_head list;
	struct rcu_head rcu;
};

struct leaf_info {
	struct hlist_node hlist;
	struct rcu_head rcu;
	int plen;
	struct list_head falh;
};

struct tnode {
	unsigned long parent;
	t_key key;
	unsigned char pos;		/* 2log(KEYLENGTH) bits needed */
	unsigned char bits;		/* 2log(KEYLENGTH) bits needed */
	unsigned int full_children;	/* KEYLENGTH bits needed */
	unsigned int empty_children;	/* KEYLENGTH bits needed */
	union {
		struct rcu_head rcu;
		struct work_struct work;
		struct tnode *tnode_free;
	};
	struct rt_trie_node __rcu *child[0];
};

#ifdef CONFIG_IP_FIB_TRIE_STATS
struct trie_use_stats {
	unsigned int gets;
	unsigned int backtrack;
	unsigned int semantic_match_passed;
	unsigned int semantic_match_miss;
	unsigned int null_node_hit;
	unsigned int resize_node_skipped;
};
#endif

struct trie_stat {
	unsigned int totdepth;
	unsigned int maxdepth;
	unsigned int tnodes;
	unsigned int leaves;
	unsigned int nullpointers;
	unsigned int prefixes;
	unsigned int nodesizes[MAX_STAT_DEPTH];
};

struct trie {
	struct rt_trie_node __rcu *trie;
#ifdef CONFIG_IP_FIB_TRIE_STATS
	struct trie_use_stats stats;
#endif
};

static void put_child(struct trie *t, struct tnode *tn, int i, struct rt_trie_node *n);
static void tnode_put_child_reorg(struct tnode *tn, int i, struct rt_trie_node *n,
				  int wasfull);
static struct rt_trie_node *resize(struct trie *t, struct tnode *tn);
static struct tnode *inflate(struct trie *t, struct tnode *tn);
static struct tnode *halve(struct trie *t, struct tnode *tn);
/* tnodes to free after resize(); protected by RTNL */
static struct tnode *tnode_free_head;
static size_t tnode_free_size;

/*
 * synchronize_rcu after call_rcu for that many pages; it should be especially
 * useful before resizing the root node with PREEMPT_NONE configs; the value was
 * obtained experimentally, aiming to avoid visible slowdown.
 */
static const int sync_pages = 128;

static struct kmem_cache *fn_alias_kmem __read_mostly;
static struct kmem_cache *trie_leaf_kmem __read_mostly;

/*
 * caller must hold RTNL
 */
static inline struct tnode *node_parent(const struct rt_trie_node *node)
{
	unsigned long parent;

	parent = rcu_dereference_index_check(node->parent, lockdep_rtnl_is_held());

	return (struct tnode *)(parent & ~NODE_TYPE_MASK);
}

/*
 * caller must hold RCU read lock or RTNL
 */
static inline struct tnode *node_parent_rcu(const struct rt_trie_node *node)
{
	unsigned long parent;

	parent = rcu_dereference_index_check(node->parent, rcu_read_lock_held() ||
							   lockdep_rtnl_is_held());

	return (struct tnode *)(parent & ~NODE_TYPE_MASK);
}

/* Same as rcu_assign_pointer
 * but that macro() assumes that value is a pointer.
 */
static inline void node_set_parent(struct rt_trie_node *node, struct tnode *ptr)
{
	smp_wmb();
	node->parent = (unsigned long)ptr | NODE_TYPE(node);
}

/*
 * caller must hold RTNL
 */
static inline struct rt_trie_node *tnode_get_child(const struct tnode *tn, unsigned int i)
{
	BUG_ON(i >= 1U << tn->bits);

	return rtnl_dereference(tn->child[i]);
}

/*
 * caller must hold RCU read lock or RTNL
 */
static inline struct rt_trie_node *tnode_get_child_rcu(const struct tnode *tn, unsigned int i)
{
	BUG_ON(i >= 1U << tn->bits);

	return rcu_dereference_rtnl(tn->child[i]);
}

static inline int tnode_child_length(const struct tnode *tn)
{
	return 1 << tn->bits;
}

static inline t_key mask_pfx(t_key k, unsigned int l)
{
	return (l == 0) ? 0 : k >> (KEYLENGTH-l) << (KEYLENGTH-l);
}

static inline t_key tkey_extract_bits(t_key a, unsigned int offset, unsigned int bits)
{
	if (offset < KEYLENGTH)
		return ((t_key)(a << offset)) >> (KEYLENGTH - bits);
	else
		return 0;
}

static inline int tkey_equals(t_key a, t_key b)
{
	return a == b;
}

static inline int tkey_sub_equals(t_key a, int offset, int bits, t_key b)
{
	if (bits == 0 || offset >= KEYLENGTH)
		return 1;
	bits = bits > KEYLENGTH ? KEYLENGTH : bits;
	return ((a ^ b) << offset) >> (KEYLENGTH - bits) == 0;
}

static inline int tkey_mismatch(t_key a, int offset, t_key b)
{
	t_key diff = a ^ b;
	int i = offset;

	if (!diff)
		return 0;
	while ((diff << i) >> (KEYLENGTH-1) == 0)
		i++;
	return i;
}

/*
  To understand this stuff, an understanding of keys and all their bits is
  necessary. Every node in the trie has a key associated with it, but not
  all of the bits in that key are significant.

  Consider a node 'n' and its parent 'tp'.

  If n is a leaf, every bit in its key is significant. Its presence is
  necessitated by path compression, since during a tree traversal (when
  searching for a leaf - unless we are doing an insertion) we will completely
  ignore all skipped bits we encounter. Thus we need to verify, at the end of
  a potentially successful search, that we have indeed been walking the
  correct key path.

  Note that we can never "miss" the correct key in the tree if present by
  following the wrong path. Path compression ensures that segments of the key
  that are the same for all keys with a given prefix are skipped, but the
  skipped part *is* identical for each node in the subtrie below the skipped
  bit! trie_insert() in this implementation takes care of that - note the
  call to tkey_sub_equals() in trie_insert().

  if n is an internal node - a 'tnode' here, the various parts of its key
  have many different meanings.

  Example:
  _________________________________________________________________
  | i | i | i | i | i | i | i | N | N | N | S | S | S | S | S | C |
  -----------------------------------------------------------------
    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15

  _________________________________________________________________
  | C | C | C | u | u | u | u | u | u | u | u | u | u | u | u | u |
  -----------------------------------------------------------------
   16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31

  tp->pos = 7
  tp->bits = 3
  n->pos = 15
  n->bits = 4

  First, let's just ignore the bits that come before the parent tp, that is
  the bits from 0 to (tp->pos-1). They are *known* but at this point we do
  not use them for anything.

  The bits from (tp->pos) to (tp->pos + tp->bits - 1) - "N", above - are the
  index into the parent's child array. That is, they will be used to find
  'n' among tp's children.

  The bits from (tp->pos + tp->bits) to (n->pos - 1) - "S" - are skipped bits
  for the node n.

  All the bits we have seen so far are significant to the node n. The rest
  of the bits are really not needed or indeed known in n->key.