[SCSI] FC Transport support for vports based on NPIV

This patch provides support for FC virtual ports based on NPIV.
For information on the interfaces and design, please read the
Documentation/scsi/scsi_fc_transport.txt file enclosed within
the patch.

The RFC was originally posted here:
http://marc.info/?l=linux-scsi&m=117226959918393&w=2

Changes from the initial RFC:
- Bug fix: needed a transport_class_unregister() for the vport class
- Create a symlink to the vport in the shost device if it is not the
    parent of the vport.
- Made symbolic name writable so it can be set after creation
- Made the temporary fc_vport_identifiers struct private to the
transport.
- Deleted the vport_id field from the vport. I couldn't find any good
  use for it (and symname is a good replacement).
- Made the vport_state and vport_last_state "private" attributes.
  Added the fc_vport_set_state() helper function to manage state
  transitions
- Updated vport_create() to allow a vport to be created in a disabled
  state.
- Added INITIALIZING and FAILED vport states
- Added VPCERR_xxx defines for errors to be returned from vport_create()
- Created a Documentation/scsi/scsi_fc_transport.txt file that describes
  the interfaces and expected LLDD behaviors.

Signed-off-by: James Smart <James.Smart@emulex.com>
Signed-off-by: James Bottomley <James.Bottomley@SteelEye.com>

1 files changed, 450 insertions, 0 deletions

diff --git a/Documentation/scsi/scsi_fc_transport.txt b/Documentation/scsi/scsi_fc_transport.txt
new file mode 100644
index 000000000000..ab057afc757f
--- /dev/null
+++ b/Documentation/scsi/scsi_fc_transport.txt
@@ -0,0 +1,450 @@
+                             SCSI FC Tansport
+                 =============================================
+
+Date:  4/12/2007
+Kernel Revisions for features:
+  rports : <<TBS>>
+  vports : 2.6.22 (? TBD)
+
+
+Introduction
+============
+This file documents the features and components of the SCSI FC Transport.
+It also provides documents the API between the transport and FC LLDDs.
+The FC transport can be found at:
+  drivers/scsi/scsi_transport_fc.c
+  include/scsi/scsi_transport_fc.h
+  include/scsi/scsi_netlink_fc.h
+
+This file is found at Documentation/scsi/scsi_fc_transport.txt
+
+
+FC Remote Ports (rports)
+========================================================================
+<< To Be Supplied >>
+
+
+FC Virtual Ports (vports)
+========================================================================
+
+Overview:
+-------------------------------
+
+  New FC standards have defined mechanisms which allows for a single physical
+  port to appear on as multiple communication ports. Using the N_Port Id
+  Virtualization (NPIV) mechanism, a point-to-point connection to a Fabric
+  can be assigned more than 1 N_Port_ID.  Each N_Port_ID appears as a
+  separate port to other endpoints on the fabric, even though it shares one
+  physical link to the switch for communication. Each N_Port_ID can have a
+  unique view of the fabric based on fabric zoning and array lun-masking
+  (just like a normal non-NPIV adapter).  Using the Virtual Fabric (VF)
+  mechanism, adding a fabric header to each frame allows the port to
+  interact with the Fabric Port to join multiple fabrics. The port will
+  obtain an N_Port_ID on each fabric it joins. Each fabric will have its
+  own unique view of endpoints and configuration parameters.  NPIV may be
+  used together with VF so that the port can obtain multiple N_Port_IDs
+  on each virtual fabric.
+
+  The FC transport is now recognizing a new object - a vport.  A vport is
+  an entity that has a world-wide unique World Wide Port Name (wwpn) and
+  World Wide Node Name (wwnn). The transport also allows for the FC4's to
+  be specified for the vport, with FCP_Initiator being the primary role
+  expected. Once instantiated by one of the above methods, it will have a
+  distinct N_Port_ID and view of fabric endpoints and storage entities.
+  The fc_host associated with the physical adapter will export the ability
+  to create vports. The transport will create the vport object within the
+  Linux device tree, and instruct the fc_host's driver to instantiate the
+  virtual port. Typically, the driver will create a new scsi_host instance
+  on the vport, resulting in a unique <H,C,T,L> namespace for the vport.
+  Thus, whether a FC port is based on a physical port or on a virtual port,
+  each will appear as a unique scsi_host with its own target and lun space.
+
+  Note: At this time, the transport is written to create only NPIV-based
+    vports. However, consideration was given to VF-based vports and it
+    should be a minor change to add support if needed.  The remaining
+    discussion will concentrate on NPIV.
+
+  Note: World Wide Name assignment (and uniqueness guarantees) are left
+    up to an administrative entity controling the vport. For example,
+    if vports are to be associated with virtual machines, a XEN mgmt
+    utility would be responsible for creating wwpn/wwnn's for the vport,
+    using it's own naming authority and OUI. (Note: it already does this
+    for virtual MAC addresses).
+
+
+Device Trees and Vport Objects:
+-------------------------------
+
+  Today, the device tree typically contains the scsi_host object,
+  with rports and scsi target objects underneath it. Currently the FC
+  transport creates the vport object and places it under the scsi_host
+  object corresponding to the physical adapter.  The LLDD will allocate
+  a new scsi_host for the vport and link it's object under the vport.
+  The remainder of the tree under the vports scsi_host is the same
+  as the non-NPIV case. The transport is written currently to easily
+  allow the parent of the vport to be something other than the scsi_host.
+  This could be used in the future to link the object onto a vm-specific
+  device tree. If the vport's parent is not the physical port's scsi_host,
+  a symbolic link to the vport object will be placed in the physical
+  port's scsi_host.
+
+  Here's what to expect in the device tree :
+   The typical Physical Port's Scsi_Host:
+     /sys/devices/.../host17/
+   and it has the typical decendent tree:
+     /sys/devices/.../host17/rport-17:0-0/target17:0:0/17:0:0:0:
+   and then the vport is created on the Physical Port:
+     /sys/devices/.../host17/vport-17:0-0
+   and the vport's Scsi_Host is then created:
+     /sys/devices/.../host17/vport-17:0-0/host18
+   and then the rest of the tree progresses, such as:
+     /sys/devices/.../host17/vport-17:0-0/host18/rport-18:0-0/target18:0:0/18:0:0:0:
+
+  Here's what to expect in the sysfs tree :
+   scsi_hosts:
+     /sys/class/scsi_host/host17                physical port's scsi_host
+     /sys/class/scsi_host/host18                vport's scsi_host
+   fc_hosts:
+     /sys/class/fc_host/host17                  physical port's fc_host
+     /sys/class/fc_host/host18                  vport's fc_host
+   fc_vports:
+     /sys/class/fc_vports/vport-17:0-0          the vport's fc_vport
+   fc_rports:
+     /sys/class/fc_remote_ports/rport-17:0-0    rport on the physical port
+     /sys/class/fc_remote_ports/rport-18:0-0    rport on the vport
+
+
+Vport Attributes:
+-------------------------------
+
+  The new fc_vport class object has the following attributes
+
+     node_name:                                                 Read_Only
+       The WWNN of the vport
+
+     port_name:                                                 Read_Only
+       The WWPN of the vport
+
+     roles:                                                     Read_Only
+       Indicates the FC4 roles enabled on the vport.
+
+     symbolic_name:                                             Read_Write
+       A string, appended to the driver's symbolic port name string, which
+       is registered with the switch to identify the vport. For example,
+       a hypervisor could set this string to "Xen Domain 2 VM 5 Vport 2",
+       and this set of identifiers can be seen on switch management screens
+       to identify the port.
+
+     vport_delete:                                              Write_Only
+       When written with a "1", will tear down the vport.
+
+     vport_disable:                                             Write_Only
+       When written with a "1", will transition the vport to a disabled.
+       state.  The vport will still be instantiated with the Linux kernel,
+       but it will not be active on the FC link.
+       When written with a "0", will enable the vport.
+
+     vport_last_state:                                          Read_Only
+       Indicates the previous state of the vport.  See the section below on
+       "Vport States".
+
+     vport_state:                                               Read_Only
+       Indicates the state of the vport.  See the section below on
+       "Vport States".
+
+     vport_type:                                                Read_Only
+       Reflects the FC mechanism used to create the virtual port.
+       Only NPIV is supported currently.
+
+
+  For the fc_host class object, the following attributes are added for vports:
+
+     max_npiv_vports:                                           Read_Only
+       Indicates the maximum number of NPIV-based vports that the
+       driver/adapter can support on the fc_host.
+
+     npiv_vports_inuse:                                         Read_Only
+       Indicates how many NPIV-based vports have been instantiated on the
+       fc_host.
+
+     vport_create:                                              Write_Only
+       A "simple" create interface to instantiate a vport on an fc_host.
+       A "<WWPN>:<WWNN>" string is written to the attribute. The transport
+       then instantiates the vport object and calls the LLDD to create the
+       vport with the role of FCP_Initiator.  Each WWN is specified as 16
+       hex characters and may *not* contain any prefixes (e.g. 0x, x, etc).
+
+     vport_delete:                                              Write_Only
+        A "simple" delete interface to teardown a vport. A "<WWPN>:<WWNN>"
+        string is written to the attribute. The transport will locate the
+        vport on the fc_host with the same WWNs and tear it down.  Each WWN
+        is specified as 16 hex characters and may *not* contain any prefixes
+        (e.g. 0x, x, etc).
+
+
+Vport States:
+-------------------------------
+
+  Vport instantiation consists of two parts:
+    - Creation with the kernel and LLDD. This means all transport and
+      driver data structures are built up, and device objects created.
+      This is equivalent to a driver "attach" on an adapter, which is
+      independent of the adapter's link state.
+    - Instantiation of the vport on the FC link via ELS traffic, etc.
+      This is equivalent to a "link up" and successfull link initialization.
+  Futher information can be found in the interfaces section below for
+  Vport Creation.
+
+  Once a vport has been instantiated with the kernel/LLDD, a vport state
+  can be reported via the sysfs attribute. The following states exist:
+
+    FC_VPORT_UNKNOWN            - Unknown
+      An temporary state, typically set only while the vport is being
+      instantiated with the kernel and LLDD.
+
+    FC_VPORT_ACTIVE             - Active
+      The vport has been successfully been created on the FC link.
+      It is fully functional.
+
+    FC_VPORT_DISABLED           - Disabled
+      The vport instantiated, but "disabled". The vport is not instantiated
+      on the FC link. This is equivalent to a physical port with the
+      link "down".
+
+    FC_VPORT_LINKDOWN           - Linkdown
+      The vport is not operational as the physical link is not operational.
+
+    FC_VPORT_INITIALIZING       - Initializing
+      The vport is in the process of instantiating on the FC link.
+      The LLDD will set this state just prior to starting the ELS traffic
+      to create the vport. This state will persist until the vport is
+      successfully created (state becomes FC_VPORT_ACTIVE) or it fails
+      (state is one of the values below).  As this state is transitory,
+      it will not be preserved in the "vport_last_state".
+
+    FC_VPORT_NO_FABRIC_SUPP     - No Fabric Support
+      The vport is not operational. One of the following conditions were
+      encountered:
+       - The FC topology is not Point-to-Point
+       - The FC port is not connected to an F_Port
+       - The F_Port has indicated that NPIV is not supported.
+
+    FC_VPORT_NO_FABRIC_RSCS     - No Fabric Resources
+      The vport is not operational. The Fabric failed FDISC with a status
+      indicating that it does not have sufficient resources to complete
+      the operation.
+
+    FC_VPORT_FABRIC_LOGOUT      - Fabric Logout
+      The vport is not operational. The Fabric has LOGO'd the N_Port_ID
+      associated with the vport.
+
+    FC_VPORT_FABRIC_REJ_WWN     - Fabric Rejected WWN
+      The vport is not operational. The Fabric failed FDISC with a status
+      indicating that the WWN's are not valid.
+
+    FC_VPORT_FAILED             - VPort Failed
+      The vport is not operational. This is a catchall for all other
+      error conditions.
+
+
+  The following state table indicates the different state transitions:
+
+    State              Event                            New State
+    --------------------------------------------------------------------
+     n/a                Initialization                  Unknown
+    Unknown:            Link Down                       Linkdown
+                        Link Up & Loop                  No Fabric Support
+                        Link Up & no Fabric             No Fabric Support
+                        Link Up & FLOGI response        No Fabric Support
+                          indicates no NPIV support
+                        Link Up & FDISC being sent      Initializing
+                        Disable request                 Disable
+    Linkdown:           Link Up                         Unknown
+    Initializing:       FDISC ACC                       Active
+                        FDISC LS_RJT w/ no resources    No Fabric Resources
+                        FDISC LS_RJT w/ invalid         Fabric Rejected WWN
+                          pname or invalid nport_id
+                        FDISC LS_RJT failed for         Vport Failed
+                          other reasons
+                        Link Down                       Linkdown
+                        Disable request                 Disable
+    Disable:            Enable request                  Unknown
+    Active:             LOGO received from fabric       Fabric Logout
+                        Link Down                       Linkdown
+                        Disable request                 Disable
+    Fabric Logout:      Link still up                   Unknown
+
+         The following 4 error states all have the same transitions:
+    No Fabric Support:
+    No Fabric Resources:
+    Fabric Rejected WWN:
+    Vport Failed:
+                        Disable request                 Disable
+                        Link goes down                  Linkdown
+
+
+Transport <-> LLDD Interfaces :
+-------------------------------
+
+Vport support by LLDD:
+
+  The LLDD indicates support for vports by supplying a vport_create()
+  function in the transport template.  The presense of this function will
+  cause the creation of the new attributes on the fc_host.  As part of
+  the physical port completing its initialization relative to the
+  transport, it should set the max_npiv_vports attribute to indicate the
+  maximum number of vports the driver and/or adapter supports.
+
+
+Vport Creation:
+
+  The LLDD vport_create() syntax is:
+
+      int vport_create(struct fc_vport *vport, bool disable)
+
+    where:
+      vport:    Is the newly allocated vport object
+      disable:  If "true", the vport is to be created in a disabled stated.
+                If "false", the vport is to be enabled upon creation.
+
+  When a request is made to create a new vport (via sgio/netlink, or the
+  vport_create fc_host attribute), the transport will validate that the LLDD
+  can support another vport (e.g. max_npiv_vports > npiv_vports_inuse).
+  If not, the create request will be failed.  If space remains, the transport
+  will increment the vport count, create the vport object, and then call the
+  LLDD's vport_create() function with the newly allocated vport object.
+
+  As mentioned above, vport creation is divided into two parts:
+    - Creation with the kernel and LLDD. This means all transport and
+      driver data structures are built up, and device objects created.
+      This is equivalent to a driver "attach" on an adapter, which is
+      independent of the adapter's link state.
+    - Instantiation of the vport on the FC link via ELS traffic, etc.
+      This is equivalent to a "link up" and successfull link initialization.
+
+  The LLDD's vport_create() function will not synchronously wait for both
+  parts to be fully completed before returning. It must validate that the
+  infrastructure exists to support NPIV, and complete the first part of
+  vport creation (data structure build up) before returning.  We do not
+  hinge vport_create() on the link-side operation mainly because:
+    - The link may be down. It is not a failure if it is. It simply
+      means the vport is in an inoperable state until the link comes up.
+      This is consistent with the link bouncing post vport creation.
+    - The vport may be created in a disabled state.
+    - This is consistent with a model where:  the vport equates to a
+      FC adapter. The vport_create is synonymous with driver attachment
+      to the adapter, which is independent of link state.
+
+    Note: special error codes have been defined to delineate infrastructure
+      failure cases for quicker resolution.
+
+  The expected behavior for the LLDD's vport_create() function is:
+    - Validate Infrastructure:
+        - If the driver or adapter cannot support another vport, whether
+            due to improper firmware, (a lie about) max_npiv, or a lack of
+            some other resource - return VPCERR_UNSUPPORTED.
+        - If the driver validates the WWN's against those already active on
+            the adapter and detects an overlap - return VPCERR_BAD_WWN.
+        - If the driver detects the topology is loop, non-fabric, or the
+            FLOGI did not support NPIV - return VPCERR_NO_FABRIC_SUPP.
+    - Allocate data structures. If errors are encountered, such as out
+        of memory conditions, return the respective negative Exxx error code.
+    - If the role is FCP Initiator, the LLDD is to :
+        - Call scsi_host_alloc() to allocate a scsi_host for the vport.
+        - Call scsi_add_host(new_shost, &vport->dev) to start the scsi_host
+          and bind it as a child of the vport device.
+        - Initializes the fc_host attribute values.
+    - Kick of further vport state transitions based on the disable flag and
+        link state - and return success (zero).
+
+  LLDD Implementers Notes:
+  - It is suggested that there be a different fc_function_templates for
+    the physical port and the virtual port.  The physical port's template
+    would have the vport_create, vport_delete, and vport_disable functions,
+    while the vports would not.
+  - It is suggested that there be different scsi_host_templates
+    for the physical port and virtual port. Likely, there are driver
+    attributes, embedded into the scsi_host_template, that are applicable
+    for the physical port only (link speed, topology setting, etc). This
+    ensures that the attributes are applicable to the respective scsi_host.
+
+
+Vport Disable/Enable:
+
+  The LLDD vport_disable() syntax is:
+
+      int vport_disable(struct fc_vport *vport, bool disable)
+
+    where:
+      vport:    Is vport to to be enabled or disabled
+      disable:  If "true", the vport is to be disabled.
+                If "false", the vport is to be enabled.
+
+  When a request is made to change the disabled state on a vport, the
+  transport will validate the request against the existing vport state.
+  If the request is to disable and the vport is already disabled, the
+  request will fail. Similarly, if the request is to enable, and the
+  vport is not in a disabled state, the request will fail.  If the request
+  is valid for the vport state, the transport will call the LLDD to
+  change the vport's state.
+
+  Within the LLDD, if a vport is disabled, it remains instantiated with
+  the kernel and LLDD, but it is not active or visible on the FC link in
+  any way. (see Vport Creation and the 2 part instantiation discussion).
+  The vport will remain in this state until it is deleted or re-enabled.
+  When enabling a vport, the LLDD reinstantiates the vport on the FC
+  link - essentially restarting the LLDD statemachine (see Vport States
+  above).
+
+
+Vport Deletion:
+
+  The LLDD vport_delete() syntax is:
+
+      int vport_delete(struct fc_vport *vport)
+
+    where:
+      vport:    Is vport to delete
+
+  When a request is made to delete a vport (via sgio/netlink, or via the
+  fc_host or fc_vport vport_delete attributes), the transport will call
+  the LLDD to terminate the vport on the FC link, and teardown all other
+  datastructures and references.  If the LLDD completes successfully,
+  the transport will teardown the vport objects and complete the vport
+  removal.  If the LLDD delete request fails, the vport object will remain,
+  but will be in an indeterminate state.
+
+  Within the LLDD, the normal code paths for a scsi_host teardown should
+  be followed. E.g. If the vport has a FCP Initiator role, the LLDD
+  will call fc_remove_host() for the vports scsi_host, followed by
+  scsi_remove_host() and scsi_host_put() for the vports scsi_host.
+
+
+Other:
+  fc_host port_type attribute:
+    There is a new fc_host port_type value - FC_PORTTYPE_NPIV. This value
+    must be set on all vport-based fc_hosts.  Normally, on a physical port,
+    the port_type attribute would be set to NPORT, NLPORT, etc based on the
+    topology type and existence of the fabric. As this is not applicable to
+    a vport, it makes more sense to report the FC mechanism used to create
+    the vport.
+
+  Driver unload:
+    FC drivers are required to call fc_remove_host() prior to calling
+    scsi_remove_host().  This allows the fc_host to tear down all remote
+    ports prior the scsi_host being torn down.  The fc_remove_host() call
+    was updated to remove all vports for the fc_host as well.
+
+
+Credits
+=======
+The following people have contributed to this document:
+
+
+
+
+
+
+James Smart
+james.smart@emulex.com
+