Introduction to Fibre Channel (FC)
If you are new to Fibre Channel SAN technology you should take the time to familiarise yourself with the basic terminology. Fibre Channel (FC) was developed in the late 1980s and became an American National Standards Institute (ANSI) standard in 1994 with it’s physical topology as well as logical topology and protocols are all defined by the ANSI T11 standard. As the name suggests, it was designed to be based on fibre optic cabling as the physical transport to overcome some of the limitations of the SCSI physical layer. Fibre Channel can actually run on copper cables over short distances, where Fibre Channel can easily go 10KM. A storage area network (SAN) is a specialised high-speed network that connects host servers to high-performance storage subsystems. The SAN components include host bus adapters (HBAs) in the host servers, switches that help route storage traffic, cables, storage processors (SPs), and storage disk arrays.
Fibre Channel Switches and Fabrics
What Is a Fabric?
A Fibre Channel switch on its own, or two or more switches connected together, forms a Fibre Channel Fabric. A switch can only be in one Fabric. Some switches can now be partitioned into what are effectively multiple switches or VSAN’s, but each partition is still in only one Fabric. The primary function of a Fabric is to correctly route traffic from one port to another, where the source (initiator) and destination (target) ports may or may not be connected to the same switches but must be in the same Fabric. SANs are usually comprised of at least two separate Fabrics, for redundancy. Servers and storage have connections to both Fabrics for resilience and server side software multipaths between the devices.
To transfer traffic from host servers to shared storage, the SAN uses the Fibre Channel protocol that packages SCSI commands into Fibre Channel frames. Fibre Channel was designed specifically for storage networks with inherent features of providing lossless delivery of raw block data, as well as in order delivery of packets, both of which improve reliability and efficiency. The Fibre Channel protocol is fast, cost effective, and reliable over a wide variety of storage workloads.
Server Host Bus Adapters
To be able to connect to a SAN a server requires one or more Fibre Channel Host Bus Adapters (HBAs) to be installed and configured. Fibre Channel HBAs handle all the processing of the Fibre Channel stack using onboard ASICs. Fibre Channel requires very little CPU usage from the host. Qlogic QLE2562 HBA pictured below.
Fabric Name Server
The Fibre Channel standard defines a name service called the Fabric Simple Name Server. The “Simple” is commonly omitted. The Fabric Name Server stores information about all the devices in the Fabric. An instance of the name server runs on every Fibre Channel switch in a SAN.
World-Wide Names
All Fibre Channel devices have a unique IEEE issued 64 bit World-Wide Name (WWN). This is used to identify devices in a SAN.
Some devices allow their WWN to be changed (e.g. some Fibre Channel tape drives) and this can be useful in certain service situations but, as with changing MAC addresses on Ethernet Network Interface Cards, this needs to be done carefully.
Ports in Fibre Channel SAN
A port is the connection from a device into the SAN. Each node in the SAN, such as a host, a storage device, or a fabric component has one or more ports that connect it to the SAN. Ports are identified in a number of ways.
WWPN (World Wide Port Name)
A globally unique identifier for a port that allows certain applications to access the port. The FC switches discover the WWPN of a device or host and assign a port address to the device.
Fabric Login
When a device (e.g. server HBA port, array controller port, Fibre Channel tape drive port) is connected to a Fibre Channel switch, it goes through an initialization process called a Fabric Login.
- The switch registers the device’s WWN with the Fabric Name Server which dynamically generates a 24 bit Port Address for the device which is unique within the Fabric.
- If the device is an initiator (typically a server HBA port) it will then attempt to log in to every other device it can see in the Fabric to find out what it is. This process is required so that the server can discover the characteristics of the devices and so “connect” them correctly to its operating system.
Port_ID (or port address)
Each port has a unique port ID that serves as the FC address for the port. The Port Address assigned to a device describes the location of the device in the SAN and has information that identifies the switch and switch port the device is connected to. The port ID is valid only while the device is logged on. The information provided by the Port Address enables traffic to be routed very efficiently across the Fabric. You can see what is currently logged in by looking that the flogi database with the below command.
SAN-MDS-1# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc1/1 100 0xb00600 21:00:00:24:ff:34:9c:c2 20:00:00:24:ff:34:9c:c2
fc1/2 100 0xb00700 21:00:00:24:ff:34:9c:c3 20:00:00:24:ff:34:9c:c3
fc1/3 100 0xb00100 21:00:00:24:ff:37:0a:80 20:00:00:24:ff:37:0a:80
fc1/25 200 0x080600 21:00:00:24:ff:34:c0:c4 20:00:00:24:ff:34:c0:c4
fc1/26 200 0x080700 21:00:00:24:ff:34:c0:c5 20:00:00:24:ff:34:c0:c5
fc1/27 200 0x080000 21:00:00:24:ff:37:0a:81 20:00:00:24:ff:37:0a:81
Total number of flogi = 6.Principal Switch
In a Fabric one of the switches is the Principal Switch. It is responsible for keeping all of the Name Server instances in the Fabric up to date. The decision about which switch in a Fabric is Principal can be forced by the administrator, or the switches can be left to decide for themselves using built-in logic. If the Principal Switch fails, another will take over the role automatically.
If a new device is connected to a Fabric, the Name Server on the Principal Switch is responsible for generating a Port Address and notifying all of the other switches. Similarly, if a device is removed from the Fabric, then the Name Server on the Principal Switch is notified and it then sends notifications to all of the other switches.
If a new switch is connected to a Fabric, it exchanges information with the Principal Switch and vice-versa, and then the Name Server on each of the other switches in the Fabric is updated. A switch joining a stable Fabric cannot become the Principal.
SAN Access Control
Fibre Channel Switch Zones
A zone is made up of a number of devices grouped by their World Wide Names (WWN), or is a group of switch ports. Zoning information is managed and stored in the Fabric Name Server and is communicated to new switches when they join a fabric. Devices can only see other devices in the same zone, so zones enable servers and the storage devices they use to be isolated from other servers and their storage devices. Zones can overlap, i.e. a switch port can be in more than one zone and a device WWN can be in more than one zone. Zones can span multiple switches, i.e. they can contain device WWNs or ports from anywhere in a Fabric.
Fibre Channel, SCSI LUNs and LUN Masking
What is seen as a LUN from a server’s point of view is more often than not actually a slice of a RAID set or pool created within an array that is presented to the SAN by the array controller. An array presents a Fibre channel port or ports to a SAN and the slices are then presented on the ports using the SCSIFCP protocol, and the servers see them as LUNs. Fibre channel switches see the controller port and its WWN but do not “speak SCSI” and so cannot control access to the slices presented on a port on an individual basis. As many servers may be sharing an array controller port in a SAN, it is necessary to have a way of controlling access to the slices presented to the SAN, and LUN Masking provides that. LUN Masking is a function of the array controller. When LUN Masking is enabled, each slice in an array has an access control list which is a list of the WWNs of the HBAs that are allowed to access it. This access control list is defined by the storage administrator using array management tools. When a server scans the SAN for storage devices, it will only see the slices to which it has been granted access. Since most servers will be connected to a SAN with at least two HBA ports via two separate Fabrics, access to a slice/partition will have to be granted to both of the server HBA port WWNs.
Multipathing
When transferring data between the host server and storage, the SAN uses a technique known as multipathing. Multipathing allows you to have more than one physical path from the ESXi host to a LUN on a storage system and provides the added features of redundancy and the potential for improved performance with multipathing used to increase bandwidth and throughput . If a switch, HBA, or fibre optic cable fails, the connection to the storage is not lost.
Path Fail-over
Generally, a single path from a host to a LUN consists of an HBA, switch ports, connecting cables, and the storage controller port. If any component of the path fails and multipathing has been setup, the host selects another available path for I/O.
Fibre Channel supports the following connection topology’s between host and storage.
Active-active storage system
A system in which access to the LUNs simultaneously through all the storage ports that are available without significant performance degradation. All the paths are active, unless a path fails.
Active-passive storage system
A system in which one storage processor is actively providing access to a given LUN. The other processors act as a backup for the LUN and can be actively providing access to other LUN I/O. I/O can be successfully sent only to an active port for a given LUN. If access through the active storage port fails, one of the passive storage processors can be activated by the servers accessing it.
Asymmetrical storage system
Asymmetric Logical Unit Access (ALUA). ALUA-compliant storage systems provide different levels of access per port. With ALUA, the host can determine the states of target ports and prioritise paths. The host uses some of the active paths as primary, and uses others as secondary.
Currently Fibre Channel speeds up to 128 Gbps are available. However, the most common implementation in use today is 32Gbps, usually with a mix of lower speed 16 Gbps or 8 Gbps devices on the SAN.
The overall result of using a Fibre Channel SAN is a system that is very reliable, very fault tolerant with a very high performance.
Links to Fibre Channel posts
Set up Cisco MDS DS-C9148-16p-K9 with 2 VSAN’s
How to adjust the Round Robin IOPS limit from default 1000 to 1 ESXi 7
