Key Emerging Data Center Standards

NETWORK VIRTUALIZATION

Several standardization activities related to network virtualization are currently in process.  TRILL is emerging as a replacement for the Spanning Tree Protocol, while EVB addresses requirements imposed on the network due to server virtualization.

TRILL – Transparent Interconnection of Lots of Links
Specified in IEEE 802.1D, the Spanning Tree Protocol (STP) has been a very useful method to limit loops in Layer 2 networks.  The protocol has seen widespread use since it was first proposed by Dr. Radia Perlman in the 1980s.  More recent STP enhancements to support faster convergence, using 802.1w Rapid Spanning Tree Protocol (RSTP), and multiple instances and VLANs using Multiple Spanning Tree Protocol (MSTP), have also seen adoption.

Though it’s widely used today, STP protocol has several clear limitations.  For one, it disables ports that can cause loops in the network, thus limiting options for load balancing.  Also, because all traffic effectively follows the same path based on priorities set by configuration, STP does not provide options for multipathing.  MSTP addresses some of these issues, but it does not provide the same flexibility as a Layer 3 router.

Enter TRILL (Transparent Interconnection of Lots of Links), which provides the benefits of Layer 3 on a Layer 2 network. This effort within the Internet Engineering Task Force (IETF) was first proposed by Dr. Perlman along with others.  TRILL introduces a new kind of network node called the Routing Bridge (RBridge), which uses network style routing but on a Layer 2 network. RBridges communicate with each other using the Intermediate System-Intermediate System (IS-IS) protocol, with extensions to establish shortest paths across a Layer 2 topology.  Using TRILL, Layer 2 Ethernet frames are encapsulated and forwarded hop-by-hop as they would be by a router.  A hop count is present in each encapsulated frame, and this is decremented at each RBridge allowing temporary loops to be controlled.  TRILL permits multipathing and efficient multicasting.

Information about the TRILL Working Group (WG) and its documents can be found at http://datatracker.ietf.org/wg/trill/.  The TRILL Problem Statement can be found in the informational RFC 5556 at http://datatracker.ietf.org/doc/rfc5556/.  The IEEE also has an initiative for shortest path bridging within its IEEE 802.1aq WG. More details are available at http://www.ieee802.org/1/pages/802.1aq.html.

EVB – Edge Virtual Bridging
Server virtualization has seen increased use in data centers due to its ability to increase the efficiency of previously underutilized servers.  Multiple physical machines are now represented by virtual machines (VMs) inside a single server.  VMs run on top of a hypervisor or virtual machine manager, which often includes a virtual switch to enable switching of traffic between VMs and across to the physical network link.  The physical link is connected to an external physical switch.

The IEEE 802.1Qbg Edge Virtual Bridging (EVB) Working Group has been chartered to specify protocols and procedures to allow multiple VMs to share a common physical link to the switch in a consistent and coordinated manner.  For example, a host may contain an embedded virtual switch for switching traffic between VMs, rely on an external physical switch for VM to VM traffic forwarding, or have a VM talk directly to an external physical switch through a physical adapter. 802.1Qbg is involved in specifying the mechanisms to discover, configure, and control these types of connections to an external switch. 
The IEEE 802.1 Working Group has agreed to base the IEEE 802.1Qbg EVB standard on Virtual Ethernet Port Aggregator (VEPA) technology because of its and minimal impact and changes required to existing network products and devices.

The 802.1Qbh Bridge Port Extension Working Group is specifying the protocols and procedures related to providing additional (logical) MAC ports on a single physical port in a switch. This Port Extension capability is useful since it collapses the number of switch layers in the network. With this functionality, you do not have to build complex software virtual switches on the servers.  802.1Qbh extends the work being done in EVB by adding the remote replication service for virtualized physical servers.  This provides the flexibility to apply filtering and forwarding rules from the physical switch.

Work on both these efforts began in late 2009.  The 802.1Qbg WG information is available at http://www.ieee802.org/1/pages/802.1bg.html. The 802.1Qbh WG information is available at http://www.ieee802.org/1/pages/802.1bh.html.

FABRIC CONVERGENCE TO ETHERNET

Disparate data center networks are beginning to converge into a single Ethernet-based fabric, and several key standardization activities related to this effort are now underway.  FCoE has emerged to carry Fibre Channel frames inside Ethernet frames, and DCB standardization initiatives are targeted towards enhancing Ethernet to become the single fabric.

FCoE – Fibre Channel over Ethernet
Consider a typical data center installation. Each server has an Ethernet adapter used to connect the server to an access Ethernet switch. This connection is used for communication with IP enabled devices within or outside the data center. For storage connectivity, each server typically includes a Fibre Channel (FC) adapter to connect to an FC switch.  The local area network (LAN) and the storage area network (SAN) were two separate infrastructures.

Over the past few years, there have been efforts to integrate the network and storage infrastructure connectivity through the use of converged networks and adapters.  In such installations, FC traffic from servers is carried inside Ethernet frames.  The standard which addresses this is known as Fibre Channel over Ethernet (FCoE). Server adapters which support FCoE are known as Converged Network Adapters (CNAs).  These 10Gbps Ethernet adapters present an FC interface to the server software, while transparently encapsulating the FC frames into Ethernet frames on the link. FCoE frames are terminated at an Ethernet switch which is also connected to the Fibre Channel infrastructure. 

Converged networks which carry FCoE traffic need to be lossless so as to accurately emulate the FC infrastructure.  The section on Data Center Bridging standardization outlines efforts in this area.

FCoE has been specified by ANSI in FC-BB-5.  This specification is available at http://www.t11.org/ftp/t11/pub/fc/bb-5/09-056v5.pdf.  Further standardization and enhancement efforts are being driven by ANSI T11 (http://www.t11.org/index.html).

DCB – Data Center Bridging
Data Centers impose a specific set of requirements on the Ethernet network. These include lower latency, the ability to prioritize specific traffic types (e.g. storage traffic), and the flexibility to manage traffic assignment.  The storage traffic requirement has gained prominence due to the Fibre Channel over Ethernet (FCoE) standard from the ANSI T11.  This standard requires Ethernet to provide a lossless capability similar to Fibre Channel networks today.  

To address the requirements of the data center, the IEEE has chartered the Data Center Bridging Task Group (see http://www.ieee802.org/1/pages/dcbridges.html). This task group focuses on enhancements to the IEEE 802.1 specifications to satisfy data center requirements.  There are four major initiatives to address the enhancements required:

• Priority-based Flow Control (PFC – being addressed in the 802.1bb EG, see http://www.ieee802.org/1/pages/802.1bb.html) – this is a link level flow control mechanism which enhances the current PAUSE scheme in Ethernet by incorporating up to 8 priorities for the various types of traffic.
• Congestion Notification (CN – being addressed in the 802.1au WG, see http://www.ieee802.org/1/pages/802.1au.html)  – this is an end to end congestion management scheme that can be used by the network. It involves notification of congestion to the source of the traffic.
• Enhanced Transmission Selection (ETS – being addressed in the 802.1az WG, see http://www.ieee802.org/1/pages/802.1az.html)  – this allows flexible bandwidth management for various traffic classes.
• Data Center Bridging Exchange (DCBX) – this is a discovery and capability exchange protocol used between neighboring switches for consistent configuration across the network.  The Link Layer Discovery Protocol (LLDP) will be used as the base protocol for this work. It will be addressed in the 802.1az WG.

SCALING ETHERNET TO EVER HIGHER SPEEDS

Ethernet speeds have been steadily increasing since the time the technology was first proposed as a 10 Mbps technology. While 10 Gigabit Ethernet (GbE) is experiencing broad adoption in the data center, there is a need for higher speed Ethernet connectivity at the server edge, as well for higher capacity interconnects in the data center core. This section details the standardization activities in this area.

40 Gbps and 100 Gbps Ethernet Standardization
The IEEE 802.3ba Task Force is specifying high speed Ethernet at two link speeds – 40 and 100 Gbps. Both the speeds use a common architecture and retain the familiar IEEE 802.3 Ethernet MAC and frame format. There are multiple PHYs supported at these speeds.  40Gbps PHYs include a backplane PHY with reach of up to at least 1m, shielded copper assembly with a reach up to 7m, multimode fiber with a reach up to 100m and single mode fiber PHY with a reach of up to at least 10 km.

The 100 Gbps PHY options include shielded copper cable assembly with a reach up to 7m, multimode fiber PHY with a reach up to 100m, two single mode fiber PHYs, one with a reach up to at least 10km and the other with a reach up to at least 40km.

The 802.3ba standard is expected to be ratified in June of 2010. The chair for the IEEE 802.3.ba Task Force is Force10’s John D’Ambrosia.  More information can be obtained from http://www.ieee802.org/3/ba/.