40G and 100G Ethernet technology is no longer a pipe dream for data centers – it’s reality. As data centers of all types continue to grow in terms of traffic and size, 100G is set to become the new standard for high bandwidth and intelligent architecture. It will take some time to spread industry-wide, but 100G is well on its way.

IEE Ethernet Standard

During the initial development of 40G and 100G Ethernet, the IEEE 802.3 Ethernet working group considered only multimode optics and multimode fiber cable solutions (with a reach of up to 150m for 40G and 100m for 100G) for data center applications.

IEEE 40G and 100G singlemode fiber solutions have been developed for a reach of 10km to 40km in metropolitan area networks; however, these options are often far too expensive for data centers. Especially for 100G Ethernet technology deployment in data centers, there was a huge gap between the 100m (multimode fiber) and 10km (singlemode fiber) reaches. Hyperscale data centers need cost-effective solutions for 500m and low-cost CWDM-based 2km reach. Because high-volume physical medium dependent sublayers (PMDs) – which define the details of transmission and reception of individual bits on a physical medium – are missing in the IEEE Ethernet technology standard, non-standard MSA PMDs have filled the gap:

  • At 40G, there are four IEEE standard (40G-SR4, 40G-FR, 40G-LR4 and 40G-ER4) PMDs and six non-standard PMDs (40G-BiDi, 40G-SWDM, 40G-eSR4, 40G-PSM4, 40G-LR-lite, 40G-Universal)
  • At 100G, there are six IEEE PMDs (100G-SR10, 100G-SR4, 100G-LR4, 100G-ER4, 100G-SR2 and 100G-DR) and eleven non-standard PMDs (100G-eSR4, 100G-BiDi, 100G-SWDM4, 100G-PSM4, 100G-CWDM4, 100G-CLR4, 100G-LR4-lite, 100G-CWDM4-lite, 100G-eCWDM4, 100G-eLR4 and 100G-ER4-lite)

According to LightCounting market reach, this year, more than 60% of 40G PMD shipments, and more than 35% of 100G PMD shipments, are from non-standard PMDs.

In 2014 and 2016, IEEE 802.3 formed 802.3bs (200G and 400G) and 802.3cd (50G, 100G and 200G) taskforces for next-generation Ethernet technology development; many of the data-center-oriented PMDs have become the main focus of these taskforces. Their new objectives are to develop the following solutions to support cost-effective data center interconnects:

  • 3m solution with twin-axial direct-attach copper (DAC)
  • 100m solution in multimode fiber
  • 500m/2km/10km solution in singlemode fiber

Next Generation Data Centers - Are You Ready For Scale?Industry Alliances and a Fragmented Market

Industry consortiums, such as the 25G Ethernet Consortium, 100G CWDM4 MSA, 100G CLR4 Alliance, 100G PSM4 MSA and OpenOptics MSA, have paved the way for fast 25G and 100G Ethernet technology development outside the IEEE 802.3 Ethernet working group.

Along with the IEEE 802.3cd and 802.3bs working groups, many new industry consortiums have formed to work on developing next-generation 200G and 400G Ethernet technology interfaces and form factors, including 400G CDFP MSA, 400G CFP8, 200G/400G QSFP-Double Density MSA and 400G OSFP MSA.

To develop the application-driven technologies not addressed in the IEEE 802.3 working group objectives, many new industry alliances were founded. For example, the SWDM4 Alliance was formed to develop specifications for short wavelength-division multiplexing (SWDM) in multimode fiber for 40G and 100G applications. 4WDM MSA defines specifications for low-cost, low-power 100G (4× 25G) transceivers with 10km, 20km and 40km reach.

As speed and port density become bottlenecked, the Consortium for On-Board Optics was formed to define specifications for next-generation on-board optics solutions as an alternative to current dominant-edge pluggable optics.

The data center ecosystem has evolved so quickly that it has become a fragmented market. For example, hyperscale data centers have different technology preferences and migration paths to suit their own applications and expansion plans. This will accelerate technology development and benefit the rest of the market in product availability for different market segments.

New Ethernet Interface

To better understand the Ethernet roadmap, we must keep in mind that connectivity technology has always been developed to support switch ASICs (application-specific integrated circuits); switch ASIC bandwidth is driven by applications.

Ethernet Switch ASIC

Moving beyond 25G-per-lane speed, the switch ASICs in BGA (ball grid array) packages will require eventual migration to higher electrical lane speeds to support higher bandwidth; switch ASIC connectivity is mainly limited by the serializer-deserializer (SerDes) I/O.

The industry recognizes the value of leveraging common technology developments across multiple applications by implementing multiple lane configurations, such as 50G in single lane, 100G in two lanes and 200G in four lanes.

For example, the QSFP double-density (QSFP-DD) form factor will be developed to support 200G and 400G Ethernet technology applications with eight parallel electrical lanes; µQSFP is another MSA form factor under development to support ~4× the lane density of SFP in a similar form factor.

Interfaces blog

Next-generation Ethernet form factors (Source: Ethernet Alliance)

On-Board Optics

On-board optics (Source: Avago/Broadcom)

Another exciting technology development involves on-board-optics (OBO) that allow placement of optical modules very close to switch ASIC I/O; this eases signal integrity issues and considerably improves switch faceplate density. OBO specifications are under development by the Consortium for On-Board Optics (COBO), which will allow attaching high-density MPO connectors directly at the faceplate. For example, with 50G-per-lane technology, a 32-port switch with an MPO-12 (Base-8) interface can support a bandwidth of 6.4 Tbps with a single wavelength, or 25.6 Tbps with four wavelengths per fiber.

Next-Gen Ethernet Switch Interface

If your organization is planning next-generation Ethernet technology deployment and has unanswered questions, contact Belden. Our data center experts can help you prepare for 40G, 100G and beyond.