100-Gbit Networks in Data Centers: Technology, Optics, and Planning

Putting 100GbE into Perspective

100GbE primarily refers to a nominal Ethernet port speed of 100 Gbit/s at the MAC layer. This does not imply a specific connector type, fiber type, lane allocation, or transmission distance. The Ethernet roadmap continues to classify 100GbE alongside 25, 50, 200, 400, and higher data rates; the appropriate tier depends on the traffic profile, port density, and deployment goals.

The nominal port capacity should not be equated with an application's throughput. Ethernet, IP, and transport protocol overhead reduce the available payload; in addition, the CPU, NIC driver, PCIe interface, memory or storage performance, and the remote end may also impose limitations. Competing data streams also share a port’s capacity. 100 Gbit/s are therefore an interface property and do not guarantee data transfer for a single application.

For example, a server with a 100GbE NIC may operate well below the port speed when copying large amounts of data if the source or destination cannot read or write fast enough. Conversely, a 100G uplink increases the total capacity of a server pod without automatically boosting every individual host to 100 Gbit/s. Therefore, end systems, data paths, and peak loads must all be considered together during planning.

100GbE is not necessarily too small or too large. 25GbE may be sufficient for general server connections, while 50GbE serves as an intermediate step for higher host density. Depending on the design, 100GbE is suitable for servers, storage, leaf uplinks, or smaller spine layers. For highly aggregated workloads, 200 or 400GbE can provide the required port density. Published IEEE papers cover multiple Ethernet speeds and PHY variants; however, speed alone does not define a complete network architecture.

Lanes, PHYs, and Error Correction

To make sound decisions, three levels must be evaluated separately: the electrical lanes within the device, the optical PHY on the link, and the passive infrastructure consisting of cables, fibers, and patch panels. A 100GbE port can use multiple lanes internally or at the host’s electrical interface. Older approaches, which are still widely used, often bundle four 25 Gbit/s lanes; newer interfaces specify electrical lanes up to 100 Gbit/s.

The number of lanes does not, in and of itself, indicate how an optical module transmits data. A PHY can use parallel optical paths or transmit 100 Gbit/s over a single wavelength. This results in different requirements for fiber type, connectors, and patch paths. Therefore, a matching data rate and the same form factor are not sufficient to prove compatibility: the switch, network card, and transceiver must support the specific PHY variant.

The Error correction Forward Error Correction (FEC) is a feature that must be verified based on the PHY and platform documentation when selecting and commissioning a specific 100G connection. A universal FEC setting cannot be determined based solely on the port speed or the module form factor. The specifications and supported operating modes of the devices and transceivers at both ends are decisive.

The QSFP28 form factor primarily describes the physical design and electrical interface family, but does not imply universal interchangeability of optics or breakout cables. A breakout cable requires that the port mode, lane allocation, peer profile, and cabling be designed to support it. Therefore, prior to procurement and migration, the approvals from the respective switch and NIC manufacturers for the module, cable, PHY, FEC, and breakout mode must be verified and aligned. IEEE 802.3ck addresses electrical interfaces up to 100 Gbit/s per lane, but does not replace such platform verification.

100G in the Leaf-Spine Network

In a routed Leaf-Spine Network Servers, storage systems, and other endpoints first connect to leaf switches. Each leaf is connected to spine switches via multiple peer-to-peer Layer 3 connections. Data traffic between two servers in different leaf zones thus flows from the source leaf through a spine to the destination leaf, rather than having to traverse a central Layer 2 network.

This structure is designed for east-west traffic—that is, communication within the data center between applications, databases, storage, and compute nodes. The Layer 3 underlay provides multiple equivalent paths. ECMP (Equal-Cost Multi-Path) can distribute data flows across these paths using a hashing algorithm. This allows for the utilization of aggregated bandwidth; however, whether sufficient capacity remains after an uplink failure depends on the specific system configuration.

The role of 100GbE is determined by the port profile. A densely packed compute or storage server may require 100GbE at the host; in another pod, 100G ports aggregate multiple 25G servers as leaf uplinks. Similarly, 100G links can form the spine layer of a smaller design. The key factor is the ratio of connected host capacity to uplink capacity—that is, the deliberately planned Overbooking.

High overall capacity does not arbitrarily accelerate a single data flow. In many implementations, ECMP assigns a flow to a specific path; its speed remains limited by the bandwidth of that path and by end systems. Unevenly distributed hash values or a small number of very large flows can therefore lead to bottlenecks despite available capacity on other links. Capacity planning must therefore take into account both the sum of all flows and dominant individual flows, as well as failure scenarios.

Compare 100G Optics and Cables

Selecting a 100G connection doesn't start with the QSFP28 housing, but with the entire transmission path. A transceiver defines a specific PHY with requirements for fiber, optical paths, and range. The same form factor can therefore accommodate different technologies that are not automatically compatible with each other, either optically or electrically.

For short parallel connections, use 100GBASE-SR4 Multimode fiber and, typically, MPO-12/MTP-12 connectors. The four optical transmit and receive paths make correct polarity a key consideration in the design process. OM3 is specified for distances up to 70 m, and OM4 for distances up to 100 m; existing duplex LC cabling cannot therefore be used as an SR4 link without the appropriate infrastructure.

The documented Juniper modules for DR1, FR1, and LR1 use single-mode fiber and duplex LC connectors; the specified ranges are 500 m, 2 km, and 10 km. The module connector is, however, a specific product characteristic and conforms to the manufacturer’s documentation. The PHY, optical medium, number of fibers, and module connector must be verified separately. Connector surfaces, couplers, splices, and reserves must also be included in the attenuation calculation.

The designation "LR" requires special care: LR1 and LR4 Although both can reach up to 10 km over single-mode fiber, they are different PHYs. LR1 operates with a single 100 Gbit/s wavelength, while LR4 uses a different multi-wavelength method. Therefore, the same two-fiber SMF infrastructure and mechanically compatible product variants do not guarantee optical interoperability. The specific module connector also depends on the product; for example, the documented Cisco CPAK-100GBASE-LR4 variant uses a duplex SC connector.

DACs and AOCs are a separate consideration for short distances. There is no universal rule for the range of passive or active copper cables and active optical cables that can be derived from „100G.“ Before purchasing, you must check the compatibility matrix for both port sides, as well as the specific cable length and the intended bend radius.

Planning Uplinks Using Capacity Calculations

The following illustrative calculation example specifically describes a single-homing design: 48 hosts each have a total of one 25 GbE port. Twenty-four hosts each are connected to exactly one of two leaves. This results in 600 Gbit/s of connected host capacity per leaf and 1,200 Gbit/s across the entire pod. This is a capacity calculation, not measured application throughput.

Each Leaf has two uplinks, each with a capacity of 100 Gbit/s. The uplink capacity is therefore 200 Gbit/s per Leaf and 400 Gbit/s across both Leafs. For each Leaf, 600 divided by 200 results in an assumed overbooking ratio of 3:1. This figure represents installed link capacity; usable throughput also depends on data streams, remote endpoints, and protocols.

If a single uplink fails, the affected leaf still has 100 Gbit/s available for its 600 Gbit/s host connections. The theoretical ratio there increases to 6:1. In this deliberately non-redundant host connection model, the failure of an entire leaf renders its 24 hosts unreachable. Therefore, two leaves alone do not provide a fail-safe connection for every server.

A dual-homing design must be calculated separately. With 48 hosts, each having two 25G ports, the total physical bandwidth is 2,400 Gbit/s. Whether both ports carry traffic simultaneously or one serves as a backup affects the load assumptions. Therefore, document the number of ports, the active operating model, the capacity per leaf, and the desired residual capacity for each failure scenario under consideration.

Traffic distribution is also part of the planning process. ECMP distributes flows across equivalent routed paths; link aggregation bundles physical links into a single logical connection. Both methods must be evaluated separately. With hash-based distribution, a single flow does not automatically receive the entire aggregate bandwidth. Plan for dominant data flows, backup windows, and reserves in addition to the calculated ratio.

Migration from MPO to Duplex SMF

The transition from an SR4 environment to DR1 or FR1 is an infrastructure project, not an automatic improvement in quality. SR4 uses parallel multimode paths with MPO-12/MTP-12. DR1 and FR1, on the other hand, use duplex single-mode fiber; the module’s connector type is a product feature. The DR1 and FR1 modules documented in the Juniper overview have duplex LC connectors. Existing cabling runs, patch panels, and available capacity therefore determine whether a parallel migration or a phased transition makes more sense.

First, the actual range requirements are determined: DR1 is intended for 500 m, and FR1 for 2 km. If all paths are within a single room, a functional SR4 cabling setup may still be appropriate. If paths regularly exceed the SR4 limits or if sections of a building need to be connected, duplex SMF with the appropriate PHY creates a different planning framework.

• Document existing fiber paths, fiber classes, patch panels, and MPO polarity for each connection.
• Determine the required range and the appropriate PHY; do not select DR1 and FR1 based solely on the connector type.
• Plan duplex SMF fiber paths, including product-specific module connections, attenuation margin, connector faces, and redundant paths.
• Evaluate future port speeds, available slots, and the timing of the transition for each pod.
• Verify the switch, NIC, transceiver, and, if applicable, breakout options as a specifically approved combination.

The MPO Polarity This should be documented as a separate test point as long as parts of the parallel SR4 cabling remain in place. A mechanically compatible adapter does not automatically answer questions regarding signal assignment, the number of fibers, or the permissible transmission method. As a planning recommendation, therefore, it is advisable to clearly separate and label the patch paths: SR4 over MMF on one hand, and DR1 or FR1 over SMF on the other.

Duplex SMF can simplify expansion paths, but it is not universally „better.“ The costs of new cabling, available routes, operational processes, and expected port growth are just as relevant as range. The earlier classification of 100-Gbit/s PON refers to an access network approach and is not a compatibility or architectural template for Ethernet in the data center; therefore, it should not be confused with 100GbE PHYs.100 Gbit/s PON at a Glance

Before making a purchase, it is essential to verify the specific combination of switch, network card, and transceiver on both ends. This involves checking supported PHYs, FEC specifications, optical encoding, module connectors, and permissible cables. Only this approval ensures that the planned passive cabling connects to an actively supported link; a compatible enclosure size alone is not sufficient.

Acceptance Testing, Telemetry, and Operation

The acceptance testing of a 100GbE link begins with a comprehensive inventory, not with a single link-up event. For each connection, document the complete fiber path, including the route, patch panels, connectors, and port designations. Include the type of optical module used, fiber class, connector type, target range, and the FEC setting selected on both ends. Manufacturer specifications categorize 100G optical modules by range, fiber, and transmission method; these documented product specifications must match the actual link.

• Check the link status and remote station assignment at both ends.
• Verify the optical type, serial number, fiber type, and documented patch path.
• Record the FEC status and error count before handover and after a defined monitoring period.
• Trace redundancy paths systematically and verify the associated alarm procedures.
• Enter the escalation contact, replacement lens, and current compatibility approval for the respective port.

The value specified in the optical documentation Reach describes the respective module variant and its intended transmission medium; it does not constitute a blanket approval of every installed link. For acceptance testing, a project-specific attenuation analysis is therefore also recommended. This documentation records the actual patch points, connectors, splices, and planned reserves. Clean connector surfaces and complete route documentation are also operational testing requirements and should be explicitly specified in the acceptance plan.

During operation, monitoring systems should record link flaps, FEC-corrected and uncorrectable errors, symbol or PCS errors, optical diagnostic values, and the load per path, provided that the platform and transceivers make these values available. A single counter value does not constitute a diagnosis: what matters are the trend, direction, and temporal relationship to changes in patching, optics, or configuration. If anomalies are detected, the operations team first determines whether only one path, a single device, or multiple similar connections are affected.

For redundancy, it is not enough to simply have two logical links. Verify that they actually run over separate ports, fibers, and—where applicable—separate patch paths, and that alarm rules report a loss in a timely manner. Requirements for physical resilience and data center operations are part of the overall planning; the article explains this in more detail. Criteria for Data Center Location, Security, and Scalability broader contextual factors.

Systematically Narrow Down 100G Faults

If a 100G link fails or becomes unstable, a fixed troubleshooting sequence is more efficient than immediately replacing multiple components. First, check the approved combination of switch or NIC, transceiver, cable, and PHY, as well as the expected FEC mode, at both ends. Electrical interfaces with 100 Gbit/s per lane are standardized; however, this does not automatically mean that a specific platform supports every optical module or breakout variant.

This is followed by the passive section: fiber type, connector face, patch path, labeling, and—for parallel optics in particular—the MPO Polarity . Reversed polarity can prevent a link from establishing, while confusing OM3 and OM4 affects the permissible design length. SR4 requires multimode and parallel optical paths; DR1, FR1, and LR1, on the other hand, operate over single-mode duplex links. Therefore, a cassette that looks similar or a matching connector does not prove that the transmission modes are compatible.

If error rates are elevated, compare the error counters on both sides and attribute the onset of errors to changes. If values increase on only one link, the optics, connector surfaces, patch cord, and port are obvious points to check. If errors occur on multiple connections of the same type, common factors such as FEC settings, optical encoding, or a platform profile must be investigated. Documented transceiver specifications help verify that the range and fiber requirements match the network topology.

You should only investigate unexpectedly low application data rates after establishing a stable physical link. A single flow may be limited by its path speed, ECMP hash distribution, or the remote end, even though the aggregated fabric has spare capacity. In addition, the PCIe interface, NIC drivers, CPU, packet sizes, protocol overhead, and storage performance all limit payload throughput. The 100GbE port is therefore a potential bottleneck, but not necessarily the primary one.

Evaluating RoCE and Expansion Decisions

For RDMA over Converged Ethernet, port speed alone is not a sufficient selection criterion. The NVIDIA source used here explicitly describes Cumulus Linux 4.0 and even refers to more recent product documentation. It therefore serves as historical, product-specific evidence for the mechanisms discussed. Its commands and thresholds do not constitute a general, up-to-date operating guide for other platforms.

This documentation distinguishes between RoCEv1 at the security layer and RoCEv2 at Layer 3. For RoCEv1, it describes Priority Flow Control (PFC). For RoCEv2, it specifies Explicit Congestion Notification (ECN) for end-to-end signaling over routed networks, as well as flow control for low-loss Ethernet. However, neither a current universal PFC configuration nor the portability of the historical Cumulus Linux 4.0 settings to other platforms can be inferred from this.

For a specific deployment, review the current documentation for the NICs, switches, and software versions in use. Which RoCE version is planned, which traffic classes will be used, and which congestion control mechanisms does this combination support? Also, define how queue utilization, packet loss, ECN markings, and PFC pauses will be monitored. Do not adopt historical thresholds or global activation recommendations without platform-specific validation.

The choice between 25, 50, 100, 200, or 400 GbE depends on the load profile, the host connection, and the required residual capacity in the event of a failure. The Ethernet roadmap outlines technological development trends, including 100G–800G connections and emerging 1.6 Tbit/s Ethernet. However, it does not specify a particular leaf-spine architecture or the suitability of a particular speed for each server class. Separate capacity planning remains necessary for this purpose.

Finally, distinguish between published standards, ongoing projects, and product features that are actually available. As of this writing, the IEEE lists P802.3dj as an active project for 200, 400, and 800 Gbit/s, as well as 1.6 Tbit/s. This does not constitute a commitment regarding existing hardware. For each upgrade path, verify port support, optics, fiber routes, power consumption, and operating costs based on the products actually selected.