Since the standard for 10 Gigabit Ethernet (IEEE802.3ae) has been ratified in 2002, 10GbE deployments have grown rapidly year by year. And it is now mainly used to interconnect servers and routers in data center, especially in the Spine-Leaf network. Previously, IT managers often use 10GbE switch as the leaf switch, which provides high performance but cost a lot. Nowadays, as the price and power consumption of 10GBASE-T technology continue to drop, 10G copper switches are more popular in the Spine-Leaf network than before. This post will discuss how to build a Spine-Leaf network with 10GBASE-T switches.
Generally, there are four 10GbE links for Spine-Leaf network Switching: 10GBASE-CX4, SFP+ fiber, SFP+ Direct Attach Copper (DAC), and 10GBASE-T. Among them, 10GBASE-CX4 link is an early favorite for 10 GbE deployments, but it is now outdated for its bulky and expensive cables plus its limited reach of 15 meters. SFP+ fiber links of 10G fiber switches are great for latency and distance (up to 300m), but it is the most costly solution. SFP+ DAC is a lower cost alternative to SFP+ fiber, but its reach for passive cables is limited to 7m, and it is not backward-compatible with existing GbE switches.
10GBASE-T link of 10G copper switch uses the lowest cost twisted-pair copper cables for networking and supports lengths up to 100m, giving IT managers a great level of flexibility in accommodating top of rack network topology. In addition, it is also backward-compatible with 1000BASE-T therefore can be deployed in legacy GbE network infrastructures. Hence, 10G copper switch is much more favored than 10G fiber switch for Spine-Leaf network in data centers. The following table shows the 10GBASE-T switch vs 10G fiber switch:
|10GBASE-T switch||10G fiber switch|
|Modern Market Entrance||Brand New||Existing|
|Existing LAN Cable Usage (Cat5e)||Yes (Up to 2.5/5GbE)||No|
|Cabling for 10GbE Cost||Cheap||Expensive|
|10 GbE Network Adapter Cost||Cheap||Expensive|
There are mainly three components involved for a Spine-Leaf network: spine switches, leaf switches, and servers. This part will take 40G Spine-Leaf network as an example to explain how to build Spine-Leaf network with 10GBASE-T switches.
According to the above, a 40G Spine-Leaf network may need 40G spine switches, leaf switches with 40G QSFP+ uplinks, and servers & storage devices.
Here we use the S8050-20Q4C (20 QSFP+ ports and 4 QSFP28 ports, US$ 5499) as the spine switch, and the 48 port 10GBASE-T switch with 4xQSFP+ uplink —S5850-48T4Q (48 10GBASE-T ports and 4 QSFP+, US$ 4599) as the leaf switch. Servers and storage devices can be either simple Network Attached Storage (NAS) devices or a complex Storage Area Network (SAN). This depends on your actual needs (Read NAS vs SAN: What’s the Difference? for more details).
In the 40G Spine-Leaf network, links between spine and leaf switches are 40G, and links between leaf switches and servers are 1/10G. Therefore, the S5850-48T4Q are often connected to the spine switch (S8050-20Q4C) through the 4OG QSFP+ ports by MTP/MPO fiber cables, while the 10GBASE-T copper ports of leaf switch (S5850-48T4Q) are used to connect to servers with Cat6a cables. One thing to note is that the uplink numbers of each leaf switch determines the number of spine switches, and the 40G QSFP+ ports on each spine switch determines the number of leaf switches we can have. In our case, it can be 2 spine switches and 10 leaf switches, So in building this Spine-Leaf architecture, the maximum amount of 10G servers is 480 at 3:1 oversubscription:
Except for switches, transceivers, fiber and copper cables are also necessary for building such 40G Spine-Leaf network.The following table shows all the fiber optics used in the above Spine-Leaf network:
|S8050-20Q4C (Spine Switch)||US$ 5499|
|S5850-48T4Q (Leaf Switch)||US$ 4599|
|40G QSFP+ Transceiver||From US$ 49|
|MTP/MPO Fiber Patch Cable||From US$ 32|
|Cat6a Copper Cable||From US$ 3.4|
Using 10GBASE-T SFP+ switch in the Spine-Leaf network will make the best of your existing Cat6a UTP structured cabling ecosystem. Further more, it will help to save much money because copper cable infrastructure is far less expensive than the fiber optics of 10 SFP+ switch.