What is Data Center Interconnect/Interconnection?

Data Center Interconnection means the implementation of the Data Center Interconnect (DCI) technology. With the DCI technology advances, better and cheaper options have become available and this has created a lot of confusion. This is compounded by the fact that a lot of companies are trying to enter this market because there is a lot of money to be made. This article is written to straighten out some of the confusion.

According to the various applications, there are two parts of the data center interconnections. The first is an intra-data center interconnect (intra-DCI) which means connections within the data center. It can be within one building or between data center buildings in a campus. Connections can be a few meters up to 10km. The second is inter-data center interconnect (inter-DCI) which means connections between data centers from 10km to 80km. Of course, connections can be much longer but most of the market activity for inter-DCI is focused at 10km to 80km. Longer connections are considered Metro or Long-haul. For reference, please see the table below.

DCI	Distance	Fiber Type	Optics technology	Optical Transceivers
intra-DCI	300m	MMF	NRZ / PAM4	QSFP28 SR4
	500m	SMF		QSFP28 PSM4
	2km			QSFP28 CWDM4
	10km			QSFP28 LR4
inter-DCI	10km	SMF	Cohernet	QSFP28 4WDM-10
	20km			QSFP28 4WDM-20
	30km to 40km			QSFP28 4WDM-40
	80km to 2000km			CFP2-ACO

Intra-DCI

The big bottlenecks are in the intra-DCI and therefore, the highest volume of optical transceivers are sold here generating the most revenue, however, it is low margin revenue because there is so much competition. In this space, may of the connections are less than 300m and Multi-Mode Fiber (MMF) is frequently used. MMF is thicker, and components are cheaper because the tolerances are not as tight, but the light disperses as it bounces around in the thick cable. Therefore, 300m is the limit for many types of high speed transmission that use MMF. There is a data center transceiver with a transmission distance up to 100m over OM4 MMF for example.

100G QSFP28 SR4 for MMF up to 100m

In a data center, everything is connected to servers by routers and switches. Sometimes a data center can be one large building bigger than a football field and other times data centers are built on a campus of many buildings spanning many blocks. In the case of a campus, the fiber is brought to one hub and the connections are made there. Even if the building you want to connect to might be 200m away, the fiber runs to a hub, which can be more than 1km away, so this type of routing increases the fiber distance. Some of the distances between buildings can be 4km, requiring Single Mode Fiber (SMF), which has a much narrower core, making it more efficient, but also increasing the cost of all related components because the tolerances are tighter. Therefore, with data centers growing, so has the need for SMF as the connections get longer within the data center. With SMF you have the option to drive high bandwidth with coherent technology, and we'll see more of this in the future. Previously coherent was only used for longer distances, but with cost reductions and greater efficiency versus other solutions, coherent is now being used for shorter reaches in the data center.

100G QSFP28 LR4L for SMF up to 4km

500m is a new emerging market and because the distance is shorter, a new technology is emerging, and that is silicon photonics modulators. EMLs (Externally Modulated Lasers) perform modulation within the laser, but with silicon photonics, the modulator is outside the laser and it's a good solutions for distances of 500m. In an EML, the modulator is integrated into the same chip, but is outside the laser cavity, and hence is "external". For silicon photonics, the laser and modulator are on different chips and usually in different packages. Silicon photonics modulators are based on the CMOS manufacturing process that is high scale and low cost. A continuous wave laser with silicon photonic modulation is very good for 500m applications. EMLs are more suitable for longer reaches, such as 2-10km. Therefore, with data centers growing, so has the need for single mode fiber as the connections get longer within the data center. With SMF you have the option to drive high bandwidth with coherent technology, and we'll see more of this in the future. Previously coherent was only used for longer distances, but with cost reductions and greater efficiency versus other solutions, coherent is now being used for shorter reaches in the data center.

100G QSFP28 PSM4 for SMF up to 500m/2km

100G QSFP28 CWDM4 for SMF up to 2km

100G QSFP28 LR4 for SMF up to 10km

Inter-DCI

Inter-DCI is typically between 10km and 80km, including 20km and 40km. Before we talk about data center connectivity, let's talk about why data centers are set up the way they are and why 80km is such an important connection distance. While it is true that a data center in New York might backup to tape in a data center in Oregon, this is considered regular long-haul traffic. Some data centers are geographically situated to serve an entire continent and others are focused on a specific metro area. Currently, the throughput bottleneck is in the metro and this is where data centers and connectivity are most needed.

100G QSFP28 4WDM-20 for SMF up to 20km

100G QSFP28 4WDM-40 for SMF up to 40km

Say you have a Fortune 100 retailer and they are running thousands of transactions per second. The farther away a data center is, the more the data is secure because the data center is so far away and separate from natural disasters, but with the increased distance there are more "in flight" transactions are at risk of being lost due to latency. Therefore, for online transactions there might be a primary data center that is central to retail locations and a secondary data center that is around 80km away. It's far enough away not to be affected by local power outages, tornadoes, etc, but close enough that there is only a few hundred milli-seconds of latency; therefore, in the worst case a small number of transactions would be at risk.

In another example of inter-DCI, as if a certain video is getting a lot of views, the video is not only kept in its central location, but copies of the video are pushed to metro data centers where access is quicker because it's stored closer to the user, and the traffic doesn't tie up long haul networks. Metro data centers can grow to a certain size until their sheer size becomes a liability with no additional scale advantage and thus they are broken up into clusters. Once again, to guard against natural disasters and power outages, data centers should be far away. Counterbalancing this, data centers need to have low latency communication between them, so they shouldn't be too far away. There is a compromise and the magic distance is 80km for a secondary data center, so you'll hear about 80km data center interconnect a lot.

It used to be that on-off keying could provide sufficient bandwidth between data centers, but now with 4K video and metro bottlenecks, coherent transmission is being used for shorter and shorter distances. Coherent is likely to take over the 10km DCI market. It has already taken over the 80km market but it might take time before coherent comes to 2km. The typical data center bottlenecks are 500m, 2km, and 80km. As coherent moves to shorter distances, this is where the confusion comes.

The optical transceiver modules that were only used within the data center are gaining reach, and they're running up against coherent solutions that were formerly only used for long distances. Due to the increasing bandwidth and decreasing cost, coherent is being pulled closer into the data center.

The other thing to think about is installing fiber between data centers. Hopefully this is already done, because once you dig, it's a fixed cost, so you put down as many fibers as you can. Digging just for installing fiber is extremely expensive. In France when they lay fiber, they use another economic driver. Whenever you put in train tracks, you put in fiber at the same time, even if it is not needed. It's almost for free because they are digging anyway. Fibers are leased to data centers one at a time; therefore, data centers try to get as much bandwidth as possible onto each fiber (this is also a major theme in the industry). You might ask, why not own your own fiber? You need to have a lot of content to own your own fiber. The cost is prohibitive. In order to make the fiber network function, all the nodes need to use the same specification and this is hard. Therefore, carriers are usually the ones to install the full infrastructure.

Article Source: John Houghton, a Silicon Valley entrepreneur, technology innovator, and head of MobileCast Media.