To date, most server, embedded and storage systems typically have had a single-master in their hierarchy.  One CPU is designated the master and enumerates all the components in the entire system, allocates memory to the end-points and manages most of the transactions -- staying true to its designation of a master.  For those applications requiring failover and redundancy, designers needed to use two CPUs, which tend to cause conflict between the two masters.  In order to ensure a single-master hierarchy in a PCI Express (PCIe)-based system, those designers used the non-transparent port in a PCIe switch to isolate the two CPUs and remove any conflict between the two masters.

Now, a new-generation of storage systems and other bandwidth-demanding applications are on the horizon that need to use multiple CPUs that improve performance.  This is achieved by focusing the CPUs on a smaller number of end-points and maximizing their operational efficiency.   In turn, the goal requires a new generation of PCIe switches that can support multiple hosts.  Such switches provide multiple upstream ports to connect to several CPUs, thereby reducing the number of system components as well as latency.  In addition to their newfound efficiencies, these systems also provide several points of failover for protection.

PCIe Switches with Multiple Hosts Improve Connectivity

Figure 1 shows an example of how a high-density PCIe switch with multiple upstream ports provides better connectivity.  In this usage model, the PCIe switch with multiple upstream ports makes it possible for a single-chip solution to deliver high-speed throughput.  If such a device were not available, designers would have to use a two-chip solution to connect the four GPUs to the CPU, thus adding to both the part count and cost of the system.  However, a PCIe switch with two x16 wide upstream ports provides the necessary connectivity and enables the CPU to maximize the operations for each GPU.

Figure 1. Ultra-gaming machine

Servers Using PCIe Switches with Multiple Hosts

Figure 2 shows an example of a high-density PCIe switch with multiple upstream ports providing better performance in a server application.  In this application, the PCIe device acts as a virtual switch to isolate the end-points from the respective CPUs.  With such isolation, each CPU is now free to focus on the transactions, and hence the performance, of the end-points assigned to it.  In case of a failover, the end-points of the failing CPU can be assigned to one of the surviving CPUs.  If designs did not have a single PCIe switch supporting multiple hosts, they would be forced to use multiple PCIe switches with a single upstream port and then have to write complex software to enable failover. Also, designs would have to worry about layout and routing issues for these multiple PCIe switches. A single-chip PCIe solution goes a long way toward solving these issues.

Figure 2. Servers using PCIe switch with multiple hosts

Communication Systems Using PCIe Switches with Multiple Hosts, Multicast

In addition to using switches with multiple hosts, communication systems also benefit from switches that feature multicast, due to the number of ports transferring data at the same time.  Multicast allows an incoming packet to be sent simultaneously to multiple devices without involving the CPU for every transaction. Figure 3 shows an example of a PCIe switch with both features - multiple upstream ports and multicast. The PCIe switch provides multiple upstream ports and also offers multicasting ability from any port to any port – the PCI-SIG specification for multicast provides up to 64 multicast groups. In addition to having multiple lanes and ports, these two features (multiple hosts and multicast) are key for communication systems.

Figure 3. Communication systems using PCIe switch with multiple hosts and multicast

Storage Systems Using PCIe Switches with Multiple Hosts, Multicast

It is not just communication systems but also storage systems that need support for multiple hosts and multicast features in a PCIe switch.  Storage standards such as U320 SCSI, SAS, SATA2 and multi-gigabit Fibre Channel have been delivering performance improvements with increased connection speeds. PCIe takes these speeds one step further by providing the ability to deploy advanced storage hardware without encountering logjams of data.  The aforementioned storage standards are achieving speeds in excess of 10 Gbit/s, and multicast and other PCIe switches’ advanced can unleash that performance.

Figure 4 shows an application of multicast and multiple hosts.  Here, the multiple upstream ports in the PCIe switch on IO drawers is being used to connect to processor boards and multicast feature is being used to make copies of the data being received to ensure that there is no data-loss in the system.

PCIe provides storage systems with the most robust interconnect available, and dramatically improves their reliability and data availability.  Since it’s a point-to-point architecture, PCIe eliminates many of the drawbacks of older and/or less-powerful interconnects, such as those based on a shared bus.  (A shared bus often masks how many devices are residing on it and what impact the various devices are having on bus bandwidth.) 

Figure 4. Storage systems using PCIe switch with multiple hosts and multicast

Debugging PCI Express Simplified through Switches

In addition to the array of innovative features in the new generation PCIe switches cited above, perhaps some of the most overlooked among them are their diagnostics and debug tools.  While some designers may choose to use a protocol analyzer for the final phase of their product development, not all have the budget or other resources for such a step.  Alternatively, an internal debug probe in some PCIe switches (such as from PLX) can perform many of the debug tasks, performance monitoring and eye diagram functions as would a protocol analyzer. 

Conclusion

As can be seen from the above usage models, systems – whether they be storage, communications or embedded -- need innovative features found in a new generation of PCIe switches.  These systems’ requirements for lanes and ports are increasing rapidly and along with it the need for multiple upstream ports and multicast.  Ultimately, when these requirements are met we’ll see a dramatic increase in performance, reduced cost of the system (i.e., a single-chip solution) and simplified system software.

Krishna Mallampati is product marketing director at PLX Technology, Sunnyvale, Calif.