The transition from shared multi-drop parallel buses such as PCIto point-to-point serial links has major performance and architectural ramifications. Andrew Brown reports.
Although 14-year-old PCI (and 10-year-old CompactPCI) are growing somewhat long in the tooththey still provide adequate bandwidth for many embedded applications. Neverthelesstraditional buses fall far short for various types of systems with very heavy-duty I/O handling requirementsand they can quickly bog down.
Systems collecting large quantities of sensor data in geophysicalmedical and military applications are typical examples. Herethe more flexible and extensible point-to-point links run rings around buses.
The Inova 3U CompactPCI Express CPUs combine the rugged hardware of the 3U Eurocard form factor with the diagnostic techniques of IPMIproducing a high-performance platform that is both rugged and backward compatible with existing 3U CompactPCI.
PCI Express has the most momentum among today’s high-speed point-to-point serial linksand its adaptation for the rigorous environmental requirements of embedded applications has created a capable foundation that will last well into the future.
With this adaptation of the CompactPCI Express (PICMG EXP.0 R1.0) standardapplications traditionally reserved for 6U boards and systems can now be realised in the more compact and robust 3U form factor. And even more robust solutions are possible by integrating the intelligent platform management interface (IPMI) technology of the enterprise server worldwith its remote diagnostic and maintenance functionsto create a platform that’s suitable for the most mission-critical embedded applications. CompactPCI Express Robustness is what one might call it.
Bandwidth unlimited
PCIe differs entirely to the 32/64-bit PCI and PCI-X shared parallel buses as it is a scalable high-speed serialpacket-based architecturethat was designed to overcome the bandwidth limitations of PCI/PCI-Xyet still provide new standardised featuresand help speed up a project's time to market.
A conventional 32-bit33MHz PCI bus has a maximum data transfer rate of 132Mbytes/second. Doubling the frequency to 66MHz boosts that figure to 264 Mbytes/seconda capability which both 3U (100 x 160mm) and 6U (233.35 x 160mm) CompactPCI boards can support. With their additional connector space6U CompactPCI boards can also double data path width to 64bits to achieve 264Mbytes/secondand if they double both data rate and bus widththey reach a rate of 528Mbytes/second. The multiplication of lines in a 64-bit implementation and the additional real estate of the 6U formatof courseadd substantially to board and system cost.
PCI Expressby contrastbegan life beyond the GHz frontier at 2.5GHzand 5GHz is now on the horizon. At 2.5GHzthe minimal PCI Express configuration – a x1 (‘by one’) link consisting of a single ‘lane’ – supports an equivalent unidirectional data transfer rate of 250Mbytes/s that equates to an impressive bidirectional rate (in full-duplex mode) of 500Mbytes/s. (A PCI Express lane contains two sets of differential pair wiring: one set for input [receive] and one set for output [transmit].)
A x4 configuration boosts that to 1 Gbyte/second
and 2 Gbytes/second respectively with larger lane counts bringing yet greater rates – to a maximum of 4 and 8Gbytes/secondfor unidirectional and bidirectional communication using a x16 implementation at the initial PCI Express frequency of 2.5GHz. When considering performance in terms of bandwidth/linethe
point-to-point edge over buses becomes quite dramatic.
Architectural ramifications
The transition from buses to point-to-point links also has architectural ramificationswhich themselves affect the performance picture. A bus is a shared resourcemanaged by a bus arbiterwhich only one device can make use of at one time. In a switched fabricmade up of individual point-to-point linksmany of these links may operate simultaneouslycreating an additive bandwidth effect. Ifsayfour x4 links in a switched fabric are all active at the same timethat adds up to 4 and 8Gbytes/second (unidirectional/bidirectional) in aggregate bandwidth.
Fabrics based on switches and point-to-point links are also far more extensible than busessupporting an essentially unlimited number of devices. PCI buses typically support only four boards at maximumand CompactPCI extends that to just eight. Granteda PCI-based bus can accommodate larger numbers of boards by combining multiple bus ‘segments’ through the use of bridgesbut this can quickly become excessively complex and unwieldynot to mention heavy in latency and costly. In fairnessit must be mentioned that fabrics also create their own complexities and latencies with issues such as packet routingdealing with out-of-order packetsetc.
In light of the high and extensible bandwidth provided by wedding PCI Express to the CompactPCI foundationCompactPCI Express marginalises the 6U performance advantage over the 3U Eurocard and greatly expands the performance range possible in the 3U form factor. The smaller 3U form factor not only has the edge in compactness and cost over 6Ubut it also provides a more rugged platform due to its smaller sizelower weightreduced inertiaand freedom of mechanical stiffeners. While it may be possible to design a 6U system with the same level of mechanical robustnessshock tolerance and vibration resistance as a 3U systemthis would require a considerable amount of post engineeringas well as substantial expense.
The inherent robustness of 3U CompactPCI hardware has been well proved under the most strenuous real-world conditionsand CompactPCI Express has expanded the application range for that hardware with its performance and architectural boost. By adopting techniques developed for enterprise-class serversembedded servers can exist with enhanced robustness and dramatically increased reliability.
The Intelligent Platform Management Interface (IPMI)championed by IntelHewlett-PackardNEC and Dellis a well established open software standard devoted to reducing the TCO (Total Cost of Ownership) of large IT infrastructures by optimising diagnostics and maintenance procedures. It provides the wherewithal to deliver the RAS (reliabilityavailabilityserviceability) trio of capabilities to enterprise systems for which downtime is too disruptive and expensive to be acceptable.
Current estimates place some 80 per cent of the cost of an embedded system in software related components and events. Clearlytime is money in high-end commercial applicationsbut RAS is at least as critical in mission-critical embedded applications in industrial controlmilitarymedicaltransportation and other environments. Where downtime has enormous repercussionsIPMI benefits such as pre-boot diagnostics and operating system self-repair will be most welcomenot just in new CompactPCI Express systemsbut in traditional CompactPCI systems as well.
The heart of the IPMI infrastructure is the so-called Baseboard Management Controller (BMC)an autonomous microcontroller to which the CPU and major on-board components are connected via a System Management Bus (SM-BUS). Based on a straightforward request/response protocolcommunications over this bus are conducted using a set of standardised messages between the BMC and various board components. An Intelligent Platform Management Bus (IPMB)in turnconnects the BMC to a so called Satellite Management Controller (SMC) and provides access so to all system components. When connected to an external server running a remote management consolethe BMC communicates using either an IPMI-capable Ethernet controller or a standard RS232 serial line.
Among other aspectsIPMI supports the concept of sensor data records (SDRs) and event logsdynamically logging such parameters as voltagecurrent and temperature and maintaining the information in non-volatile memory. This enables proactive identification of potential hardware problemsallowing preventative adjustments to be made to operating parameters when possibleand making system operation more predictable.
Gilding the lily
Although IPMI does not require that changes be made to a system’s BIOSdoing so can significantly enhance the remote management of a system and its auto-recovery capabilities. An IPMI-aware BIOSfor exampleis capable of sending alert messages during the boot process to inform the remote management console about any malfunctions.
It is also possible to send IPMI messages from the remote management console to the BIOS to remotely control the boot process or to change BIOS settings. Application code canfurtherbe remotely updated without operating system involvement. The common system malfunction scenario of a hard disk image crash can be easily recovered remotely by commanding the IPMI-compliant BIOS to boot a recovery image from a medium such as a CD or a link such as RS232a
local-area network (LAN) connection or the Universal Serial Bus (USB).
The aforementioned IPMI capabilities are further enhanced by integrating a bootable µLinux operating system kernel into the flash BIOS of the CPU board. Such a kernel can be utilised to expand remote diagnostic and maintenance capabilities by incorporatingfor examplecomprehensive test functionsrich network functionality (including access to Windows- and UNIX- based servers) and support for various file systems in order to handle disk drive image repairs and updates.
Moreoverfor some applicationsa µLinux kernel may also be used as the main operating systemproviding all the necessary driver modules for supporting on-board system componentsnetworking and diagnostic tasks. A built-in kernel provides the ultimate in rapid system booting (3 seconds)and it reduces TCO by doing away with costly operating system licenses. Furtherby integrating the operating system in robustnon-wearing and vibration-resistant flashthe system is immunised against the effects associated with conventional rotating-disk mass storage devices in extreme environmental conditionssuch as high humidity and/or temperatureor under severe shock and vibration conditions.
Within the embedded system lifecyclesome 60 per cent of the operating costs are typically reserved for regular maintenance tasks. Nowthanks to the IPMI and µLinux kernelthese maintenance tasks can be fulfilled automatically in a transparent mannerthereby streamlining system management and greatly improving the system up-time.
By defining a backplane environment where CompactPCI and CompactPCI Express boards coexistthe PICMG EXP.0 R1.0 standard simplifies the integration of the old and the new through a flexible backplane conceptwhile easing the transition between buses and point-to-point links. Relatively undemanding functions will continue to make use of the CompactPCI busfor examplewith boards residing in CompactPCI Express legacy slots. If and whenever a bandwidth boost is requiredon the other handa CompactPCI board in a hybrid slot can be swapped out for an updatedCompactPCI Express versionwithout backplane or software change.
It is these scalable high-bandwidthserial interconnection links that solve the bottleneck problems associated with previous (parallel) bus standards such as CompactPCI (cPCI). With the new high-speed serial linksapplications reserved typically for 6U boards and systems can be realised in the more compact and robust 3U form factor. The many advantages the smaller 3U form factor has to offersuch as mechanical robustness and stabilitytolerance to shock and vibrationcompactnessand economic designpermit the creation of optimised solutions.
Andrew Brown is with INOVA Computers GmbHKaufbeurenGermany. www.inova-computers.de
"