When fortunes can be made or lost in a microsecond, network testing demands unprecedented levels of precision. ”It can be done”,
says Daryl Cornelius, Spirent Communications

Innovation disperses ever faster – and the breakthrough business strategy or technology that yesterday gave you the competitive edge, today lands you square in midfield. For the financial trader that means relying on similar trading strategies used by all the competition. So one lucky trader gets the optimum price and, a fraction of a second later, you’re the loser.

High Frequency Trading (HFT) generates massive trading volumes – an attractive proposition for financial exchanges – but puts enormous pressure on the exchange to force down latency, while maintaining reliability and market integrity. At such speeds and volumes a new challenge has emerged, called the ”microburst”. A microburst is a very short duration spike in traffic volume, that can pass undetected – both because it gets lost in the per second average traffic statistics (see Fig 1, showing max, min and average traffic plots), and also because it can be faster than the circuit design rate. Even at that speed it can result in lost packets and, for high speed traders, that’s a nightmare – especially because microbursts are most likely to occur in times of peak volume and volatility, when trading is most critical.

Fig 1 – Microbursts lost among average traffic

The industry is already addressing these pressures via technological upgrades, such as bigger buffers, extra processing capacity and the use of faster chips to meet not only normal market conditions but also freak events like the May 6th 2010 “flash crash”.

This is a very necessary process, but it will miss the mark unless network test procedures can move ahead of the actual operating conditions. It is comforting to know that your system has exceptionally low latency, but that is nothing compared to the value of metrics that not only quantify latency but also provide a detailed analysis of variations under a whole range of normal and exceptional operating conditions. To achieve this, you need a comb with finer teeth than the shortest microburst hazard: sophisticated monitoring capabilities that sample, inspect and time stamp traffic so precisely that nothing capable of damaging a trade can pass undetected. To really drill down to bottlenecks in the system, it is not even enough to measure round trip latency to this accuracy, you need to measure one-way latency with levels of precision previously unattainable by the finest round trip measurements.

The good news is that all this is now possible thanks to a new partnership between Spirent Communications – a world leader in subjecting systems to exhaustive ”real world” testing and cPacket, whose technology can inspect and classify data packets according to payload and header, at multi-Gbps speeds.

The resulting partnership established market leadership by launching a family of Traffic Access devices delivering functionality previously thought to be impossible, but now needed by the finance industry and increasingly by a wider number of organisations needing to analyse fast-moving data, such as real-time risk or performance management.

The chip that rewrites the rules

Software solutions already existed for ”deep packet inspection” – examining a packet’s content bit by bit – a vital function in network monitoring and secuity applications. Software performance, however, is inadequate because it adds latency and cannot keep up with today’s 10 Gbps or even 1Gbps line rates. cPacket’s purpose-built hardware solution, however, does the job on the fly and in real time. The time each packet takes to pass through the system is deterministic and independent of the content. Effectively, it performs like a streamlined ”bump in the wire” with no slow software processing in the data path or external memory bottlenecks. Unlike other deep packet inspection co-processors, this unique chip performs not only pattern matching but also packet header parsing in a single pass on the fly – hence the term Complete Packet Inspection (CPI) used by cPacket.

Consuming a mere five watts, the chip delivers about ten times the performance at one tenth of the cost of slower, more complex and expensive alternatives. Whatever the packet protocol The chip’s pattern-matching algorithms inspect traffic profiles defined by a simple user interface. The chip examines the stream bit-by-bit and can count, tag, redirect, replicate, or drop traffic according to user-specified criteria. Users can specify complex traffic profiles without worrying about such low-level protocol details as chained virtual local-area networks, or case-insensitive pattern searches. The unique algorithm enables linear scaling of chip performance to support 40 Gbps.

cPacket’s CPI opens up new possibilities for the integration of traffic monitoring, network security, test, and lawful intercept for a new generation of intelligent switches and network devices. Spirent have grasped this opportunity with their family of Traffic Access devices.

Spirent Traffic Access devices are designed to inspect network traffic on the fly. This replaces the time consuming process of recording the entire data stream to disk and then post-processing the recorded data for critical data inspection and pattern matching. The devices incorporate cPacket’s hardware and software architecture and come in a range of sizes – currently 12, 24 or 32 ports – handling any combination of 10G or 1G Ethernet. CPI allows the data stream to be searched for any user-specified combination of header fields and payload content and, based on the analysis, the packet can be switched to one or several ports. Packets can also be precisely time-stamped with GPS clock synchronisation. Detailed performance metrics are provided in a graphical web dashboard as well as being stored in standard CSV files that can be imported into SQL databases, spreadsheets, and monitoring frameworks.

 
Fig 2 above shows one example of a user-defined filter, combining both header and payload inspection. Shown below, Fig 3, is a sample graphical report of bandwidth in or out of the device as well as network behavior information such as TCP events, frame size distribution, and protocol breakdown. New filters are easily added to the reports from a simple and easy to use user interface.The data is also available as tables showing the instantaneous traffic rates , statistics such as minimum, maximum, mean and standard deviation over the 60 seconds running window, as well as cumulative values.

The potential range of benefits and uses is enormous, so we start with a few examples. Take the transition to higher speed Ethernet – few companies can afford a forklift upgrade, invariably there is considerable investment in legacy 1G equipment that does not support 10G. So a 10G data stream can be split into 10x1G streams with a single Traffic Access device and multiplexed back to 10G as needed.

Service providers can also use the device for multiplexing and de-multiplexing, or they can use it as a powerful laboratory tool for testing networks, applications and services. As such it can be used for streaming data to multiple devices simultaneously to take full advantage of the diagnostics against the selected data set, or one can apply specific traffic filtering & forwarding capability to send different data sets to each tool for selective analysis. Considerable time is saved because of the ability to isolate the relevant data, filter it, and perform multiple operations at the same time.

In addition, service providers can compare data sets being modeled in the lab with real live traffic traversing their network. This is an application close to Spirent’s heart because customers all across the globe use Spirent equipment and services to emulate real world traffic conditions to create stringent capacity tests on their networks and systems. Spirent Avalanche allows one to create traffic patterns to specification, or simply to record actual traffic and use that as the template for a test deluge. So, how much traffic must one record to get a usable ”average”? An hour? a day? a week? It could amount to a massive storage burden. Instead one can simply run the traffic through the Traffic Access device to profile the network behavior and feed the relevant metrics in to Spirent Avalanche traffic generation templates.

In the live network, network managers now have the ability to replicate traffic to multiple network tools, to aggregate traffic to a single tool, and to selectively filter traffic for enhanced security or application performance monitoring. Being able to profile and isolate traffic patterns can help to identify weaknesses or bottlenecks that hamper optimal network performance.

Keeping tabs on financial services
That above example, where a rolling average provides all the data needed without the need for massive data storage, points the way to a very important potential benefit in areas such as financial services where accountability demands an archive of transactions going back over a long period.

The first question is: do all transactions need to be recorded, or is that simply a fall-back position? The Traffic Access device would allow one to define precisely what traffic needs to be archived and what can be discarded. In many cases, this reduces the total storage burden by a factor of ten or more.

Even more useful is the opportunity to filter and sort traffic on the fly. So that, for example, all traffic involving one client – or a specific type of transaction – could be routed to a distinct archive. It is all being archived as before but, instead of one massive multi-terabit archive, it is neatly filed into chosen subsets of data, making subsequent inspection or data mining hugely faster and more efficient.

If packets get lost during a microburst you can lose revenue and also key applications such as algorithmic trading engines can receive stale data and need to be re-synced, adding further delay. If you cannot measure these spikes, you cannot fix them, but detecting microbursts requires sampling traffic in extremely fine time increments – see Fig 1. In that respect, the speed of cPacket’s chip is way ahead of the current requirements.

The precision time stamping function is another vital asset for the financial community, where timing to the sub-microsecond can make or break a deal. If two clients claim the same purchase at the same instant, how can one determine which was first without an accurate time stamp? A sale cannot be made unless the goods have already been purchased and cleared – these truths are so obvious, but with online trading they can be annulled on a minute delay. Application protocols record times to the microsecond, but cPacket’s precision time stamps are accurate to 30 nanoseconds.

In Fig 4 we see precision time stamps recorded before and after the firewall: the difference between the transaction time and the time of arrival at the firewall provides a measure of the public network latency, while the difference between the time of entering the internal network and of arriving at the transaction server, is a measure of internal network latency. This can also record the latency of the firewall, and the total internal processing time (the difference between confirmation arrival time and order arrival time).

Fig 4 – Multiple time stamps for complete and segmented latency measurements

The accountability demands of banking and financial services makes the finance industry an obvious example, but the same principle would apply to many large corporations that require transactions to be archived for review or later audit. In terms of efficiency, such pre-storage filtering makes the difference between searching through a ceiling-high mountain of unsorted paperwork, and searching a well-maintained filing system.

Compared with on-line financial trading, ordering the week’s shopping from a supermarket website might seem a very leisurely process – until the instant when the purchase button is clicked. If two customers are in contention for the last item in stock, then the precise moment of the transaction becomes a critical issue, just as it would be for the financial trader.

Security and data surveillance
Complete packet inspection at 10G Ethernet speeds, on the fly and in real-time facilitates advanced filtering and intrusion detection and prevention. Denial of service attacks can be instantly detected, code strings associated with malicious software can be detected and quarantined without impacting legitimate traffic performance. The payload searches accommodate both anchored and unanchored strings of data with wildcards included, amounting for a powerful and sophisticated search criteria.

More generally, the ability to identify and report detailed traffic patterns over a short and longer timescales can be used for behavioural detection of suspicious annomalies.

Conclusion
It is not often that such a powerful and broadly applicable new technology as the cPacket hardware-software and chip becomes available, and it can take time to wake up to the full potential it offers.

Traffic Access has a clear role to play as part of our test and measurement offering with a focus on increasing the speed and sophistication of Ethernet deployments. Flexible traffic access devices provide greater efficiencies in use of equipment and storage. However, as we have seen, the applications of traffic access devices goes way beyond pure testing.

When it comes to hair-trigger data processing, the finance industry is ahead of the game. But innovation spreads fast, and an increasing range of organisations will be needing our test solution in order to gain and maintain the confidence that only high precision test and measurement can ensure.