NTT DATA CORPORATION
Almost any analytic logic can be implemented, without doing any programming, by using the Sybase Aleri Streaming Platform during the development stage. High development-productivity like this is one of the strongest features that Sybase products offer.
Assistant Manager, NTT DATA
NTT DATA CORPORATION, based in Japan, is a global provider of IT services including business consulting, technology, and outsourcing services. With expertise cultivated from years of experience, NTT DATA takes a proactive approach to solutions that enable real-time data analysis that leads to greater innovation capabilities for customers.
- Enhances ability of applications to collect and analyze data generated at rapid rates by thousands of sources.
- Enables complex event processing across multiple data sources
- Provides high-performance real-time data analysis
- Allows for high-productivity in analysis logic development
- Offers compatibility with external math libraries
- Systems Integrator
Enhancing Bridge Monitoring with Complex Event Processing Technology
NTT DATA continually strives to provide services beyond what other IT service firms can offer.
A prime example of this mission is a product that the company resells, deploys and supports—
the BRIMOS, bridge monitoring system. The solution remotely monitors bridges in real time to provide valuable information for managing responses to natural disasters, maintaining bridge structures, and estimating the extent of structural fatigue.
BRIMOS consists of various sensors, high-speed data transmission networks, and an information center that collects and disseminates high-level information about bridges. The solution relies on multiple optical-fiber sensors installed on highway overpasses to detect anomalies or damage by continuously collecting and analyzing a variety of data about level differences, spacing, vibration, tilting, and other factors in bridge beams and bridge piers.
By extrapolating vehicle weight from warping data, the system also obtains weight information about passing vehicles, one of the primary factors behind damage to highway overpasses. In addition, BRIMOS collects weather information, and the comprehensive data collected plays an important role in checking the safety of highway overpasses after an earthquake.
To enhance the ability of BRIMOS to analyze large, complex data sets collected at rapid rates, particularly during high-traffic time periods, NTT DATA decided to conduct a proof-of-concept test to determine the feasibility of integrating BRIMOS with complex event processing (CEP) database technology. In addition to performing regular conversion of sensor data, the POC tests would analyze anomaly-detection processing for highway overpasses based on an analysis of numerous elements including real-time data and stored data.
Eight-Month Proof-of-Concept Proves Effectiveness of Sybase Technology
Over the course of the eight-month POC, which involved testing multiple CEP platforms, a combination of Sybase technologies proved to offer the most effective solution. This includes Sybase’s rapid data-analysis platform, Sybase RAP - The Trading Edition, and the real-time data-analysis CEP engine, Sybase Aleri Streaming Platform.
“The main goal of the proof-of-concept was to determine the processing speed (latency) of each platform,” explains Yoko Inaba, an Assistant Manager at NTT DATA. “To take advantage of CEP’s ability to simultaneously process multiple data sets, we built a system for real-time anomaly detection. We also decided to analyze the level of performance required to collect and manage data over long time periods.”
With anomaly detection, measurements are converted based on real-time data and data collected in the past from the multiple optical sensors. The amount of incline, strain, and other factors are then determined, and an anomaly is deemed to have occurred if an amount exceeds an established threshold. Because data comes rapidly from a variety of sources, including past data, uniformity between the data must be maintained.
Sybase CEP Technologies Process Data From More Than 10,000 Optical Sensors
During the POC, NTT DATA conducted a variety of tests on Sybase RAP and Sybase Aleri using a server mounting Intel® Xeon® Processor E7-4860 (2.26GHz) 4-CPU/40-core. Tests included an aggregation test to determine how many pieces of sensor data could be processed, a latency test to determine processing speed, an external math libraries compatibility test, and a test of long-term data handling.
The Sybase technologies earned positive marks across all categories while collecting data from more than 10,000 sensors at expected speeds. The Sybase technologies also demonstrated dynamic model-switching capabilities as well as the ability to facilitate software development and integration.
The results of the Sybase POC far exceeded NTT DATA expectations—with data-collection tests showing that sensor data from more than 10,000 sensors could be processed by a single server. Processing even cleared the latency target hurdle of eight milliseconds. The tests also established the high level of compatibility between Sybase products and math libraries, which is necessary to achieve advanced signal processing.
Based on the results of the proof-of-concept, Masatoshi Yokogawa, a Senior IT Expert at NTT DATA says, “For anomaly-detection processing, the detection threshold needs to be changed dynamically according to the situation. To an extent, long-term trends need to be taken into account to calculate this threshold. With Sybase RAP, analysis models can be dynamically changed according to the real-time data while also grasping long-term trends through batch processing and data collection. This function is very desirable for anomaly-detection processing, which would normally take a long time to implement.”
Run-time performance was not the only test receiving good marks. “Almost any analytic logic can be implemented, without doing any programming, by using the Sybase Aleri Streaming Platform during the development stage,” Inaba says. “High development-productivity like this is one of the strongest features that Sybase products offer.”
Project Success Leads to New Uses of Sybase CEP Technology
Although the objective of this POC focused on BRIMOS integration, the ultimate goal for NTT DATA was to test the use of CEP for a broad range of applications. “We researched many CEP applications to find the one that would be the best match for our many specialty-software solutions,” says Yokogawa. “This proof-of-concept proved to us that Sybase RAP and Sybase Aleri will be very effective in applications for any of our sensor networks. This will be very important as we expand our business going forward.”
Thanks to the positive results of the proof-of-concept, NTT DATA is now developing a large-scale real-time data analysis solution that will manage real-time data as well as long-term data. The initial release of this new solution is tentatively scheduled for 2012. Sybase Aleri (CEP) will be used to handle real-time data processing, the Sybase RAP in-memory function will handle processing of chronological data of up to a few hours, while the store function will manage data for up to several months. Hadoop will be used to handle really huge volumes of data spanning years.
“The comprehensive and flexible configuration that Sybase technology allows lets us create a hybrid infrastructure according to the amount of data and latency needed for analysis,” Yokogawa says. “Because of this flexibility and the performance of the technologies, we expect Sybase RAP and Sybase Aleri to play major roles in enhancing the capabilities of our applications.”