Handling Data Volumes, Business Demands Deftly

For more than a decade, IT organisations have strived to improve the performance and reliability of their enterprise data integration infrastructure, in the face of growing data volumes and business demands. They have used a variety of approaches that have proved time-consuming and costly. For instance, third-party data accelerators may speed performance, but they also introduce a new layer of complexity and administrative overhead. Manual development and tuning of load-balancing algorithms in a multi-node grid computing environment, may improve reliability, but they consume scarce IT staff resources and can be prone to errors. Improving data integration performance and reliability doesn’t need to be a large-scale and painful IT initiative.

The Need for Greater Data Integration Performance and Reliability

Your enterprise data integration infrastructure needs to provide the performance and reliability your IT organisation needs, to meet the demands of the business. Data integration technology is no longer merely a tool to update a data warehouse with a weekly batch load. Today, data integration technology lies at the center of mission-critical processes in every industry. Companies depend on data integration technology, to power data transformation and exchange, among core financial departments, customers, and supply chain operational systems.

More from Biztech

Future Group - Reliance Retail Deal approved by CCI RBI ban on cryptocurrencies takes effect; prohibition could force investors to tap the black market

Other companies operate near real-time enterprise data warehouses (EDWs) and operational data stores, designed to deliver timely and consistent information to decision makers, support staff, and external customers and partners. Data synchronisation, consolidation, and migration initiatives demand high performance and reliability to support 24x7 global operations.

As IT organisations strive to keep pace with growing business requirements, they face another challenge—growing data volumes. A June 2009 Forrester Research report says that in 2009, Forrester sees anecdotal evidence that approximately more than two-thirds of today’s EDWs contain between 1 TB and 10 TB of data. “Forrester estimates that by 2015, a majority of EDWs in large enterprises will be 100 TB or larger, with petabyte-scale EDWs becoming well-entrenched in such sectors as telecommunications, finance, and Web commerce,” Forrester’s report says.

The volatile economy has raised the stakes for data integration performance and reliability. In an uncertain climate, speed in reacting to changing market conditions is critical. Agility in meeting customer demands for value and service is paramount as consumers curtail spending.

Performance and reliability are vital to meeting these objectives—and to cope with surging growth in data volumes and complexity. A high-performing and reliable data integration technology is one that handles:

• Any data volume: The explosion in data volumes is forcing enterprises to address the performance and scalability of their data integration technologies. This issue will remain at the forefront as organisations accumulate petabytes of data generated, from their operations worldwide and struggle to leverage that data for insights and decision making.
• Any data type: As volumes grow, so does data complexity. The typical enterprise manages dozens or hundreds of mission-critical source systems in a heterogeneous environment. This means an overload of datatypes, structures, and formats, which can compromise performance and overwhelm ad hoc data integration solutions.
• Any data latency: IT organisations must manage a wide spectrum of time frames and latencies for data integration, depending on the application and use. The timing can range from weeks and days to seconds. IT organisations need the flexibility to deliver trusted, high-quality data whenever applications or users need it—whether in real-time, batch, or changed data capture (CDC).
• Any role: Many different people are involved in data integration projects. These people include data stewards, data analysts, architects, administrators, and developers. They have different skill sets and different tasks to accomplish. At the same time, they all need to work together to share artifacts and tasks, increase cross-team productivity, and ensure that IT results align with business needs.
• Any time: Data must be made highly available. Downtime in data integration processes can mean costly interruptions, customer dissatisfaction, and violation of service-level agreements. The recession has forced business and IT alike to scrutinise operations for inefficiencies and target affordable, low-risk investments with the potential for high payback and a tangible impact on business fortunes. Amid volatile economic conditions, business leaders are looking to IT to help manage initiatives that enable them to weather the challenging times and to lay a foundation for long-term prosperity. Data is at the core of many of these business initiatives.

A high degree of data integration performance and reliability can help IT organisations enable their companies to:

• Manage risk and compliance
• Improve operational efficiencies
• Streamline and quickly realise value from mergers and acquisitions
• Modernise the business
• Strengthen customer relationships

Optimising Data Integration Performance and Reliability

Data integration performance is usually defined in terms of throughput and scalability:

• Throughput measures show how rapidly a given quantity of data can be processed—for instance, reading, transforming, and loading 10 GB of data in x number of minutes.
• Scalability refers to the capacity to accelerate throughput, and/or handle greater data volumes, by modifying the software and hardware environment in which data integration processes are executed.

Improving data integration performance is not always straightforward. Your IT organisation needs to consider use, datatype and complexity, partitioning options, sources and targets, hardware and network infrastructure, and other factors. Rarely do you find a one-size-fits-all solution to accelerating throughput and building scalability.

To a lesser extent, improving the reliability of a data integration infrastructure also depends on variables. In some cases, simple failover of data integration processes to a secondary compute node may be sufficient. In certain mission-critical uses, IT organisations may be inclined to build in more advanced capabilities for resilience and recovery.

An array of features and options available with the perfect solution gives your IT organisation flexibility in strategically implementing performance and reliability enhancements that zero in on your most pressing business and IT needs.

Whatever course you chose, you don’t need to rip and replace your existing systems. The right solution’s features and options are engineered to complement your existing environment and help your organisation improve the use of all its IT resources—servers, data integration and other enterprise software, and the IT personnel who run your data infrastructure. These technologies include:

• 64-bit architecture
• High availability
• Grid computing
• Pushdown optimisation
• Partitioning

In time, the economy will stabilise. Economic growth will resume as it was in the past. But there’s no end in sight to the growth of your data volumes. To help your IT organisation continue to meet business demands and SLAs, it needs superior performance and reliability of data integration processes. Think a few years down the line. Or 5 or 10 years—how much data will your organisation have? How important will leveraging that data rapidly and reliably be to achieving your business objectives? What new business requirements will your organisation face that demand timely, trusted, and consistent data?

Superior data integration performance and reliability are distinct strategic advantages that bolster chances for both short-term survival in uncertain economic times and long-term prosperity in the years to come.

The author is MD, Informatica, South Asia.