Recent Posts


Avoid Pitfalls in Data Management and Integration (ETL / ELT) in the Cloud
Posted on September 17, 2015
Author: Kirby Lunger, Performance Architects

One of the biggest differences between cloud-based and on premise enterprise software deployments is how data is integrated and stored, although this remains a mystery to many of us as this cartoon demonstrates:

Source: http://cloudtweaks.com

In the on premise software world, the data management and integration market is fairly mature and well-defined. Technology research firm Gartner defines this market as, “the practices, architectural techniques and tools for achieving consistent access to and delivery of data across the spectrum of data subject areas and data structure types in the enterprise, to meet the data consumption requirements of all applications and business processes.” This area is also referred to as “Extract, Transform and Load” or “Extract, Load and Transform” software (ETL or ELT; Performance Architects published a blog entry on the differences between the two terms here).

Gartner issues an annual “magic quadrant” for different software categories, including data management and integration. The 2015 magic quadrant demonstrates the key players haven’t changed in years:

Source: www.gartner.com

Over the last couple of years, business software has started to move – quickly – to the cloud (if you’re wondering what “The Cloud” is, visit this Performance Architects blog entry). As a result of this shift to software offered as a service, data management and integration has naturally had to evolve because the data in your solutions now lives outside your company’s four walls and the process simply needs to be handled differently.

As a result, a whole new class of solutions that Gartner calls “Integration Platform as a Service” (or iPaaS) has sprung up to address these differences. The category is defined here as, “a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.”

Interestingly, some of the market leaders are the same as in the traditional data management and integration market…but there are many new software vendors vying for leadership in this arena:

Source: www.gartner.com

So what does this mean for your cloud implementation? You need to make sure you understand what your cloud or “Software-as-a-Service” vendor is using for their data management and integration capabilities to avoid implementation pitfalls; there are several models for SaaS vendor offerings here, including:

• Provides application programming interfaces (APIs) or connectors to traditional data management and integration and/or iPaaS solutions. If this is the focus, make sure that the SaaS data integration options integrate well with your current ETL solution and that you understand how their product passes off data if there are one or more data integration solutions embedded in their product.

• “White labels” iPaaS solutions into their products. This means that the SaaS provider is licensing the iPaaS solution and is integrating it as part of their environment. In this case, you need to make sure you’re comfortable with the financial viability and functionality of the iPaaS provider they’ve selected and that you also have the conversation about how this works with your current ETL process.

• Uses flat files instead of a data management and integration or iPaaS solution. The majority of SaaS vendors are still in this camp, because the field is changing so rapidly. This isn’t optimal, but it works! Make sure you understand how to securely transfer data using these files into and out of their system, and if and how you can automate data import and export so that you’re not stuck executing data integration manual processes behind the scenes.

If you’re interested in learning more about how data management and integration works with cloud applications, we encourage you to sign up for our upcoming webinar, “How to Successfully Implement Oracle BI Cloud Service (BICS),” which includes a discussion of data integration techniques for this and related BI solutions.

We welcome questions and comments either in the forms below the blog entry, or email us directly at communications@performancearchitects.com to schedule a more detailed conversation.


© Performance Architects, Inc. and Performance Architects Blog, 2006 - present. Unauthorized use and/or duplication of this material without express and written permission from this blog's author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Performance Architects, Inc. and Performance Architects Blog with appropriate and specific direction to the original content.

Leave a Reply

Your email address will not be published. Required fields are marked *