Blog Details

Choosing the Right Approach for Data Integration

Posted by Marbenz Antonio on August 16, 2022

How do you approach data integration?

According to a recent survey of enterprise and solution architects conducted by The Open Group, 62% of organizations are currently using or intend to use a particular data integration approach like data virtualization, data fabric, or data mesh. However, this is usually done on a case-by-case basis. How would you choose the best strategy for your company?

It’s not ideal to respond to questions like this “off the top of your head.” You want to know how the different strategies would function in your unique case. You wish to read case studies and explanations for them. You should speak with others who are experiencing the same issues to learn what they are doing or have done and how it relates to what you wish to do. You want to be able to keep to best practices and standards.

We have a clear understanding of the state of data integration in businesses today, as well as the challenges encountered by enterprise and solution architects, thanks to the Open Group survey. It serves as a foundation for the creation of best practices and standards that will direct architects in the future.

Data Integration’s Current Situation

Initially, a few members of The Open Group Architecture Forum filled out the survey. Members of the Association of Enterprise Architects then finished it after making slight changes in response to comments. There were almost 600 answers in all.

Many of the characteristics of the picture painted by the survey will likely be familiar to you. The majority of business leaders see data as a strategic corporate asset, yet business unit usage of data is usually specialized. Some data is on-premises and some are on the cloud. Overall, there are varying degrees of great, poor, and average data quality. There are usually “quality” data islands with varying management practices.

Respondents identified several improvement points:

• Governance and Stewardship (47%)
• Accelerating speed of discovery and delivery of data – e.g. DataOps (20%)
• Creating a data platform (18%)
• Self Service (7%)
• Systematically protecting data (3%)
• Culture, data and content modelling, silos, technical capabilities, and understanding value were also mentioned.

The data that needs to be merged originates from corporate departments and business lines. Most of it is in databases, but it’s also frequently in electronic documents and occasionally comes from real-time sensors or social media. The chief information officer (CIO) and business analysts, sometimes business departments, define the information requirements. Some data have quality qualities indicated, but not all of them. The information could include personally identifying data (PII).

17% of respondents had point-to-point interfaces between applications and services internally and externally, 16% had a line of business data silos, and 36% had a mix of these. About 29% of respondents had a corporate integrated information-sharing environment, such as data warehouses, data lakes, or archives. However, 62% of those respondents worked for companies that were already employing or intended to utilize a specific method of data integration, such as data virtualization (37%), data fabric (27%), or data mesh (23%).

Pain points

One open-ended question in the poll asked participants to list their top issues and concerns with data integration. The responses can be divided into five categories.

1. Lack of commitment from business units – The refusal of departments to disclose their data. They are unaware of the benefits to business from doing so. Finding the necessary data is challenging, and getting subject-matter specialists to explain it is challenging as well.
2.  Lack of commitment at corporate level – Enterprise data integration is not seen as an investment-worthy business project.
3. Different sources and toolboxes – Different data platforms, web services with various languages and operating systems, SaaS providers with different interfaces, and described as parts with different processing requirements all exist.
4. Conflicting data models – Enterprise data models are usually absent. The information is not uniform. Data from both old and new systems are included. It incorporates outside data that is either at odds with internal data or ontologically and taxonomically non-normalized.
5. There is no data management culture – There is usually no data governance task group and no data-specific policies. There are problems with data quality, such as duplicate records and conflicting data from several sources.

Here at CourseMonster, we know how hard it may be to find the right time and funds for training. We provide effective training programs that enable you to select the training option that best meets the demands of your company.

For more information, please get in touch with one of our course advisers today or contact us at training@coursemonster.com