What is Data Virtualization and Why Do We Need It?

In today‘s data-driven world, organizations are accumulating vast amounts of data from a myriad of different systems and sources. This data resides in multiple silos spread across the enterprise infrastructure. Making sense of this disparate data to gain valuable insights is hugely challenging without a way to integrate and deliver unified views of the vital information locked away in these silos. This is where data virtualization comes into the picture.

The Growing Problem of Data Silos

The data landscape at most organizations today resembles a fragmented archipelago, with data spread out in isolated silos disconnected from each other. Some eye-opening statistics about the state of data silos:

• Up to 73% of all enterprise data remains isolated in silos, unavailable for business insights (Source)

• 91% of organizations struggle with cross-departmental data access (Source)

• Only 32% of companies have an integrated data architecture to break down data silos (Source)

As you can see, data silos remain an immense headache preventing organizations from unlocking the full value of their data assets. This data is often duplicated across systems, rendering it difficult to get accurate views. Managing and making sense of such fragmented information is hugely challenging.

What is Data Virtualization?

Data virtualization provides a flexible architecture to unlock unified access to vital enterprise data trapped in silos. It delivers a single, consolidated view of information from across disparate sources like databases and files, without requiring data replication.


The key to data virtualization is the creation of a virtual data layer that acts as a single access point to integrate and deliver data from multiple back-end systems like databases and applications. Rather than having to query each system separately, users and analytics tools can simply query the virtual layer.

This approach does not require moving data into a central repository, saving significant overhead of replicating and synchronizing data across systems. The virtualization engine handles mapping data from physical sources into virtual data services in real-time.

In effect, data virtualization provides a abstraction that decouples physical data from the logical consumption layer. The loose coupling enables greater agility to respond to evolving integration and reporting needs. New data sources can be quickly onboarded.

Key Capabilities of Data Virtualization

Mature data virtualization platforms today provide a robust set of capabilities:

Virtual data services – The ability to access, integrate and deliver data from heterogeneous sources is the foundation. SQL-based access across major database types is standard.

Data abstraction and loose coupling – The virtualization layer insulates applications and users from underlying systems. This minimizes the impact of changes to physical data schemas and objects.

Data federation and aggregation – Data from multiple sources can be aggregated in real-time without persistence. This enables unified views.

Transformation and enrichment – ETL-like data processing including cleansing, joins, aggregation etc. can be applied at the virtual layer to improve information quality.

Caching and performance optimization – In-memory caching, materialized views, and multi-dimensional cubes help accelerate data analysis involving large datasets.

Metadata management – Robust metadata management provides definitions, lineage, and dependency analysis across physical and virtual objects.

Security – Role-based access control, dynamic data masking, and encryption help protect data security without affecting sources.

Monitoring and management – Monitoring capabilities track data changes at source systems and handle propagation to ensure consistency at the virtual layer.

Connectivity – Standard connectors eliminate the need for custom coding to interface with major types of sources like RDBMSes, files, apps, cloud services, etc.

Key Benefits of Implementing Data Virtualization

Adopting data virtualization confers important benefits that drive value for organizations:

• Increased agility – New data sources can be added, or existing ones modified, without disrupting applications and users dependent on that data. Enables faster adaptation.

• Reduced costs – With data remaining in source systems, costs for storage, replication etc. are avoided. Less data movement means lower network bandwidth utilization and facilities spend.

• Greater scalability – It is easy to start small and scale up data access and integration capabilities by onboarding new data sources in a modular fashion.

• Increased productivity – Self-service access to integrated data cuts delays resulting from complex IT data integration processes. Accelerates analytics.

• Enhanced security – With data residing in existing secured systems, additional attack surfaces are not introduced. Fine-grained access controls improve security.

• Faster time to value – Rapid implementation compared to moving/consolidating data means faster ROI from business insights using the virtualized data.

• Simplified compliance – Compliance with regulations like GDPR is easier as sensitive data can be masked before virtualization and tracked to sources.

• Better data quality – Data quality issues can be addressed at the virtual layer rather than cleaning data physically from every source. Consolidated profiling.

According to IDC, organizations utilizing data virtualization enjoy 50-70% faster delivery of integrated data services while cutting costs by 48% on average. The business benefits clearly make this a compelling technology.

How Data Virtualization Architectures Work

There are a few different architectural approaches used to create the virtual data layer in enterprise data virtualization solutions:

Query-Based Architecture

This model relies on querying source systems on-demand when consumers request data from the virtual layer. The virtualization engine handles translating queries, dispatches requests to sources, and compiles the results.

Best suited for read-intensive workloads where real-time data is critical. Enables accessing diverse data on demand without duplication.


ETL-Based Architecture

In this model, data from sources is periodically extracted, transformed, and loaded into the virtual layer after some light processing. Consumers query the ETL-processed virtual layer.

Better suited for analyzing large data volumes with transform logic upfront. The ETL process improves consumption performance.


Hybrid Architecture

Combines aspects of both query-based and ETL-based models. Some data is sourced live from systems while other data is processed/stored in the virtual layer. Provides balance of real-time and performance.

This hybrid model allows architects to tune the virtualization architecture based on usage patterns and performance needs. For instance, frequently used reference data can be processed and optimized in the virtual layer while transactional data is accessed directly.

Data Virtualization in the Cloud

Modern data virtualization solutions are increasingly leveraging the elastic infrastructure of the cloud. Cloud-native implementations simplify deployment and management – users don‘t have to provision infrastructure.

Cloud data virtualization services like AWS Glue, Azure Virtual Data Factory, and Google Cloud Data Fusion eliminate overhead for organizations by providing fully managed environments. The hyper-scalable cloud infrastructure allows elastically scaling resources to meet data processing and querying needs.

Cloud data virtualization combines the benefits of cloud with the ability to integrate data across cloud and on-premise systems. Hybrid and multi-cloud architectures are supported.

Considerations for Implementing Data Virtualization

There are a few important considerations to factor in when implementing data virtualization in an enterprise:

• Upfront planning – Take time to thoroughly inventory data sources, map integration needs, identify optimal architectural approach etc. Data virtualization projects fail without upfront analysis and design.

• Phased rollout – Start with a targeted use case like self-service BI then expand systematically. Manage scope. Take lessons from each rollout phase to the next.

• Governance – Institute strong data governance practices across virtual and physical data. Define usage policies, ownership, standard definitions, meta model etc.

• Reusability – Design virtual objects (schemas, services, functions etc.) to support reuse. Avoid point-to-point mappings between sources and consumers.

• Scalability – Build with scalability in mind. Cloud-based data virtualization can provide easiest scalability. Plan to elastically scale resources per consumption patterns.

• Monitoring – Implement robust monitoring across sources, data processes, and consumption. Logs, alerts, dashboards etc. help manage performance and errors.

• Security – Secure the virtual layer and connectivity to sources. Isolate sensitive data through encryption and masking. Enable audits.

Getting these aspects right from the start prevents pitfalls and instability down the road as adoption grows.

Selecting the Right Data Virtualization Approach

Once the need for data virtualization has been established, organizations need to decide on the best solution for their needs:

• Commercial software – Products from vendors like Denodo, Tibco, Oracle, IBM provide full-featured enterprise-grade capabilities like connectors, caching, management etc. that can handle large-scale implementations. Licensed based on number of data sources or virtual services.

• Open source platforms – Open source solutions like Apache Drill, Presto, and Spark SQL offer good basic data virtualization features. Usually more hands-on to implement and tune. Customization can be complex but provides flexibility.

• Cloud data services – Cloud providers like AWS, Azure, and GCP offer fully-managed data virtualization services charged per usage. Quick and low overhead way to get started but can lack advanced features present in commercial products. Might lock you in.

• Custom implementation – Building custom virtual interfaces using application code or integration platforms like Mulesoft, Informatica, Talend etc. provides extreme flexibility but requires more effort. Useful for very specific custom needs only.

Make sure to match the solution capabilities to your current and expected future needs. Evaluating options is a key part of planning.

Real-World Examples of Data Virtualization Benefits

Data virtualization delivers quantifiable improvements in speed, productivity and decision making across sectors:

• Banking – BBVA achieved a 60% decrease in time to market for new customer offerings by enabling self-service access to data using virtualization.

• Retail – Lowe’s saw a 465% ROI within 6 months of implementing data virtualization for real-time analytics across 4,500 stores.

• Healthcare – The NHS provided clinicians self-service access to integrated health data, accelerating diagnosis and treatment.

• Government – The US Navy consolidated data across hundreds of ships into a virtual layer, gaining insights 33x faster.

The measurable improvements underline how data virtualization solves real business problems around utilizing data spread across siloed sources.

Emerging Data Virtualization Trends

As data volumes and sources continue exploding, new forms of data virtualization are emerging:

• Active data virtualization uses live queries to process data dynamically rather than storing predefined aggregates like traditional ETL. Enables flexible analyses.

• Real-time data virtualization streams live data from event streams and micro-batches to support real-time decision making rather than batch processing.

• Graph-based data virtualization leverages graph technology like Neo4J to map relationships across data for interactive analytics across silos.

• Cloud data virtualization simplifies managing data across on-premise and multi-cloud sources, enabling insights across them.

• Augmented data virtualization applies machine learning techniques to optimize data preparation, mapping, caching and delivery to enhance speed and quality.

These trends illustrate how data virtualization continues evolving alongside other bleeding edge data technologies to enable new ways of utilizing enterprise data.

Best Practices for Data Virtualization Success

Based on observed patterns in successful enterprise data virtualization initiatives, here are some best practices to boost your chances of a successful outcome:

• Begin by identifying high-value use cases that deliver quick wins and maximize business impact. Starting big often backfires.

• Engage business and IT stakeholders at design stage itself to get buy-in and incorporate diverse needs upfront.

• Start with a sharp focus on specific data domains and grow carefully from there. Avoid “boil the ocean” attempts.

• Utilize an iterative delivery model to build capabilities gradually rather than trying big bang implementations.

• Implement strong change management and communication processes as changes span technologies and teams.

• Build reusability into the way virtual components are designed right from the start.

• Put SLAs, performance monitoring and management dashboards in place to maintain and improve service quality.

There are certainly pitfalls to avoid like lack of planning, overscoped initiatives without milestones, and poor change management. But following proven strategies will help you maximize the likelihood of getting solid payback from your investment in data virtualization.

Data Virtualization vs. Data Visualization

Data virtualization and data visualization are related technologies often used together in data-driven organizations, but they serve quite distinct purposes:

Data Virtualization	Data Visualization
Enables access to and integration of data from multiple sources	Presents data in a graphical or visual format to help people understand and interpret the data
Involves creating a virtual view of data that can be accessed and queried without moving or copying the data	Involves selecting and transforming data to create charts, graphs, or other visualizations
Provides a virtual data layer or interface that can be accessed by users or applications	Produces graphical or visual outputs that can be viewed by people
Often used in scenarios where data is stored in multiple locations, formats, or systems or where it is not practical to consolidate the data physically	Often used to communicate complex insights, highlight key trends, or support executive decision making
Relies on technical software and tools to create the virtualization layer	Leverages visualization-focused tools like Tableau, Power BI, D3.js etc. to represent data graphically

Data virtualization focuses on enabling unified access to distributed data sources. In contrast, data visualization is about creating graphical representations of data that help human users interpret and understand the information better.

Together, they empower deriving sharper insights from enterprise data: data virtualization brings together scattered data, while visualization renders it intuitive. Many organizations implement the technologies hand-in-hand to unlock the full value of their information assets.

Conclusion

Data virtualization delivers a flexible, high performance architecture to overcome the bottleneck of disparate data locked away in siloed sources like databases and cloud apps. Without moving data, it creates a unified virtual data layer that delivers consolidated live access to information on demand.

Leading organizations leverage data virtualization to break down data silos and drive faster unified insights for decision making and innovation. With compelling benefits like increased agility, productivity and security, coupled with lower costs, smarter enterprises will continue adopting virtualization to maximize the value of data and analytics investments.