Alexis Kestler
Amazon Glue is quickly gaining popularity as more and more companies start leveraging managed data integration services. ETL (extract, transform, load) is the process of moving data from source databases into a data warehouse and is complex and difficult to implement across enterprise data because of its complexity. To address this problem, Amazon introduced AWS Glue, a fully managed ETL service. ETL developers and data engineers use Glue to build, run, and monitor scalable ETL jobs.
What is AWS Glue?
AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development.
Glue provides a simple and flexible way to quickly integrate data from disparate sources, clean and validate it, and make it available for analytics and machine learning workloads.
Key benefits include:
- Serverless service – no infrastructure to setup or manage
- Automated crawling of data sources to discover schemas and track metadata
- Auto-generated PySpark or Scala code for ETL jobs
- Visual job monitoring and logging
- Integration with AWS services for analytics and machine learning
- Pay only for resources used
In short, AWS Glue simplifies and automates complex data integration tasks like data discovery, cleansing, mapping, and job scheduling. Developers can focus on business logic rather than infrastructure management.
What is AWS Glue Used For?
There are several common use cases where AWS Glue provides value:
Query Data Lakes
Glue Crawlers can crawl Amazon S3 data lakes and track metadata in the AWS Glue Data Catalog. This metadata enables serverless queries with Athena without moving data around.
Understand Data Assets
The centralized Data Catalog provides a consistent view of data sources across services like S3, RDS, and Redshift. You can easily search for datasets and understand what is available.
Data Transformation
Develop ETL jobs with Glue to clean, format, validate, and transform data for analytics. Integration with Apache Spark provides scale when needed.
Data Integration
Use Glue workflows to integrate and consolidate data from diverse sources like databases, object stores, data streams, and software as a service (SaaS) applications.
Event-driven ETL
Triggers can invoke Glue ETL jobs based on events like new data landing in S3. This allows building reactive workflows.
Machine Learning
Clean, validate, and prepare data for machine learning with Glue ETL jobs. The processed datasets can then be readily consumed for model training.
In summary, Glue provides a flexible, scalable, and serverless platform to solve many common data integration challenges required for analytics, business intelligence, and machine learning.
Components of AWS Glue
AWS Glue is comprised of several integrated components:
Data Catalog
This managed metadata repository stores structural and operational metadata for data sources. Integration with crawlers allows tracking schemas, relationships, and more.
Databases and Tables
Relational view of data that can be queried using standard SQL. Integrates with various data sources.
Crawlers
Discover data, extract schemas, and populate the Data Catalog. Crawlers connect to sources like S3, RDS, and JDBC.
Classifiers
Apply custom rules to classify and categorize data discovered by crawlers based on formats, schemas, or custom logic.
ETL Jobs
Scalable Spark jobs defined in Python or Scala to extract, transform, and load data. Jobs can be triggered on demand or on schedule.
Development Endpoints
Managed Apache Zeppelin notebooks for iterative job development. Supports Python, Scala, and Spark SQL.
These components work together to provide a full-featured environment for diverse data integration tasks.
Benefits of AWS Glue
Key benefits of using AWS Glue include:
- Serverless service with auto-scaling execution
- Integration across AWS services like S3, RDS, Redshift
- Flexible scheduling for ETL jobs
- Visual monitoring and logging for ETL jobs
- Push-down optimization for data stores
- Automatic schema inference and Evolution
- Pay only for resources used
By leveraging AWS Glue, organizations can focus on the business logic for data integration rather than managing underlying infrastructure. The serverless ETL engine, job scheduler, and auto-scaling make Glue a highly reliable and cost-efficient choice.
Top Features of AWS Glue
Here are some of the standout capabilities AWS Glue provides:
Crawlers
Discover data and track structural metadata across a variety of sources. Crawlers integrate with the Data Catalog.
Auto-generated ETL Scripts
Get Python or Scala code generated automatically for data transforms. Just point Glue to sources and targets.
Flexible Scheduling
Schedule and run ETL jobs precisely when needed – hourly, daily, weekly or monthly.
Scalable Spark Environment
Apache Spark powers the distributed ETL engine. Auto-scaling ensures right resources.
Data Catalog
Central metadata store tracks data lineage, schemas, ownership, statistics. Enables data discovery.
Machine Learning
Inbuilt FindMatches ML transforms can deduplicate messy real-world data for analytics.
DataBrew
Visually explore, clean, and transform data without coding. 250+ pre-built transformations.
Monitoring
Track metrics and logs to monitor health, performance, and debug ETL jobs.
AWS Glue Pricing
AWS Glue is priced based on compute resources used for Crawlers and ETL jobs, plus Data Catalog storage and requests.
Key pricing dimensions:
-
Crawlers: per minute pricing, billed per second. Starts at $0.44 per hour.
-
ETL Jobs: per minute pricing for provisioned DPUs, billed per second. Starts at $0.44 per DPU hour.
-
Data Catalog: $1 per month for storage and access requests.
-
DataBrew: per job pricing starting at $0.48 per hour.
-
No minimum fees or upfront commitments.
For most use cases, Glue pricing is very cost effective compared to managing similar ETL infrastructure. The serverless consumption-based model ensures paying only for resources used.
Steps to Setup AWS Glue
Getting started with AWS Glue only takes a few steps:
1. Create an IAM Role for Glue Access
The AWS Glue service needs access to related resources like S3 buckets. An IAM role provides these permissions.
2. Add Data Sources
Connect to the data sources you want Glue to work with. Common options are S3 buckets, RDS databases, and Redshift clusters.
3. Run a Crawler
Crawlers catalog metadata from sources in the Glue Data Catalog. Analysis of schemas and formats.
4. Develop ETL Scripts
Use the auto-generated PySpark or Scala scripts as a starting point to transform your data.
5. Schedule ETL Jobs
Trigger your ETL jobs to run on demand or on a schedule using a cron expression.
Once these steps are complete, you have an operational serverless ETL pipeline! You can further build upon this with additional data sources, transformations, and job orchestration.
Hands-on Example: S3 to Athena
Let‘s walk through a simple example to see AWS Glue in action. We will:
- Set up an S3 bucket with sample data
- Crawl the data with an AWS Glue Crawler
- Query the resulting table in Athena
Step 1: Create a Sample S3 Bucket
First, we create an S3 bucket and upload a sample CSV file:
Step 2: Define an IAM Role for Glue
Next, create an IAM role Glue can assume to access the S3 bucket:
Step 3: Run a Crawler
With our bucket and role ready, we can now run a crawler to catalog the CSV data:
Give the crawler a name, specify the S3 path, choose the IAM role, and run it.
The crawler will analyze the CSV, infer schema and formats, and register the metadata in the Glue Catalog.
Step 4: Query in Athena
With data cataloged, we can immediately query it in Athena:
Athena integrates directly with the Glue Data Catalog allowing query access without moving data.
And that‘s it! In just a few clicks we have discovered, cataloged, and are querying S3 data with Glue and Athena. You can build upon this simple workflow to transform, enrich, move, and analyze data at scale.
Comparison to Alternative ETL Tools
There are many commercial and open-source tools that address various data integration challenges. Here we compare Glue to some popular alternatives:
Airflow
Airflow is an open-source workflow scheduler designed to programmatically author, schedule and monitor pipelines. Key strengths include:
- Open source with a robust community
- Define workflows as Python code
- Extensive library of operators
- Active development and maintenance
Airflow provides more coding flexibility compared to Glue‘s configuration-driven approach. Airflow is self-hosted, while Glue is a fully managed service.
Glue is better suited for developers who want a faster, serverless setup without infrastructure management overhead. Airflow appeals more to engineers wanting to code their data pipelines.
Matillion ETL
Matillion ETL is designed specifically for cloud data platforms like Snowflake, Redshift, and BigQuery. Key features:
- Intuitive browser-based UI
- Progressively generated SQL scripts
- Scheduling, monitoring, and alerts
- Broad range of integration connectors
Matillion ETL makes it very easy to load data into cloud platforms by auto-generating SQL scripts and providing connectors to many data sources. Limited orchestration capabilities compared to Glue.
Glue has deeper AWS integration, more enterprise features, and out-of-box machine learning capabilities like FindMatches. But Matillion lets you get started faster thanks to connect-and-load simplicity.
Stitch
Stitch is an extensible ELT service supporting over 100 sources and destinations. Core features:
- SaaS integrations like Salesforce, Marketo, etc.
- Replicate data to cloud warehouses
- Easy to configure UI workflows
- Scheduling, monitoring, and alerts
Stitch makes it simple to move data from common SaaS applications into data warehouses. Light on transform capabilities compared to Glue.
With Glue, you get more flexibility to handle messy raw data and build complex data pipelines going beyond basic ELT. But Stitch simplifies integrating data from SaaS apps.
Alteryx
Alteryx aims to empower data analysts with an intuitive workflow builder for self-service data integration. Key capabilities:
- Code-free workflow builder
- Connectors for files, databases, apps
- Workflow scheduling and monitoring
- Growing analytics and ML tools
Alteryx allows less technical users to rapidly build workflows with drag-and-drop simplicity. Broad range of connectors. Light on provisioning and orchestration features compared to Glue.
Glue provides greater scale, automation, and enterprise capabilities out of the box. But Alteryx enables analytics-focused data prep without extensive coding.
Conclusion
AWS Glue provides a fully managed, serverless platform for enterprise-grade, scalable data integration challenges. It automates and simplifies many complex ETL tasks like data discovery, metadata tracking, job scheduling, monitoring, and more.
Glue integrates seamlessly across various AWS services like S3, Athena, Redshift, EMR, and QuickSight. It continues to evolve capabilities for data preparation, cataloging, ETL, and machine learning.
For organizations running on AWS, Glue is certainly a top choice to evaluate to make data integration and processing easier, faster, and more productive.
