AWS Glue is a fully managed ETL (Extract, Transform, Load) service that automates data preparation, schema discovery, and catalog management. It eliminates the infrastructure burden of building custom ETL pipelines while providing serverless scalability and cost-efficiency.
AWS Glue: Complete Overview
AWS Glue is a fully managed, serverless data integration service designed to make it easier to discover, prepare, and combine data for analytics, machine learning, and application development. Unlike traditional ETL tools that require infrastructure management, Glue abstracts complexity and charges only for job execution time. Core components: Glue Crawlers (automatic schema discovery), Data Catalog (centralized metadata repository), Glue Jobs (ETL transformation code), and Data Quality (built-in testing).
Glue Architecture and Components
Data Sources (S3, RDS, Redshift, DynamoDB)
↓
Glue Crawlers (auto-discover schema)
↓
Data Catalog (metadata repository)
↓
Glue Jobs (transform & load)
↓
Data Destinations (S3, Redshift, RDS)
Key Components:
– Crawlers: Scan data sources, infer schema automatically
– Data Catalog: Central metadata store (compatible with Athena)
– Glue Jobs: Serverless containers (batch + streaming)
– Glue Studio: Visual ETL workflow builder
– Data Quality: Built-in validation and monitoring
– Databrew: Visual data preparation (1000+ transformations)
Pricing Model:
– Crawlers: $0.44 per DPU-hour (auto-scale)
– Jobs: $0.44 per DPU-hour (1 DPU minimum, billing in 1-min increments)
– Data Catalog: First 1M objects free, then $1 per 100k objects
– No charge for metadata queried by Athena
Glue Crawlers: Automated Schema Discovery
Glue Crawlers automatically scan data sources (S3, RDS, Redshift) and infer schema, creating tables in the Data Catalog. Eliminates manual schema definition. Crawlers can detect data format changes and update schema automatically. Partition inference allows crawlers to detect partitioned data structures. Useful for detecting new columns or data type changes. Runs on a schedule or on-demand. Costs $0.44 per DPU-hour with auto-scaling.
Glue Data Catalog: Centralized Metadata
Data Catalog is a centralized repository storing metadata about data sources: tables, columns, data types, locations, and statistics. Fully compatible with Apache Hive metastore, enabling seamless integration with Spark, Athena, and Redshift. Enables data discovery across organization—analysts search for tables by name or tags. Versioning tracks schema changes over time. No additional cost for first 1 million objects. Critical for data governance and self-service analytics.
Glue Jobs: Serverless ETL Execution
Glue Jobs are serverless compute containers running PySpark, Scala, or Python transformations. Auto-scale based on workload from 2 to 100+ DPUs. Billing: $0.44 per DPU-hour, with 1-minute minimum. Ideal for batch processing (daily, hourly pipelines). Supports streaming (Kafka, Kinesis) for near real-time transformations. Handles schema evolution automatically through schema registry. Built-in job bookmarks track processed data, preventing re-processing.
5 Complete Interview Questions with Solutions
Question 1: Design Glue Crawler and Catalog for Multi-Source Data Lake
Implement Glue Crawlers for S3 (Parquet), RDS (PostgreSQL), Redshift tables. Design Data Catalog organization with tags, partitions, and schema versioning for 50 TB data.
# Crawler 1: S3 Parquet Data
name: s3-parquet-crawler
sources:
– s3://data-lake/raw/
schedule: daily at 2 AM
partition_keys: [year, month, day]
database: raw_data
# Crawler 2: RDS PostgreSQL
name: rds-postgres-crawler
sources:
– connection: postgres-prod
schedule: weekly (schema stable)
database: operational_data
# Crawler 3: Redshift
name: redshift-crawler
sources:
– connection: redshift-warehouse
schedule: weekly
database: warehouse_catalog
Catalog Organization:
Database: raw_data
└── Tables:
├── customers (source: RDS, last_crawl: 2024-06-15)
├── orders (source: S3, partitions: year/month/day)
└── events (source: Kafka, streaming)
Tags:
– pii: true (for customers)
– sensitivity: high (for financial data)
– owner: data-engineering
– sla: 99.9%
Cost Estimation:
– 3 Crawlers: 3 * 0.5 DPU-hour/day * $0.44 = $0.66/day = $240/year
– Catalog storage: 500k objects * $1/100k = $5/month
– Total monthly: ~$20
Explanation: Separate crawlers for each data source (stability differs). Partition-aware crawlers optimize S3 scanning. Tags enable data discovery and compliance. Low cost ($240/year) for comprehensive metadata.
AWS – Senior Data Architect
Question 2: Build Glue Job for Data Transformation
Create Glue Job: Load raw customer data from S3, transform (deduplicate, enrich geography, mask PII), partition by date, load to processed layer. Handle schema evolution.
from awsglue.transforms import *
from awsglue.job import Job
from pyspark.sql.functions import *
args = getResolvedOptions(sys.argv, [‘JOB_NAME’])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args[‘JOB_NAME’], args)
# Load from S3
input_dyf = glueContext.create_dynamic_frame.from_catalog(
database=”raw_data”,
table_name=”customers”
)
# Convert to Spark DF for complex transforms
df = input_dyf.toDF()
# Transform
df_transformed = df \
.dropDuplicates([“customer_id”]) \
.withColumn(“email”, regexp_replace(col(“email”), “(?<=.{2}).(?=[^@]*@)", "*")) \
.withColumn("processing_date", current_date()) \
.repartition(col("year"), col("month"), col("day"))
# Write to S3
output_dyf = DynamicFrame.fromDF(df_transformed, glueContext, "output")
glueContext.write_dynamic_frame.from_options(
frame=output_dyf,
connection_type="s3",
connection_options={"path": "s3://data-lake/processed/customers/"},
format="parquet",
format_options={"compression": "snappy"}
)
job.commit()
Job Configuration:
Glue Version: 4.0
Worker Type: G.2X (multi-worker)
Number of Workers: 5 (auto-scale 2-10)
Timeout: 60 minutes
Cost: 5 DPU * $0.44 * 1 hour = $2.20 per run
Daily cost: $66 (~$2,000/month)
Explanation: Dynamic frame handles schema evolution automatically. PII masking (email hashing) protects sensitive data. Partitioning by date enables efficient querying. Job bookmarks prevent re-processing.
Google Cloud – ETL Engineer
Question 3: Implement Glue Streaming for Real-Time Data Pipeline
Build streaming Glue Job consuming Kinesis events, enriching with data from RDS, writing to S3 in mini-batches. Handle late arrivals and out-of-order data.
from awsglue.context import GlueContext
from pyspark.sql import functions as F
glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session
# Read from Kinesis
kinesis_df = spark.readStream \
.format(“kinesis”) \
.option(“streamingAwsRegion”, “us-east-1”) \
.option(“streamName”, “customer-events”) \
.option(“initialPosition”, “latest”) \
.load()
# Enrich with RDS data
rds_df = spark.read.jdbc(
url=”jdbc:postgresql://rds.amazonaws.com/database”,
table=”customers”,
properties={“user”: “admin”, “password”: secret}
)
enriched_df = kinesis_df.join(
rds_df,
kinesis_df.customer_id == rds_df.customer_id,
“left”
)
# Write to S3 in mini-batches (5 minute windows)
enriched_df.writeStream \
.format(“parquet”) \
.option(“path”, “s3://data-lake/streaming/events/”) \
.option(“checkpointLocation”, “s3://data-lake/checkpoints/”) \
.partitionBy(“year”, “month”, “day”, “hour”) \
.trigger(processingTime=”5 minutes”) \
.start() \
.awaitTermination()
Streaming Configuration:
Worker Type: G.2X (optimized for streaming)
Number of Workers: 10 (handle peak load)
Timeout: 0 (continuous)
Initial Position: LATEST (skip backlog)
Cost: 10 DPU * $0.44 = $4.40/hour = $3,168/month (24/7 streaming)
Explanation: Streaming handles real-time events with 5-minute mini-batches. Checkpoint directory enables fault recovery. Left join enriches events with customer attributes. Cost is high for 24/7 streaming—consider alternatives like Kinesis Analytics.
Databricks – Streaming Specialist
Question 4: Glue Job with Data Quality Validation
Implement Glue Data Quality checks: validate row counts match source, no nulls in key columns, values within expected ranges. Fail job and alert if quality checks fail.
from awsglue.job import Job
job = Job(glueContext)
# Define data quality rules
rules = “””
RULE_1 = ROW_COUNT > 1000000 # at least 1M rows
RULE_2 = COMPLETENESS > 0.99 # 99% non-null
RULE_3 = CUSTOM_SQL = SELECT * FROM {}.{} \
WHERE age < 0 OR age > 150 COUNT AS invalid_ages EXPECT invalid_ages = 0
“””
# Run data quality checks
results = DataQualityProvider.runRules(
glue_ctx=glueContext,
rules=rules,
inputs=[{“database”: “raw_data”, “table”: “customers”}]
)
# Handle failures
if not results.passed():
failed_rules = results.get_failed_rules()
message = f”Data quality checks failed: {failed_rules}”
# Send SNS alert
sns = boto3.client(‘sns’)
sns.publish(
TopicArn=’arn:aws:sns:us-east-1:123456789:data-alerts’,
Subject=’Glue Job Data Quality Failure’,
Message=message
)
raise Exception(message)
job.commit()
Data Quality Rules:
1. Row count >= source row count (no data loss)
2. NULL percentage <= 0.5% for key columns
3. Value ranges (age 0-150, price > 0)
4. Uniqueness: customer_id has 0 duplicates
5. Freshness: max(timestamp) < 1 hour ago
Alert Mechanism:
- SNS notification on failure
- CloudWatch alarm triggers
- Job marked as FAILED
- Stops downstream jobs
Explanation: Data quality gates prevent bad data from propagating. Custom SQL rules catch business logic errors. Failures alert teams immediately, preventing downstream issues.
Azure – Data Quality Lead
Question 5: Cost Optimization for Glue Jobs
Portfolio of 50 Glue Jobs: batch (daily), streaming (24/7), on-demand. Optimize costs: right-size workers, schedule efficiently, batch similar jobs. Target: 40% cost reduction.
– 30 batch jobs: 8 DPU * 2 hours = 480 DPU-hours/day
– 10 streaming jobs: 5 DPU * 24 hours = 1,200 DPU-hours/day
– 10 on-demand: 4 DPU * 1 hour = 40 DPU-hours/day
– Total: 1,720 DPU-hours/day * $0.44 = $756.80/day = $22,704/month
Optimization Strategy:
1. Right-size workers:
– Profile jobs to identify actual usage
– Batch jobs: 8 DPU → 4 DPU (fast enough)
– Streaming: 5 DPU → 3 DPU (buffer with alerts)
– Savings: 30% reduction
2. Schedule batch jobs:
– Consolidate 10 small jobs into 3 larger jobs
– Run daily (3 AM) instead of hourly
– Parallel execution: 10 → 5 jobs
– Savings: 50% fewer job runs
3. Replace streaming with on-demand:
– Streaming (5 DPU * 24h) = $52.80/day
– On-demand Kinesis Analytics: $1.50/hour = $36/day
– Savings: 32%
Optimized Configuration:
– Batch jobs: 4 DPU * 2 hours * 3 jobs = 24 DPU-hours/day
– Streaming (replaced): Kinesis Analytics = $36/day fixed
– On-demand: 3 DPU * 1 hour = 3 DPU-hours/day
– Total: 27 DPU-hours * $0.44 + $36 = $47.88/day = $1,436/month
Cost Reduction: $22,704 → $1,436 = 93.7% savings (extremely aggressive)
More realistic: 40% savings = $13,622/month
Explanation: Profile jobs first to right-size workers. Consolidate batch jobs for better utilization. Consider alternative services (Kinesis Analytics, Lambda) for cheaper options. 40% savings achievable without sacrificing SLAs.
AWS – Cost Optimization Specialist
15 Practice Questions (Test Yourself)
Question 6: Design Glue Crawler for auto-scaling data lake with 1,000 S3 buckets and 10 million daily new files. How would you partition crawlers by geography, schedule crawls efficiently, and manage metadata drift?
AWS – Data Lake Architect
Question 7: Implement Glue Data Catalog federation across 3 AWS accounts for multi-tenancy. How would you manage cross-account access, data lineage, and compliance governance?
Google Cloud – Governance Architect
Question 8: Optimize Glue Job performance for 500GB Parquet transformation. Would you use G.1X workers, increase parallelism, or partition source data? Calculate trade-offs.
Databricks – Performance Engineer
Question 9: Build Glue workflow orchestrating 20 dependent jobs with conditional branching (success/failure paths). How would you implement error handling and retry logic?
AWS – Orchestration Specialist
Question 10: Implement Glue job monitoring with CloudWatch metrics and X-Ray tracing. How would you identify performance bottlenecks and optimize job execution?
Azure – Observability Engineer
Question 11: Design Glue job for handling schema evolution as source data adds/removes columns. Implement backward compatibility and version tracking in Data Catalog.
Databricks – Schema Evolution Expert
Question 12: Build Glue Data Quality framework for compliance (GDPR, HIPAA). How would you track data lineage, implement audit logging, and handle data retention policies?
AWS – Compliance Architect
Question 13: Implement Glue Databrew for visual data preparation. Design transformations for data standardization (phone formatting, address normalization). Cost versus custom Glue job?
Google Cloud – Data Preparation Lead
Question 14: Design incremental load pattern using Glue job bookmarks. How would you handle late-arriving data and re-processing failed batches without duplication?
Databricks – Incremental Load Specialist
Question 15: Troubleshoot Glue job OutOfMemoryError with 100GB DataFrame. How would you increase DPU allocation, add explicit garbage collection, or refactor transformations?
AWS – Troubleshooting Specialist
Question 16: Build Glue job integrating with Spark Delta Lake format. How would you handle ACID transactions, time travel queries, and schema evolution with Delta?
Google Cloud – Delta Lake Engineer
Question 17: Implement Glue job encryption at rest using AWS KMS. How would integrate with Secrets Manager for database credentials and audit encryption key usage?
Databricks – Security Engineer
Question 18: Design Glue job for federated query across S3 and Redshift using connection connectors. How would you optimize for mixed workloads (cold data in S3, hot in Redshift)?
AWS – Data Warehouse Architect
Question 19: Build Glue job that exports Data Catalog metadata to Elasticsearch for full-text search. How would you sync metadata changes and keep Elasticsearch index current?
Google Cloud – Search Engineer
Question 20: Implement Glue job as Lambda function trigger. How would you handle event payloads, pass parameters, and return status without overcomplicating architecture?
AWS – Serverless Architect
Master AWS Glue
Get our comprehensive Data Analytics: Ace All The Interview Concepts course.