Shows / AWS Big Data Blog
Episodes
- AWS Big Data Blog
Build declarative ETL pipelines with AWS Glue 6.0
AWS Glue 6.0 introduces Spark Declarative Pipelines. In this post, you build a single declarative AWS Glue 6.0 job that turns raw order records into validated…
- AWS Big Data Blog
AWS recognized as a Leader in the 2026 Gartner Magic Quadrant for Strategic Cloud Platform Services for the 16th consecutive year
On September 1, Gartner published its Magic Quadrant for Strategic Cloud Platform Services (SCPS). Amazon Web Services (AWS) is the longest-running Leader in…
- AWS Big Data Blog
How Sony LIV built real-time video streaming analytics with AWS
Real-time data analytics is transforming how streaming platforms serve their audiences. Learn how Sony LIV built a comprehensive, real-time streaming analytics…
- AWS Big Data Blog
From silos to insights: Federated data access patterns for AI agents
AI agents can reach enterprise data where it lives instead of routing every question through data engineers. This post presents three reference patterns for…
- AWS Big Data Blog
Network connectivity patterns for the next generation of Amazon OpenSearch Serverless
The next generation of Amazon OpenSearch Serverless uses standard AWS PrivateLink endpoints on the on.aws domain. This post shows nine connectivity patterns…
- AWS Big Data Blog
How Moovit achieved 33% cost optimization through architectural modernization
Learn how Moovit modernized its data platform with a multi-engine lakehouse architecture: offloading heavy aggregation workloads from Amazon Redshift to Amazon…
- AWS Big Data Blog
Query Amazon S3 Tables from Amazon EMR Trino using the Iceberg REST endpoint
Learn how to query Amazon S3 Tables from Trino on Amazon EMR using the Apache Iceberg REST catalog endpoint. This post shows how to deploy the integration with…
- AWS Big Data Blog
Building medallion architecture with Iceberg materialized views in Amazon SageMaker
With Apache Iceberg materialized views in Amazon SageMaker, you can build a Bronze, Silver, and Gold medallion architecture as three SQL statements. This…
- AWS Big Data Blog
Build a dynamic streaming data lake with Apache Iceberg and Apache Flink
Learn how to build a dynamic streaming data lake on Amazon Managed Service for Apache Flink that adapts to new event types and schema changes without stopping…
- AWS Big Data Blog
Observing and evaluating production agents using OpenSearch Agent Health
Learn how to observe and evaluate production AI agents by combining an agent running on AWS with OpenSearch Agent Health. This post walks through deploying an…
- AWS Big Data Blog
Accelerate Apache Spark debugging on Amazon EMR with AWS DevOps Agent
Extend AWS DevOps Agent to investigate Apache Spark failures on Amazon EMR. This post shows how to register the Apache Spark Troubleshooting Agent for Amazon…
- AWS Big Data Blog
Deliver real-time data to streaming tables for Apache Iceberg with Amazon Kinesis Data Streams
Amazon Kinesis Data Streams now supports streaming tables, a fully managed capability that continuously delivers your streaming data as queryable Apache…
- AWS Big Data Blog
Measuring and improving search quality with Amazon OpenSearch Service
Most teams struggle to answer a deceptively simple question: is my search returning relevant results? This post shows how to capture User Behavior Insights…
- AWS Big Data Blog
Integrate Amazon Redshift and IAM Identity Center with enhanced VPC routing
Amazon Redshift now supports AWS IAM Identity Center authentication on clusters and workgroups that use enhanced VPC routing. Create two interface VPC…
- AWS Big Data Blog
Build a real-time event pipeline with Spark Real-Time Mode on AWS Glue 6.0
With AWS Glue 6.0, you can build real-time, near-real-time, and batch data pipelines on a single platform. Using a financial market-risk example, learn how to…
- AWS Big Data Blog
Razor Group’s journey to a modern data lakehouse on AWS
Razor Group, one of Europe's leading ecommerce aggregators managing 250+ brands, migrated from always-on Amazon Redshift clusters to an open lakehouse on…
- AWS Big Data Blog
How Picnic configured multiple OAuth providers for Amazon MQ
Picnic runs RabbitMQ as the messaging backbone for hundreds of microservices on Amazon MQ for RabbitMQ. This post shows how to configure one broker to trust…
- AWS Big Data Blog
Build with geospatial and variant types in Iceberg v3 on AWS Glue 6.0
AWS Glue 6.0 with Apache Spark 4.1 adds support for Apache Iceberg v3: native geospatial types, nanosecond-precision timestamps, the VARIANT type, and DEFAULT…
- AWS Big Data Blog
Migrate an OAuth 2.0 authenticated Apache Kafka cluster to Amazon MSK with MSK Replicator
MSK Replicator now supports OAuth 2.0 (SASL/OAUTHBEARER) authentication to external Apache Kafka clusters. This post walks through the three supported grant…
- AWS Big Data Blog
Announcing in-place ZooKeeper-to-KRaft cluster upgrades for Amazon MSK
Amazon MSK now supports in-place upgrades from ZooKeeper to KRaft metadata mode. You can modernize your existing cluster's metadata management through the…
- AWS Big Data Blog
Amazon MSK Service 101: How many partitions does an Amazon MSK topic need?
How many partitions does your Amazon MSK topic need? Choosing the right partition count affects throughput, scalability, and operational complexity. This post…
- AWS Big Data Blog
AWS and DuckLabs: Building the future of analytics together
Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical…
- AWS Big Data Blog
PythonOperator and BashOperator Now Available on Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless
You can now use PythonOperator and BashOperator to run custom Python functions and shell scripts directly in the Amazon MWAA Serverless runtime, without…
- AWS Big Data Blog
Long-term system tables retention in Amazon Redshift with Amazon S3 Tables
Amazon Redshift system table integration with Amazon S3 Tables automatically delivers your system table logs to Amazon S3 Tables in Apache Iceberg format. You…
- AWS Big Data Blog
Track SageMaker Unified Studio project costs with custom tags and AWS CUR
Learn how to track Amazon SageMaker Unified Studio project costs by custom tags. This serverless solution enriches AWS Cost and Usage Report (CUR) data with…
- AWS Big Data Blog
Secure SageMaker Unified Studio access with SAML and conditional policies
Learn how to secure Amazon SageMaker Unified Studio by integrating it with an external SAML identity provider such as Okta. This post shows you how to apply…
- AWS Big Data Blog
Querying raw log data using SQL and PPL with the optimized engine in Amazon OpenSearch Service
Learn how to run fast analytical queries directly against raw log and trace data in Amazon OpenSearch Service using PPL and SQL. Follow a single incident…
- AWS Big Data Blog
Fresher insights, faster decisions: talabat’s near-real-time analytics across AWS and Google Cloud
Leading everyday app across the Middle East and North Africa, talabat, built a hybrid multi-cloud lakehouse that keeps a single Apache Iceberg copy of…
- AWS Big Data Blog
Powering agentic AI with real-time streaming data on AWS
Agentic AI applications now observe, reason, and act on streaming data in production. This post presents three architecture patterns that form a unified…
- AWS Big Data Blog
Amazon MSK simplifies configuring custom domain names
With Amazon MSK, you can now configure custom domain names for provisioned clusters using a single configuration property that works identically on ZooKeeper…
- AWS Big Data Blog
How Zepto powers sub-second search using OpenSearch Service OR2 instances
Learn how Zepto, India's fast-growing quick-commerce platform, migrated Amazon OpenSearch Service to OpenSearch Optimized (OR2) instances to scale sub-second…
- AWS Big Data Blog
IAM authentication with OAuth 2.0 for Amazon MQ for RabbitMQ
IAM authentication with OAuth 2.0 lets clients connect to Amazon MQ for RabbitMQ using their existing IAM identity instead of static broker-local credentials…
- AWS Big Data Blog
OAuth 2.0, LDAP, and HTTP auth for Amazon MQ for RabbitMQ
Amazon MQ for RabbitMQ supports OAuth 2.0, LDAP, and HTTP-based authentication backends so you can connect your broker to the identity infrastructure you…
- AWS Big Data Blog
Mutual TLS and SSL certificate authentication for Amazon MQ for RabbitMQ
Learn how to add certificate-based identity verification to Amazon MQ for RabbitMQ. This post explains SSL certificate authentication for passwordless login…
- AWS Big Data Blog
Authentication and authorization options for Amazon MQ for RabbitMQ
Amazon MQ for RabbitMQ supports multiple authentication and authorization methods, so you can connect your broker to the identity infrastructure you already…
- AWS Big Data Blog
NaranjaX manages multiple Amazon MSK Serverless clusters in different accounts from their IDP using AWS RAM and Route 53
Learn how NaranjaX built a cross-account, many-to-many connectivity model for Amazon MSK Serverless using AWS Resource Access Manager and Amazon Route 53…
- AWS Big Data Blog
How Autodesk migrated 2.3 billion documents to Amazon OpenSearch Service using Migration Assistant and intelligent routing
This post walks through how Autodesk re-architected a single-index Elasticsearch 7.1.1 domain on Amazon OpenSearch Service into four multi-index OpenSearch…
- AWS Big Data Blog
Trace cascading decision failures with a blame graph on Amazon OpenSearch Service
When multiple AI agents collaborate on a decision and get it wrong, standard logs can't tell you which agent caused it. This post shows how to build a blame…
- AWS Big Data Blog
How AppFolio transformed its data streaming architecture with Amazon MSK Express brokers
Learn how AppFolio transformed its data streaming architecture by adopting Amazon MSK Express brokers, replacing hours-long rebalances and manual storage…
- AWS Big Data Blog
How GPU acceleration builds billion-scale vector indexes on Amazon OpenSearch Service
GPU-accelerated vector (k-NN) indexing on Amazon OpenSearch Service and OpenSearch Serverless lets you build billion-scale vector indexes in hours instead of…
- AWS Big Data Blog
Centralized CloudTrail monitoring across 100+ AWS accounts
Learn how to build a centralized AWS CloudTrail monitoring solution on Amazon OpenSearch Service, with Terraform managing the full stack. It handles 200 GB/day…
- AWS Big Data Blog
How a team at Epic Games tuned Amazon OpenSearch Service for Fortnite analytics
Learn how a team at Epic Games tuned their Amazon OpenSearch Service cluster for Fortnite analytics. This post details how Epic Games partnered with AWS to…
- AWS Big Data Blog
AI-powered cost optimization agent for Amazon Kinesis Data Streams
Learn how to deploy an open-source, AI-powered agent built on Amazon Bedrock that automatically analyzes every Amazon Kinesis Data Streams stream in your…
- AWS Big Data Blog
Amazon OpenSearch Service extends version lifecycle support timelines
In November 2024, we announced Standard and Extended Support dates for legacy Elasticsearch and OpenSearch versions on Amazon OpenSearch Service. We are…
- AWS Big Data Blog
Scaling fine-grained access control for enterprise lakehouse using SageMaker Unified Studio and AWS Lake Formation
As enterprise lakehouses grow to thousands of tables across business domains and regions, fine-grained access control becomes a governance bottleneck. This…
- AWS Big Data Blog
Event-driven pipeline orchestration with Amazon MWAA and Airflow 3.0
Data engineering teams running Apache Airflow across multiple AWS accounts have no built-in way to coordinate workflows between separate Amazon MWAA…
- AWS Big Data Blog
Transforming search at Delivery Hero: A migration journey to OpenSearch Service with radial search
In this post, we walk through how Delivery Hero migrated their semantic search infrastructure to Amazon OpenSearch Service, why they chose radial search over…
- AWS Big Data Blog
Amazon Redshift multi-Region disaster recovery
In this post, we walk through the core concepts of cross-Region disaster recovery, introduce a framework for assessing your requirements, and then dive deep…
- AWS Big Data Blog
Streamline Apache Kafka cluster operations and migrations with Agent Skills for Amazon MSK
Agent Skills for Amazon MSK bring broker-type-aware expertise to operating and migrating Apache Kafka clusters. In this post, we walk through installing the…
- AWS Big Data Blog
Introducing Apache Spark troubleshooting agent for Amazon EMR on EKS
In this post, we show you how to set up the agent for Amazon EMR on EKS and walk through troubleshooting a failed job run. We demonstrate the workflow from…
- AWS Big Data Blog
Upgrade Amazon Redshift DC2 clusters to the new Amazon Redshift RG
Upgrading your Amazon Redshift DC2 clusters to AWS Graviton-based RG instances unlocks managed storage, data sharing, zero-ETL, and an integrated data lake…
- AWS Big Data Blog
Optimizing costs and performance with Advanced Managed Scaling on Amazon EMR on EC2
In this post, we discuss the benefits of Advanced Scaling for Amazon EMR on Amazon EC2 and demonstrate how it works through some example scenarios. You'll…
- AWS Big Data Blog
Deliver Apache Kafka data to streaming tables for Apache Iceberg with Amazon MSK Express brokers
Announcing delivery to streaming tables on Apache Iceberg for Amazon MSK Express brokers, a fully managed capability that continuously materializes your Kafka…
- AWS Big Data Blog
Lowering AWS KMS decrypt API costs in EMR Spark jobs
Processing encrypted data in Amazon S3 with Amazon EMR and Apache Spark can drive up AWS KMS decrypt API costs as the number of objects grows. This post shows…
- AWS Big Data Blog
Accelerate Spark on EMR Serverless with larger workers and shuffle-optimized disks
Amazon EMR Serverless now supports a 32 vCPU / 244 GB worker configuration for the most demanding Spark jobs. Across 126 TPC-DS and TPC-H queries, larger…
- AWS Big Data Blog
Automate Spark Scala migration to 4.x with AWS Spark Upgrade Agent
Learn how to automate Apache Spark 3.x to 4.0 Scala migration on Amazon EMR using the AWS Spark Upgrade Agent. This post covers API deprecations, behavioral…
- AWS Big Data Blog
Build a contract compliance search system with Amazon OpenSearch
In this post, you build a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. You deploy…
- AWS Big Data Blog
Building a scalable personalized recommendation system on AWS: From batch to real-time
Learn how the Everyday Essentials team built a scalable personalized recommendation platform on AWS using a batch-first architecture with Amazon MWAA for…
- AWS Big Data Blog
Automate creating AWS Glue Data Catalog views with AWS SDK for data mesh use case
This post shows you how to use the Catalog objects API CreateTable() to programmatically create ATHENA and SPARK dialects using cross-account IAM definer…
- AWS Big Data Blog
Efficient log management with Amazon OpenSearch Service data streams
In this post, we show you how to implement data streams with Index State Management (ISM) in Amazon OpenSearch Service. This approach automatically manages…