Shows / AWS Big Data Blog
Episodes
- AWS Big Data Blog
Understanding span queries in Amazon OpenSearch Service
Amazon OpenSearch Service is a fully managed service for deploying, scaling, and operating OpenSearch in the AWS Cloud. OpenSearch is an open source software…
- AWS Big Data Blog
Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs
The AWS Prescriptive Guidance topic Optimize user-defined functions explains that Python UDFs in PySpark can degrade performance because data must be…
- AWS Big Data Blog
Multi-statement transactions for Apache Iceberg tables in Amazon Redshift
Data pipelines often change a table in several steps, such as first upserting the day’s sales and then removing canceled orders. Amazon Redshift now supports…
- AWS Big Data Blog
Build a self-healing ETL pipeline with Amazon Bedrock AgentCore
Extract, transform, and load (ETL) pipelines break silently. Schema drift, missing fields, and type mismatches corrupt downstream analytics. The typical fix is…
- AWS Big Data Blog
Choose the right streaming ingestion path for Apache Iceberg tables on AWS
Choosing the streaming ingestion path to write to Apache Iceberg tables is a key step in building low-latency, high value data lakes. Organizations are…
- AWS Big Data Blog
Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake
A cloud data lake is a scalable, low-cost place to store data from many different sources. Over the years, data lakes built on Amazon Simple Storage Service…
- AWS Big Data Blog
Managed cross-Region disaster recovery for Amazon OpenSearch Service
Amazon OpenSearch Service already offers high availability within a single AWS Region through Multi-Availability Zone (Multi-AZ) deployments. Some…
- AWS Big Data Blog
Optimizing Apache HBase availability on Amazon EMR, Part 2: Diagnose and tune compaction
This post shows you how to diagnose and tune compaction in Apache HBase on Amazon EMR. Compaction is the background process that merges store files. When it…
- AWS Big Data Blog
Build a data foundation for agentic AI on AWS: From lakehouse to MCP
Building a data foundation ready for agentic AI is becoming a priority for organizations deploying AWS. This foundation relies on a unified lakehouse and…
- AWS Big Data Blog
Build a multi-team AI and analytics environment on Amazon SageMaker Unified Studio with Terraform
Platform engineers who support data and AI teams face a recurring challenge: analytics, machine learning (ML), and AI developer teams each need different…
- AWS Big Data Blog
How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito
This post is co-written with Ajay Gunasekaran, ML Platform Engineer at Samsung Ads. Samsung Ads, the advertising division of Samsung Electronics, keeps its…
- AWS Big Data Blog
Allocate shared Amazon MSK costs to Kafka topics for cost visibility
Your AWS bill shows Amazon Managed Streaming for Apache Kafka (Amazon MSK) charges at the cluster level, with one cost line item for each cluster. If your…
- AWS Big Data Blog
Governance compliance beyond the catalog in Amazon SageMaker Unified Studio
As your data catalog grows, how do you know which assets meet your organization’s governance standards? Amazon SageMaker Unified Studio provides the building…
- AWS Big Data Blog
How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog
Migrating from Apache Hive Metastore (HMS) to AWS Glue Data Catalog (GDC) is a priority for many organizations as they scale their data analytics operations…
- AWS Big Data Blog
46% fewer nodes, 2× faster indexing: Confluent on Amazon OpenSearch Service
Operating one of the world’s largest data-streaming platforms, Confluent, the company founded by the original creators of Apache Kafka and now part of IBM…
- AWS Big Data Blog
Multiply pre-recorded streams with Amazon Managed Service for Apache Flink
Validating a streaming pipeline beyond current peak takes production-scale test data, and you can get it by multiplying pre-recorded streams with Amazon…
- AWS Big Data Blog
Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue
If you run data pipelines, you know how much work they can take to set up, sometimes only to break without you realizing. That doesn’t even include data from…
- AWS Big Data Blog
Introducing the Workday Data Cloud integration with AWS
Connecting enterprise systems to modern analytics and AI platforms often requires building brittle pipelines, duplicating data, or sacrificing the governance…
- AWS Big Data Blog
Define metrics once and keep them correct with the dbt Semantic Layer on Amazon Redshift
Most analytics stacks model data into a curated gold layer and call the result a semantic layer. For a lot of workloads, that’s a perfectly good one. The…
- AWS Big Data Blog
Assess and track your Amazon Redshift RA3 to RG modernization with CUDOS
Amazon Redshift is a fully managed, AI-powered cloud data warehouse used by tens of thousands of customers to analyze exabytes of data with industry-leading…
- AWS Big Data Blog
Amazon Athena introduces system tables for query execution observability
Many of you use Amazon Athena to run SQL queries across your data lake, powering everything from interactive analytics to scheduled reporting pipelines, often…
- AWS Big Data Blog
Store and process large strings in Amazon Redshift SUPER columns
Storing large semi-structured data such as JSON and arrays in a data warehouse often means splitting or truncating your data. Amazon Redshift removes this with…
- AWS Big Data Blog
Deliver multi-tenant AI analytics at scale with AgentCore and Lake Formation
Governed, multi-tenant data access is a critical requirement for software and business-to-consumer (B2C) products that embed AI-powered analytics. A data agent…
- AWS Big Data Blog
Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg
Enterprises increasingly measure their readiness for AI by the readiness of their data, and that often means moving to an open lakehouse where data stays open…
- AWS Big Data Blog
Implement least-privilege access with designations in Amazon SageMaker Unified Studio
When teams move data and AI work to production in a shared environment, one question quickly becomes unavoidable: who can do what? Amazon SageMaker Unified…
- AWS Big Data Blog
Optimizing Apache HBase availability on Amazon EMR, Part 1: Coordinator startup, RegionServer recovery, and regions in transition
Running Apache HBase on Amazon EMR gives you a scalable, high-performance NoSQL store, and following well-understood operational patterns can help improve…
- AWS Big Data Blog
Introducing time series collections for the next generation of Amazon OpenSearch Serverless
Today, we’re announcing support for time series collections on the next generation of Amazon OpenSearch Serverless. With time series collections, you can…
- AWS Big Data Blog
From Problem to Fix in Minutes: Automating Search Relevance Tuning with the Amazon OpenSearch Service Relevance Agent
Automating search relevance tuning means going from “my NDCG score is low” to “here is a better search query” in minutes. The Amazon OpenSearch Service Search…
- AWS Big Data Blog
Build RAG-ready indexes from Amazon S3 with Amazon OpenSearch intelligent ingestion
Much of an organization’s most valuable knowledge sits in documents such as contracts, product manuals, research reports, support runbooks, and financial…
- AWS Big Data Blog
Enable cross-Region fine-grained data access with AWS Lake Formation and Trusted Identity Propagation
Organizations that store analytics data on AWS often need two things at once. The first is fine-grained access control that limits each person to only the data…
- AWS Big Data Blog
Trace security threats in multi-agent pipelines with Amazon OpenSearch Service
In this post, you learn how to trace two invisible threats in a multi-agent pipeline, prompt injection propagation and cross-tenant data exfiltration, using…
- AWS Big Data Blog
dbt and Amazon Redshift Data Sharing: Multi-cluster pipelines without moving a byte
Today’s global, data-driven organizations treat data as an asset and use it across different lines of businesses (LOBs) to drive timely insights and better…
- AWS Big Data Blog
Resolving concurrent write conflicts for Apache Iceberg tables in Amazon Redshift
In Getting started with Apache Iceberg write support in Amazon Redshift – Part 1, you learned how to create Apache Iceberg tables and write to them with…
- AWS Big Data Blog
Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality
Bad data rarely announces itself. It slips into an Amazon EMR pipeline quietly. A null where there should be a value. A timestamp in the wrong format. A…
- AWS Big Data Blog
Build cross-account streaming pipelines with the Amazon Kinesis Producer Library and Kinesis Client Library
Organizations use real-time streaming to act on data the moment it arrives. They detect fraud, personalize customer experiences, and respond to infrastructure…
- AWS Big Data Blog
Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service
When your MWAA-orchestrated extract, transform, and load (ETL) pipeline spans multiple AWS services, troubleshooting a failure becomes a scavenger hunt. AWS…
- AWS Big Data Blog
Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue
Many data integration jobs are SQL-centric: they filter, join, and aggregate data on a schedule, and they run frequently enough that fast startup matters. For…
- AWS Big Data Blog
Query across accounts and table formats with multi-catalog in Amazon EMR
Analytics teams on AWS often store data in more than one open table format, and that data frequently lives in more than one AWS account. Two problems follow…
- AWS Big Data Blog
Materialize once, query anywhere: Introducing Iceberg materialized views in Amazon Redshift
Amazon Redshift has progressively deepened its integration with Apache Iceberg. Earlier this year we launched Amazon Redshift RG, powered by AWS Graviton, with…
- AWS Big Data Blog
Run an automated operational review with the Amazon Redshift MCP server
It’s the end of a strong quarter, and your Amazon Redshift workloads have grown with the business. Data volumes are up, new pipelines have shipped, and more…
- AWS Big Data Blog
Optimize consumer rebalancing on Amazon MSK with next generation protocol
If you run large consumer groups on Apache Kafka and Amazon Managed Streaming for Apache Kafka (Amazon MSK), you’ve likely experienced the pain of slow…
- AWS Big Data Blog
Building an LLM-powered DAG failure analysis plugin for Amazon MWAA
Apache Airflow has become the orchestration backbone for data pipelines across industries. But as those pipelines grow to hundreds of directed acyclic graphs…
- AWS Big Data Blog
Announcing Spark Connect on Amazon EMR on EKS: Interactive PySpark development, anywhere
Today, we’re announcing support for Spark Connect on Amazon EMR on EKS, starting from EMR release 7.14 (Apache Spark 3.5.8) and emr-spark-8.1 (Apache Spark…
- AWS Big Data Blog
Aurora PostgreSQL zero-ETL integration with Amazon SageMaker
When you need quick insights from your Amazon Aurora PostgreSQL operational data, traditional analytics approaches force you to build complex extract…
- AWS Big Data Blog
Getting started with Apache Iceberg write support in Amazon Redshift – Part 3
Production data is always evolving. Tables gain and lose columns, outgrow their data types, and get re-partitioned as query patterns shift. Multiple engines…
- AWS Big Data Blog
Enforce IAM permissions boundaries for Amazon SageMaker Unified Studio Tooling blueprints
Amazon SageMaker Unified Studio now supports custom permissions boundaries for IAM roles created by the Tooling blueprint. Organizations that enforce Service…
- AWS Big Data Blog
How Delivery Hero rebuilt real-time ad measurement with Apache Flink
Learn how Delivery Hero rebuilt its ad measurement pipeline from hourly batch processing to real time on Amazon Managed Service for Apache Flink, cutting the…
- AWS Big Data Blog
Configure domain-level VPC networking in Amazon SageMaker Unified Studio
Configuring VPC networking per project across a SageMaker Unified Studio domain creates inconsistent, hard-to-audit networks. This post shows administrators…
- AWS Big Data Blog
Announcing Spark Connect on Amazon EMR on EC2: Interactive PySpark anywhere
Amazon EMR on EC2 now supports Spark Connect, so you can develop and debug PySpark interactively from Amazon SageMaker Unified Studio Data Notebooks or your…
- AWS Big Data Blog
Query unstructured data in Amazon SageMaker Catalog using generative AI
In Part 2 of this series, sign in as a data consumer, subscribe to enriched unstructured data assets in Amazon SageMaker Catalog, and query them using natural…
- AWS Big Data Blog
Best practices for scaling large consumer groups on Amazon MSK
As consumer groups on Amazon MSK scale to thousands of members, the metadata record Kafka writes during rebalances can exceed the 1 MB limit and stall the…
- AWS Big Data Blog
How Moeve standardized dbt runs across data lakes with Amazon Athena
Moeve standardized how it runs dbt across multiple data lakes by building a centralized, serverless launcher on Amazon Athena, AWS Step Functions, AWS Fargate…
- AWS Big Data Blog
Discover and govern Snowflake data using SageMaker Unified Studio
Connect Snowflake to Amazon SageMaker Unified Studio to build a unified data catalog. Query federated Snowflake tables without moving data, publish enriched…
- AWS Big Data Blog
Connect Amazon SageMaker Unified Studio to Microsoft Power BI – Part 1: IAM Identity Center (IDC)-based domains
Connect Microsoft Power BI directly to governed data in Amazon SageMaker Unified Studio using new authentication modes in the Amazon Athena ODBC driver, with…
- AWS Big Data Blog
Connect Amazon SageMaker Unified Studio to Microsoft Power BI – Part 2: IAM-based domains
Connect Microsoft Power BI directly to governed data in Amazon SageMaker Unified Studio using the Amazon Athena ODBC driver. Part 2 covers IAM-based domains…
- AWS Big Data Blog
How United Airlines uses Amazon Redshift and AWS Glue Data Catalog federation to query Databricks-managed data
Learn how United Airlines uses AWS Glue Data Catalog federation to query Databricks Unity Catalog data directly from Amazon Redshift Serverless without…
- AWS Big Data Blog
Scale down Kinesis Data Streams on-demand capacity with ODA warm throughput
Amazon Kinesis Data Streams now supports scaling down ingest capacity for on-demand Advantage streams with warm throughput. Learn how the scale-down works, how…
- AWS Big Data Blog
How to migrate from Amazon CloudSearch to Amazon OpenSearch Serverless
Learn how to migrate an Amazon CloudSearch domain to Amazon OpenSearch Serverless: assess your configuration, create a collection with explicit index mappings…
- AWS Big Data Blog
Accelerating Spark queries with Iceberg materialized views
Accelerate slow, repetitive Apache Spark analytical queries on Apache Iceberg tables without rewriting any SQL. This post shows how automatic query rewrite in…
- AWS Big Data Blog
Every team is a data team — bring Amazon Redshift analytics to ChatGPT Work
AWS is announcing the AWS Data Analytics plugin for the new Data agent in ChatGPT Work. Teams can ask questions in natural language, analyze governed data…