Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality
Bad data rarely announces itself. It slips into an Amazon EMR pipeline quietly. A null where there should be a value. A timestamp in the wrong format. A referential key that no longer exists. It travels downstream until it surfaces as a wrong number in a dashboard, a failed downstream job, or a business decision made on data nobody verified.
For teams running large-scale data processing on Amazon EMR, choosing a data quality strategy has meant picking between two trade-offs. You can build your own quality checks in Apache Spark using an open source library like Deequ. With this approach, you…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.