Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / How Cloud Contracts Now Charge for Instance Metadata API Calls
Transcript
- Lucas: Alright, Luna, here's a bill line item that wasn't on anyone's radar even eighteen months ago: your cloud provider is now charging you for every call to the instance metadata service. Luna: Wait — the metadata service? That's the thing every tool uses to figure out which instance it's running on, grab the region, get the IAM role. That used to be free. Lucas: Completely free, yeah. AWS's IMDS, Azure's IMDS, GCP's metadata server — all of them started as a zero-cost utility built into the hypervisor. But starting late last year and accelerating through 2025, all three have introduced metering. AWS charges per million IMDSv2 calls beyond a small free tier. Azure started billing for IMDS transactions in January. GCP added metadata API usage to their network egress-equivalent cost categories. Luna: Let me guess — the billing teams saw that a huge percentage of compute API calls were metadata requests and decided it was leaving money on the table? Lucas: Exactly. A finops engineer I talked to at a mid-size e-commerce company — let's call them RetailCo — pulled their metadata call logs and found they were making over 800 million metadata requests per month across their fleet. That's about 3,000 calls per instance per day. Most of it was from monitoring agents, configuration management tools, and Kubernetes node initialization scripts. Luna: And what did that 800 million cost them? Lucas: Under the new pricing, about $4,200 a month. And that was only AWS. They had similar volumes on Azure and GCP. The total across all three clouds was just under $11,000 monthly — for something they thought was free. Luna: Ouch. So who's the biggest culprit? I mean, what's hammering the metadata service that hard? Lucas: The single biggest offender was a misconfigured Kubernetes DaemonSet. Somebody had deployed a logging agent that called the metadata API every thirty seconds to refresh the node's region tag — but it was running on every node, and the refresh interval was way too aggressive. That one DaemonSat generated 400 million calls a month, roughly half the total. Luna: Right, so the fix is caching. Cache the metadata response at the application level, or at least extend the refresh interval from thirty seconds to, say, five minutes. Lucas: Exactly. RetailCo reduced the DaemonSet refresh to every ten minutes and added a local cache using the instance's tmpfs. Dropped that workload's metadata calls by 95 percent. But here's the thing — the metadata service is also called implicitly by almost every AWS SDK and CLI command. Every time you run 'aws sts assume-role' or 'aws ec2 describe-instances', the SDK hits the metadata endpoint to get the credentials. So even well-configured apps can generate a surprising baseline. Luna: So what's the actual pricing breakdown? How much per million calls on each cloud? Lucas: AWS charges $0.05 per million IMDSv2 calls after the first 10 million free per account per month. Azure is $0.04 per million IMDS transactions, also with a 10 million free tier. GCP is a bit different — they don't have a per-call price, but they've added metadata server usage to their 'other network services' category, which is billed per gigabyte of metadata response data transferred. It works out to roughly $0.06 per million for typical payload sizes. Luna: Those per-million numbers sound tiny, but when you have tens of thousands of instances running twenty-four-seven, it adds up fast. RetailCo's $11,000 is real money. Lucas: And it's a completely silent cost. There's no alert. No default dashboard. You have to explicitly enable metadata call logging in CloudTrail or Azure Monitor, then build a custom cost allocation report. Most teams never see it until the bill arrives. Luna: I want to back up for a second. Why are they doing this now? Is it just revenue generation, or is there a technical reason? Lucas: Part of it is cost recovery. The metadata service runs on the hypervisor and each call burns CPU and memory on the host. As instance densities increased with newer processor generations, the metadata request volume grew faster than the hardware could absorb for free. AWS published a paper showing that metadata calls accounted for nearly 12 percent of control-plane API traffic in some regions. So yes, there's a genuine cost — but the pricing is probably set to do more than just cover costs. It's also a nudge: don't poll the metadata service like it's a free API. Luna: Alright, so for anyone listening who manages cloud infrastructure, what's the first step to finding out if they're bleeding money on metadata? Lucas: First, enable metadata call logging. On AWS, that means enabling IMDSv2 and turning on CloudTrail data events for the EC2 instance metadata service. On Azure, enable diagnostic settings for the IMDS. On GCP, you need to enable metadata server audit logs. Then run a report for a week and look for the top callers by source IP or instance ID. Luna: And then on the fix side, the big wins are caching, reducing refresh intervals, and removing unnecessary calls. Any tools that help automate that? Lucas: There's an open-source agent called 'meta-cache' that runs as a sidecar and serves cached metadata responses to local processes. A few teams are using eBPF-based filters to block duplicate metadata requests entirely. But honestly, the simplest fix is just auditing your DaemonSets and init containers. Most metadata calls come from startup scripts that don't need to run every thirty seconds. Luna: So before we go deeper into the technical fixes — Lucas, I think this episode is a good example of the kind of practical, money-saving insight that our listeners really value. If today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. Lucas: Yeah, absolutely. It's a small way to keep these conversations going without any sponsor interruptions. And we really appreciate it. Luna: Alright, back to the metadata. So you mentioned eBPF-based filters. How do those work? Lucas: An eBPF program can intercept all outbound HTTP connections to the metadata IP — 169.254.169.254 on AWS and Azure, 169.254.169.254 on GCP as well — and deduplicate identical requests within a short time window. If five processes ask for the same IAM role credential within the same second, the eBPF filter returns the cached response for four of them and only forwards one actual call to the metadata service. That can cut call volume by 60 to 80 percent with zero application changes. Luna: That's clever. But it requires kernel expertise to deploy safely. For most teams, the low-hanging fruit is probably still the audit and config change. Lucas: Right. And there's another angle: cloud providers themselves are starting to offer managed caching. AWS recently previewed a feature called 'IMDS response caching' that lets you set a TTL on metadata responses at the hypervisor level. Azure has a similar capability in preview called 'IMDS cache mode'. So the providers are acknowledging the problem and offering a built-in solution — for an additional fee, naturally. Luna: Of course. So the cycle continues. But the takeaway is clear: metadata API calls are no longer free, and ignoring them leaves real money on the table. Lucas: Exactly. And the trend is only going one direction. I suspect within two years, the free tier will shrink to zero, and per-call pricing will become a standard line item in every cloud bill. Teams that build caching discipline now will have a permanent cost advantage. Luna: Alright, let's talk about one more surprising source of metadata calls: container orchestration tools that use the metadata service to assign pod identity. Have you seen any specific patterns there? Lucas: Yeah, that's a big one. On EKS, the IRSA — IAM Roles for Service Accounts — feature relies on the metadata service to vend temporary credentials. Every time a pod starts, it hits the metadata endpoint to assume the role. If you have a cluster with rapid pod churn — say, from batch jobs or spot instance interruptions — you can get thousands of metadata calls per minute just from identity. Luna: Is there a way to reduce that without breaking the identity model? Lucas: Some teams are moving to EKS Pod Identity, which is a newer AWS feature that doesn't require the metadata service at all. It injects credentials directly into the pod via a sidecar. On Azure, there's workload identity federation that bypasses IMDS. GCP has workload identity federation as well. These are free features that eliminate the metadata call for identity entirely. Luna: So the migration path is: move off IRSA onto Pod Identity, cache all other metadata calls, and set sane refresh intervals. That's a solid action plan. Lucas: That's exactly the playbook. And it's worth doing sooner rather than later, because once the free tier disappears, the cost of doing nothing multiplies.