Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How HashiCorp Rebuilt Terraform for Multi-Cloud Scale
Transcript
- Lucas: So Terraform just hit a hundred million monthly runs. That's not a vanity number — every one of those runs is a plan or apply against real infrastructure. And the architecture that makes that possible has changed dramatically from the Terraform most of us first used around 2016. Luna: Yeah I remember early Terraform — single binary, state file sitting on a local disk, and if you forgot to commit it you'd destroy production. How did they get from there to what we have today? Lucas: Right. The big architectural shift was moving from monolithic state handling to what they now call 'remote state with locking and isolation.' Early Terraform stored state as a JSON file on the filesystem. That worked for a single engineer, but once teams grew, you had state conflicts, corruption, and 'state drift' becoming a career-limiting problem. Luna: So the solution was to centralise state — put it in a backend like S3 or Consul. But that alone doesn't solve concurrency. Lucas: Exactly. They added state locking via DynamoDB or Consul, which gave you a mutex: only one person could run apply at a time. That was a critical step. But it introduced a new bottleneck: every plan and apply now required a round trip to the backend, and if you were running fifty concurrent pipelines, you'd queue up fast. The next shift was to what they now call 'workspace-based isolation.' Each workspace gets its own state file, its own lock, and its own variables. That let teams run hundreds of workspaces in parallel without stepping on each other. Luna: That sounds like a natural evolution, but the real magic is in the provider plugin model. How did they design that to scale? Lucas: HashiCorp made a savvy bet early on: make Terraform provider-agnostic but let the providers be developed independently. The core binary just defines a protocol — a gRPC interface that providers implement. So AWS provider, Azure provider, Kubernetes provider — each is a separate binary that Terraform launches as a child process. That means you can update providers independently, fix bugs without a core release, and the community can write their own. As of June 2026, the Terraform Registry has over two thousand providers. Luna: That's a huge surface area. How do they maintain quality? I know there's a tiering system — official, partner, community. Lucas: Right. HashiCorp maintains about fifty 'official' providers — the big clouds and core services. Those go through rigorous code review, testing, and compatibility testing against the core. Partner providers are maintained by companies like Datadog or Fastly — they sign a partnership agreement and follow a testing framework. Community ones are best-effort. The interesting architectural detail is that Terraform Core doesn't actually know about any cloud API — it just knows about the resource graph and the plan/apply lifecycle. The provider is responsible for all the CRUD operations and state handling. That separation of concerns is what let them scale from a handful to thousands of providers. Luna: So the core stays relatively stable, and the complexity lives in the providers. That's a classic platform strategy. But let's talk about performance. A hundred million runs means some of those plans must be massive — thousands of resources. How does Terraform handle dependency resolution at that scale? Lucas: Great question. Terraform builds a dependency graph from the configuration. It walks through all resources, figures out the order based on references — like 'this EC2 instance depends on a security group' — and then it parallelizes independent resources. In early versions, that graph was built serially, and if you had a graph with ten thousand nodes, it could take minutes just to construct. HashiCorp engineers optimized that by using a topological sort algorithm that runs in linear time relative to the number of edges. They also introduced 'provider aliases' and 'module expansion' optimizations that reduce the number of graph nodes when possible. Luna: But even with a fast graph, the real bottleneck is often the API calls to cloud providers. A plan that touches a thousand AWS resources still has to call a thousand AWS APIs. Did they change the architecture to handle that? Lucas: Yes and no. The core Terraform CLI still makes those calls sequentially in many cases — but Terraform Cloud and Enterprise introduced a feature called 'drift detection and continuous validation' that runs plans asynchronously on their infrastructure. And they've been investing in 'plan caching' — if your configuration hasn't changed since the last plan, and the provider version hasn't changed, they can reuse the previous plan output. That's a big win for CI/CD pipelines where many commits don't touch infrastructure. Luna: Interesting. That sounds like a form of memoization. Let's talk about the collaboration aspect. Originally Terraform was very single-user — you'd run plan, see output, run apply. How did they evolve that into a multi-user collaborative tool? Lucas: The biggest change was introducing 'run queues' and 'policy as code' in Terraform Cloud. Instead of every engineer running Terraform from their laptop, you centralize execution. You push a commit, the VCS trigger sends a payload to Terraform Cloud, it queues a run, runs the plan, and waits for approval. That model decouples execution from authority — the developer doesn't need cloud credentials at all. They only need access to the Terraform Cloud workspace. And with Sentinel or OPA policies, you can enforce rules like 'no public S3 buckets' or 'only use approved AMIs' before the apply even starts. Luna: That's a huge shift in operational model. It also means Terraform Cloud has to be highly available itself. What's their architecture for that? Lucas: HashiCorp runs Terraform Cloud on its own infrastructure — it's a multi-tenant SaaS. The control plane is built on Nomad and Consul internally — dogfooding their own products. They handle state isolation at the workspace level, and run plans in isolated containers. One interesting design decision is that they don't store the actual Terraform state in the same database as the run metadata. State is encrypted and stored in a separate object store, keyed by workspace. That way, even if the run database is compromised, state files remain encrypted and separate. They also have a 'state versioning' system — every time you apply, the previous state is preserved, so you can roll back to any prior state within the retention period. Luna: That's a good safety net. But rolling back state isn't the same as rolling back infrastructure. If you apply a change that destroys a database, restoring state doesn't bring the database back. How do they handle that? Lucas: They don't. And they're transparent about it. Terraform is not a backup tool. The state rollback is useful for reverting a configuration change that you haven't applied yet, or for auditing. For actual infrastructure recovery, you need proper backup strategies — snapshots, replication, etc. That's an important boundary for operators to understand. Luna: Fair. Let's pivot to a current challenge. Multi-cloud is the reality for many organizations now. Terraform's provider model handles it, but are there architectural tensions when you manage AWS, Azure, and GCP in the same configuration? Lucas: Absolutely. Each cloud has different resource models, different APIs, and different eventual consistency guarantees. Terraform's approach is to abstract those differences as much as possible, but you still see inconsistencies. For example, AWS resources tend to be strongly consistent after creation, but some Azure resources can take minutes to be fully available. Terraform's 'create before destroy' lifecycle can handle some of that, but engineers often have to add explicit depends_on or timeouts to make cross-cloud plans reliable. HashiCorp has been working on a 'resource lifecycle' specification that lets providers declare their consistency guarantees, so the core can schedule operations more intelligently. Luna: That sounds like a long-term bet. What about the competition? Pulumi, Crossplane, CDK for Terraform — are those forcing HashiCorp to evolve the architecture faster? Lucas: I think competition is healthy. Pulumi offers a more programmatic approach — you write real code, not HCL. Crossplane builds infrastructure as custom resources in Kubernetes. CDK for Terraform lets you use TypeScript or Python to generate HCL. Each has a different abstraction model. HashiCorp's response has been to double down on HCL as a domain-specific language, but also to improve the module ecosystem and terraform test framework. They recently introduced 'provider-defined functions' — callbacks that run during plan evaluation — which let providers do things like validate IP ranges or compute subnet masks directly in the configuration language. Luna: That's interesting — it's like extending the language without changing the core. What's a concrete example of where that matters? Lucas: Say you're building a VPC and you need to calculate the CIDR blocks for subnets automatically. Previously you'd need a separate tool or a local-exec script. Now the AWS provider can define a function called cidrsubnet that you call in your HCL. It's evaluated server-side during the plan, so the output is deterministic and visible in the plan output. Small things like that reduce the need for external scripting. Luna: And that's a direct result of the protocol-based plugin architecture — the core doesn't need to know about CIDR math, the provider registers the function. That's elegant. Lucas: Exactly. It keeps the core small and focused on the plan/apply lifecycle, while letting providers innovate independently. Luna: Lucas, I want to bring up something — this show is ad-free and listener-supported. A number of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's what keeps episodes like this coming. I think that's worth mentioning. Lucas: Yeah, absolutely. If these deep-dives into how companies actually build and scale their systems are useful to you, buy me a coffee dot com slash fexingo — it's a simple way to keep the show going. No ads, no sponsors, just curious conversations. And we appreciate every contribution. Luna: Alright, back to Terraform. One last question — if you were advising a CTO today on adopting Terraform at scale, what architectural advice would you give based on what we've discussed? Lucas: I'd say three things. First, invest in remote state and workspaces from day one — don't let any team run Terraform from a local file. Second, enforce policy as code early — it's much harder to add governance after you have a hundred workspaces. Third, treat providers as critical dependencies — version pin them, test upgrades in a staging workspace, and monitor the provider's release notes for breaking changes. The architecture is solid, but it's not a set-and-forget tool. You need to manage it like a platform. Luna: Great advice. And a good note to end on. Thanks Lucas. Lucas: Thanks Luna. See you next time.