Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Adobe Rebuilt Its Cloud Storage for 100 Petabytes
Transcript
- Lucas: Adobe's creative suite generates something like two billion API calls a day across Creative Cloud, Document Cloud, and Experience Cloud. And until recently, all those calls hit not one storage layer, but dozens. Luna: Dozens? They had separate backends for Photoshop assets, for PDFs, for marketing dashboards? Lucas: Exactly. Each product team had basically built its own storage solution over the years. You had Amazon S3 buckets, some on-premises NetApp filers, a couple of custom-built distributed file systems. It was a real patchwork. And the ops burden was enormous — you had to monitor ten different systems, manage ten different security policies, ten different scaling plans. Luna: So Adobe did what any sensible company would do: they decided to unify everything into one storage platform. How long did that take? Lucas: Roughly three years from kickoff to full migration. The project was called Atlas internally. The goal was a single multi-region object store that could handle over a hundred petabytes of creative assets and serve every product in the Adobe suite. Luna: A hundred petabytes — I want to get into the numbers, but first, what was the breaking point? What made Adobe finally say, 'We have to do this'? Lucas: Two things. First, storage costs were growing faster than revenue. Each system had its own replication strategy — most used triple replication, meaning for every file they stored, they paid for three copies. Second, product teams were spending more time maintaining storage than building features. The S3-based teams had no control over data locality; the on-prem teams were constantly running out of capacity. It was a strategic bottleneck. Luna: So the mandate was cost reduction and developer velocity. That's a pretty common pair of goals. But the hard part is always the trade-offs. What did Adobe have to give up? Lucas: The single biggest trade-off was moving from triple replication to erasure coding. Most people in distributed systems know the trade: replication is simple and fast for reads, but expensive. Erasure coding is cheaper — you store, say, ten data fragments and four parity fragments instead of three full copies — but it adds latency on reads because you have to reconstruct the object from fragments. Adobe's team decided the cost savings were worth it, but they had to be very careful about the encoding parameters. Luna: Right. If you pick a scheme that's too aggressive, every read becomes a math problem. Did they go with something standard like Reed-Solomon? Lucas: They used a variation of Reed-Solomon with a 10/4 configuration — ten data shards, four parity shards. That gives them roughly 1.4x storage overhead instead of 3x. On a hundred petabytes, that's a 40 percent cost saving. But to keep read latency acceptable, they did a few things: they pinned hot objects to faster storage, they cached frequently accessed fragments in memory, and they made sure the reconstruction was done in parallel across multiple nodes. Luna: Parallel reconstruction — so the latency hit is mostly on the first read of a cold object. After that, the reconstructed object can be cached. That makes sense. Lucas: Exactly. And for the hottest objects — like the current version of a widely shared Photoshop file — they kept a full replica anyway. The erasure coding was for the long tail of cold data, which is where most of the storage cost lives. Luna: Another challenge with unifying storage is metadata. When you have a hundred petabytes spread across regions, how do you answer queries like 'find all files modified by user X in the last week'? Lucas: That was one of the hardest parts. Adobe's old systems each had their own metadata store — some used MySQL, some used custom key-value stores, one team even used a giant JSON file on an NFS share. The Atlas team decided to build a single distributed metadata index using CockroachDB. It's a distributed SQL database that handles multi-region replication and strong consistency. Luna: CockroachDB — that's an interesting choice for metadata. A lot of people might default to something like Cassandra or DynamoDB for scale. Why SQL? Lucas: Because the queries they needed were relational. They weren't just looking up files by ID; they wanted to do things like 'find all assets in project X that were created after date Y and have been accessed less than Z times.' That's a three-way join. Doing that in a NoSQL store would have required multiple round trips and client-side logic. CockroachDB let them express those queries in SQL and get strong consistency across regions — at the cost of some latency on writes. Luna: So writes are slower, but reads are fast and correct. For a metadata store that mostly serves reads — file lookups, listings — that's a good trade. Lucas: Right. And they partitioned the metadata by user ID, so most queries only hit one partition. Cross-partition queries are rare and can be slower, but the 99th percentile latency stayed under 50 milliseconds. Luna: Now, the migration itself — moving a hundred petabytes from dozens of systems to one new system without downtime. How do you even begin? Lucas: They did it gradually, product by product. The first product to migrate was Adobe Fonts — relatively small, low risk. They built a proxy layer that could read from both the old and new storage simultaneously. Writes went to both systems for a while, then they cut reads over to the new system, and finally they decommissioned the old storage. Each product took a few months. The whole migration took about two years. Luna: But during that dual-write phase, you have to handle consistency issues. What if a write succeeds on the new system but fails on the old? Lucas: They used a reconciliation job that ran every night and compared checksums. If it found a discrepancy, it would overwrite the old system with the new system's version. The key insight was that the new system was the source of truth from day one of migration. The old systems were just caches. Luna: That must have required a lot of trust from product teams. 'Hey, we're going to start writing your data to a new system, and you have to trust that it's correct even though we're still reading from the old one.' Lucas: Absolutely. And that's where the engineering leadership really earned their stripes. They ran shadow reads — every request to the old system was also sent to the new system, and they compared the results. They did that for months before cutting over. They also had a rollback plan: if something went wrong, they could flip back to the old system in under an hour. Luna: I'm curious about one more technical detail: conflict resolution. Creative Cloud has a lot of collaboration features — two people editing the same file from different devices. How does Atlas handle that? Lucas: Atlas itself doesn't handle conflict resolution. It's a dumb object store — it stores bytes and metadata. The conflict resolution happens at the application layer. But Atlas does provide a feature called 'version vectors' that gives applications enough information to implement crdt like merge strategies. Each object stores a list of pairs, so when a write comes in, the application can check if the user's version is based on the latest state. Luna: So the storage layer provides the primitives, and the application layer provides the semantics. That's a clean separation. Lucas: Exactly. And it's worked well — they've had very few data loss incidents since the migration. The biggest lesson from the whole project, I think, is that unifying storage isn't just a technical problem. It's an organisational one. You have to convince dozens of product teams to change how they think about data. Luna: That's a great segue into something I wanted to ask. If these conversations are useful for what you're building or running, listener support is what keeps this show ad-free and independent. You can find us at buy me a coffee dot com slash fexingo. And Lucas, what's next for Atlas now that the migration is done? Lucas: The team is now working on bringing AI inference directly into the storage layer. The idea is that when a new file is uploaded, the storage system can automatically generate a thumbnail, tag it with content labels, and even detect if it contains inappropriate material — all without the data ever leaving the storage cluster. They're using NVIDIA GPUs co-located with storage nodes to run small models on the fly. Luna: That's smart. Instead of moving petabytes of data to a separate AI pipeline, you bring the AI to the data. I can see that being huge for things like automatic asset tagging in Experience Cloud. Lucas: Exactly. And it's still early days — they're targeting a 2027 rollout. But if it works, it could change how creative platforms think about storage. Not just as a place to keep files, but as an active processing layer. Luna: I love that vision. Lucas, great episode. Thanks for diving into the details. Lucas: Thanks, Luna. And thanks to our listeners. See you next time.