Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Salesforce Managed Technical Debt With Architecture Contracts
Transcript
- Lucas: When we talk about technical debt in most companies, it usually sounds like a warning shot. Someone points at a messy legacy module and says we need to refactor before the whole thing collapses. But at Salesforce, they didn't treat it as a crisis to be survived. They treated it as a contract to be enforced. The shift happened around two thousand twenty three when they introduced what they call architecture contracts for all new services entering their production environment. Luna: That is a fascinating pivot. Usually, architecture reviews feel like bureaucratic speed bumps that slow developers down. How did they make this feel like a necessary step rather than just red tape? Lucas: They made it non-negotiable but also highly automated. The core idea was that if you want to deploy a new microservice or modify a critical path API, you have to submit an architecture contract. This isn't a fifteen-page document written by lawyers. It is a structured digital form that forces the engineering team to declare their dependencies, failure modes, and recovery strategies upfront. If the answers don't align with their core engineering principles, the deployment pipeline simply blocks the merge. Luna: So it acts as a gatekeeper at the code level. Did this actually reduce the volume of incidents, or did it just make the teams happier because they had clearer guidelines? Lucas: It did both, but the incident reduction was the headline metric. By forcing teams to think about failure modes before they wrote the first line of production code, they caught structural flaws early. For example, if a team proposed a new service that relied on a synchronous call to a database that wasn't sharded, the contract would flag it immediately. They saw a forty percent drop in critical incidents related to scalability issues within eighteen months of rolling this out fully across their enterprise division. Luna: That is a significant operational win. But I imagine there was pushback from the product side who wanted speed. How did they handle the tension between rapid feature delivery and this new layer of governance? Lucas: The key was that the contracts were designed to be fast. Most took less than ten minutes to fill out if you understood the principles. The engineering leadership framed it as protecting the platform so that product could move faster in the long run. If you spend all your time fixing bugs caused by poor architectural choices, you can't ship features. The narrative was very clear: governance buys you velocity later. Luna: I see. So it was really about shifting the cost of failure earlier in the cycle. Was there a specific principle that caused the most friction during implementation? Lucas: Yes, the principle of explicit timeout handling. In many large organizations, default timeouts are ignored until a system is under load. Salesforce mandated that every external call must define its own timeout and fallback behavior. Teams that had been relying on network resilience or lucky breaks had to go back and harden their code. It was uncomfortable for a lot of senior engineers who were used to working around these constraints. Luna: It makes sense that older habits die hard. Did they provide any tools to help teams automate the validation of these contracts so it didn't become a manual review process? Lucas: Absolutely. They built an internal tool called ArchGuard. It integrates directly into their CI/CD pipeline. When a developer pushes code, ArchGuard checks the submitted contract against a library of known anti-patterns. It runs static analysis and dependency mapping automatically. If everything passes, the contract is approved instantly. Only edge cases that require human judgment are escalated to a principal engineer for review. Luna: That sounds like a much more scalable model than having a committee of architects sitting in meetings all day. How did they measure the success of this initiative beyond just incident counts? Lucas: They looked at mean time to recovery as well. Because the failure modes were predefined in the contracts, the automated remediation scripts could be pre-approved and ready to go. When something did go wrong, the recovery path was already documented and tested. They cut their average MTTR by twenty five percent. That is huge for a company of that size where even small delays ripple across millions of users. Luna: Twenty five percent is impressive. It feels like they turned architecture from a soft skill into a measurable engineering discipline. Do you think other companies can replicate this without being Salesforce? Lucas: They can, but they need to start small. You cannot roll this out to every team at once. Start with the critical path services. Identify the top three services that cause the most support tickets or downtime. Require architecture contracts only for those. Prove the value there, then expand. If you try to boil the ocean, you will just create resentment. Luna: Practical advice. What about the content of the contracts themselves? Are they proprietary to Salesforce or can we look at the general structure? Lucas: The structure is public. It typically covers four areas: data flow, error handling, security boundaries, and observability. Each section has required fields. For example, under security boundaries, you must declare how secrets are stored and rotated. Under observability, you must specify which logs and metrics will be emitted. It forces clarity. Most teams realize halfway through filling it out that they don't actually know how their system works yet. Luna: That self-awareness is valuable. It stops people from building things they haven't fully thought through. Have you seen similar approaches in other industries, maybe outside of pure software? Lucas: Interestingly, yes. The financial sector uses similar concepts in risk management. Before launching a new trading algorithm, banks must file a risk contract detailing the limits and fail-safes. It is not unlike what Salesforce did for code. It is about pre-authorizing risk within defined bounds rather than reacting to chaos after the fact. Luna: I like that comparison. It frames technical debt not as messiness but as unmanaged risk. That is a language executives understand better than 'clean code.' Lucas: Exactly. And it gives CTOs a lever to pull. Instead of asking teams to 'be more careful,' you can point to the contract and say this requirement was not met. It depersonalizes the conflict. It becomes about compliance with agreed-upon standards, not personal criticism of the engineer's work. Luna: That reduces defensiveness significantly. Let me pause here for a quick moment to ask our listeners a question. If these conversations are useful for what you're building or running... Lucas: Honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. It helps keep this ad-free and allows us to dig deeper into cases like this one. Luna: Thanks for supporting the show. Now, getting back to the contracts, did they ever find that the contracts became too rigid over time? Lucas: They did initially. About six months in, some teams started gaming the system. They would write vague answers to get past the automated checks. The solution was to introduce random audits. A rotating group of senior engineers would pick five contracts per month and do a deep dive into the actual code versus the declared intent. If they found discrepancies, the team had to redo the contract and attend a refresher session. Luna: Spot checks. That keeps everyone honest. Did the audit findings usually reveal genuine risks or just sloppy writing? Lucas: Surprisingly often genuine risks. They found teams that had changed their infrastructure underneath the contract without updating it. Or teams that assumed a third-party provider had certain SLAs that they actually didn't. The act of filling out the contract forced a reconciliation between reality and documentation. Luna: That reconciliation is probably the biggest value add. Keeping the map aligned with the territory. How does this compare to the approach Atlassian took with their technical debt management? Lucas: Atlassian focused more on cultural incentives and dedicated sprint capacity. Salesforce focused on structural enforcement. Both work, but they serve different organizational maturities. If you have a young team, you might need the cultural nudge. If you have a sprawling enterprise with inconsistent practices, you need the structural guardrails. Luna: Good distinction. For a mid-sized company listening right now, which path seems more viable? The cultural one or the structural one? Lucas: Start with the structural basics. You don't need a full contract system. Just require a one-page design doc for any project larger than two weeks. Make sure it includes a failure mode section. That single requirement will likely catch more issues than any amount of retrospective cleanup. Luna: Simple and actionable. It raises the bar just enough to force thinking without adding bureaucracy. One last thing on this. Did Salesforce share these lessons publicly or was it kept internal? Lucas: They shared a lot in their annual re:invent presentations. They were proud of the results and wanted to position themselves as leaders in engineering maturity. It was part of their broader brand story about reliability and trust. Luna: Which makes sense. Trust is their product. Demonstrating rigorous engineering supports that claim. It turns internal practice into external marketing. Lucas: Precisely. And it attracts better talent. Senior engineers prefer working in environments where good practices are enforced, not just hoped for. It signals that the company values sustainability over short-term hacks. Luna: That is a powerful incentive for leadership. It aligns the interests of the engineers with the goals of the business. So, as we wrap up, what is the one takeaway for someone managing a tech team tomorrow morning? Lucas: Stop treating technical debt as a separate backlog item. Bake it into the definition of done. If you can't explain how your system fails, you aren't done. Make that a rule. Luna: Clear and firm. Thanks for breaking this down, Lucas. It gives us a concrete model to think about instead of just vague ideals. Lucas: Anytime. It is worth considering whether your current processes allow for that kind of upfront clarity, or if you are still reacting to problems as they emerge.