Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Palantir Rebuilt Its Gotham Platform for AI-Native Decision Making
Transcript
- Lucas: Palantir's Gotham platform has always had a reputation for being the most demanding enterprise software deployment on the planet. Custom ontology models, strict access controls, and a data pipeline that has to ingest everything from satellite imagery to supply chain logs. Luna: And now they've rebuilt it for the AI era. I've been reading about their 'AIP' — Artificial Intelligence Platform — integration. That's a big architectural shift. Lucas: It is. And what's interesting is that they didn't bolt on AI. They rebuilt the core ontology layer to be ai native. So instead of a human analyst writing queries against a structured model, the platform now ingests natural language requests and maps them to the underlying data graph in real time. Luna: That sounds like a massive change to the query engine. How do they handle the translation from natural language to ontology queries without breaking the security model? Lucas: That's the clever part. They introduced what they call 'ontology grounding'. Every llm generated query is constrained by the user's existing permissions. So even if the model tries to pull data outside the user's scope, the ontology layer rejects it at the graph level. It's not just an API guard — it's baked into the semantic layer. Luna: So the AI can only see what the analyst can see anyway. That's a lot more robust than the typical 'filter results after retrieval' approach. Lucas: Exactly. And the performance numbers are striking. Palantir claims they reduced decision latency — that's time from question to answer — by about 40 percent for their core defense clients. And they're handling over 10 million daily queries now, up from about 3 million before the rebuild. Luna: Ten million queries a day is a lot of load. Did they have to rearchitect the underlying data pipeline too? Lucas: They did. The old Gotham was largely batch-oriented — data landed in a data lake, got transformed, then loaded into the ontology. For the ai native version, they moved to a streaming architecture. They're using Apache Kafka as the backbone, with Flink for real-time enrichment. That allows the ontology to update in near real time as new data arrives. Luna: So an analyst can ask 'show me all convoy movements in the last hour near this grid coordinate' and get an answer based on data that's seconds old. Lucas: Right. And that's a huge shift for Palantir's traditional customers. The military and intelligence communities are used to working with data that's hours or even days old. Now they're getting sub-minute latency. But it also creates new challenges around data fusion — merging streaming data with historical archives in the same query. Luna: How do they handle that? You can't just join a real-time stream with a petabyte-scale historical store and expect sub-second response times. Lucas: They use a tiered storage approach. Hot data — the last 72 hours — lives in memory and ssd based caches. Warm data — up to a year — sits on faster object storage. Cold data goes to cheaper archival storage. The query planner automatically routes sub-queries to the right tier based on time predicates. They published a paper showing that 95 percent of queries hit only hot and warm tiers, so the cold tier latency is rarely a bottleneck. Luna: That's smart. And it probably saves a ton on storage costs. But I want to ask about the LLM part specifically. Which models are they using, and are they running them on-premises or in the cloud? Lucas: Palantir is model-agnostic. They support OpenAI's GPT-4, Anthropic's Claude, and their own fine-tuned models. For classified environments, they run everything on-premises using something they call 'AIP on Hyperscale' — basically a hardened Kubernetes cluster with NVIDIA GPUs. For commercial clients, they use a hybrid approach: sensitive data stays on-prem, but the model inference can happen in the cloud if the client allows it. Luna: So the architecture has to support both deployment models seamlessly. That's a lot of abstraction. Lucas: It is. And the key abstraction is what they call the 'Ontology SDK'. It provides a consistent API that works whether the LLM is running locally or remotely. The SDK handles the grounding, the permission checks, and the query translation. The LLM never sees the raw data — only the ontology objects that the user has permission to access. Luna: That's a nice security boundary. But I wonder about the latency of that grounding step. Adding an extra layer of permission checking on every query must add overhead. Lucas: It does, but they've optimized it heavily. Permission checks are cached in a distributed Redis cluster, and they use bloom filters to quickly reject queries that would definitely fail. In their benchmarks, the grounding overhead adds less than 10 milliseconds on average. For most use cases, that's negligible compared to model inference time, which can be seconds for complex queries. Luna: So the big win is that analysts can now ask questions in plain English and get answers that are both fast and secure. That's a game changer for organizations that have huge amounts of data but not enough data scientists to query it manually. Lucas: Exactly. And Palantir is seeing that play out in commercial sectors too. They recently signed a deal with a major pharmaceutical company to use Gotham for drug supply chain optimization. The same architecture that works for battlefield logistics works for vaccine distribution. Luna: That's a fascinating crossover. But it also raises the question: how customizable is the ontology for different domains? A military ontology is very different from a pharma one. Lucas: Palantir's whole model is that the ontology is built per customer. They have a library of 'ontology templates' for common domains — defense, finance, healthcare — but the actual schema is always tailored. The ai native rebuild actually makes that customization easier because the LLM can help map domain-specific terminology to the ontology. So instead of a human spending weeks defining data models, the model can propose mappings based on existing documentation. Luna: That's a huge productivity gain. But it also means the ontology is partly generated by AI, which introduces potential for errors. How do they handle that? Lucas: They have a human-in-the-loop validation step. The LLM proposes a mapping, but a domain expert has to approve it before it goes into production. They also run automated tests that compare query results with known ground truth. If a mapping leads to incorrect results, it gets flagged and rolled back. Luna: So there's a safety net. That makes sense. I'm also curious about scalability — ten million queries a day is impressive, but what happens when that grows to a hundred million? Lucas: That's the trillion-dollar question. Palantir's architecture is designed to scale horizontally — they add more nodes to the Kafka cluster, more Flink workers, more Redis shards. But the real bottleneck is the LLM inference. If every query requires a model call, the GPU cost becomes prohibitive. So they've introduced a caching layer for common queries — identical questions get a cached result, and only novel queries go to the model. Luna: That's pragmatic. How big is the cache hit rate? Lucas: In their internal tests, about 60 percent of queries are repeats or near-repeats. So the cache significantly reduces load. And they're also experimenting with smaller, domain-specific models for routine queries, reserving the large models only for complex ones. Luna: So they're essentially building a tiered AI system, not just a flat one-model-fits-all. That's smart engineering. Lucas: It is. And it's a reminder that the big architectural trends we've been covering on this show — streaming, tiered storage, human-in-the-loop, model caching — are all converging in platforms like this. Palantir's rebuild is a case study in how to make AI work in production at scale. Luna: Absolutely. And if these kinds of deep-dive conversations about architecture and engineering decisions are useful for what you're building or running, we just wanted to mention that listener support is what keeps this show ad-free and independent. Lucas: Yeah, it's a small thing, but it makes a big difference. If you find value in episodes like this, you can support us at buy me a coffee dot com slash fexingo. That's it — no pressure, just a way to help us keep going. Luna: And it really does. So back to Palantir — one thing I didn't ask: what's the biggest lesson for other engineering teams rebuilding their platforms for AI? Lucas: I think the biggest lesson is that AI integration isn't just about adding an API call. It's about rethinking the entire data and security architecture. Palantir succeeded because they rebuilt the ontology layer from the ground up, not because they added a chatbot on top. That's the hard work that most teams underestimate. Luna: So the takeaway is: start with the data layer, not the model layer. Lucas: Exactly. The model is just the interface. The real magic is in how you connect it to your data and your rules. And that's what Palantir's new Gotham shows us.