Latest / Construction Technology Podcast / Construction Technology Podcast - Episode 010 - with Andriy from Nomic.ai
Transcript
- Luigi La Corte: Hello everyone. Welcome to the construction technology podcast. Luigi CEO at Provision. I'm joined by my cohost Mike from BuildVision and Andrew from JOIN. We a special guest today. It's Andre from Nomek AI. Andre, how are you? Andriy Mulyar: Hey, Luigi. Hey, everyone. Thank you so much for having me. I'm Andre, founder and CEO at NOMIC. We build agentic systems that accelerate design and engineering and construction. Luigi La Corte: Andre, great to have you. Obviously we did some research prior to you joining. We know that you pivoted into AEC are now selling, it looks like a variety of products, some ML APIs that help what seem to be technology companies have a more programmatic understanding of construction drawings and construction documentation. You've already the pivot into AEC in previous interviews. But maybe for our audience, which mostly skews towards technology construction executives within the technology space, just give us an overview of what was attractive about the industry and what led you there specifically. Andriy Mulyar: Yeah, that's a great question, and obviously we get this a lot. So a little bit around a year and a half ago, I would say, is really when we fully embraced a pivot into a verticalized AI space. And there's a lot of calculation that went into this. So Gnomic, were founded a little bit under four years ago now. We started out as a horizontal AI infrastructure company. We built AI models for search, AI models for document understanding. Some of them are extremely popular to this day. millions of downloads on open source platforms. this sort of infrastructure play, while we getting lots of, let's say, open source traction and lots of users using our products and software, we were finding, it's extremely hard to commercialize as a business because we took a very big open source approach to the whole strategy of the company. I often talk to people about, you're building an organization, have to not only, ⁓ solve several hard problems in over and over again and reach increasingly new harder problems to solve. You have to align yourself with the right exponentials that are happening in the greater environment. For example, four years ago the right exponential was not building ⁓ agents. The exponential was frankly helping the world figure out the places where AI was working and AI was not working. ⁓ I think as of like a year and a half ago, if you're an organization that's not hyper focused right now on building agents or helping organizations orchestrate Luigi La Corte: Hmm. Andriy Mulyar: trade agents, you're probably working on the wrong exponential. So about a year and change ago, we were sort of really, really thinking about what does it mean to build the next great generational company that unlocks this new transformational wave of technology for a wide audience of individuals. We were thinking about things like, what is the hardest data domain right now where AI systems just don't work, right? place where there's insane product market fit right now with AI systems is places where the input is mostly textual and the output is mostly textual. Things like code. Those are the places where domains are highly verifiable. What this means is that AI systems can progress and evolve in these data domains like coding. You can build complete software applications with a single prompt. But in places like design and engineering and pre-construction, the kind of tasks you have to solve are highly multimodal. What this means is that you need to operate over top of images, need a topic that will operate across vector graphics, you need to perform operations that language models are really bad at. Exhaustive counts, for example, if you're doing a takeoff. These places where ⁓ language models are not that good, just innately how the way they operate. They're not exhaustive. And as we were considering this and also looking at ⁓ sort of the market trends, it just became really clear to us that ⁓ the direction we to go in ⁓ was to go and solve the hardest data problems that existed out there. And it turns out hardest data problems don't exist in the digital world. ⁓ the ⁓ environment in the physical world. So actually set off and for about six months kind of put our head in the dirt. We're building out custom models that kind of close the gaps between what foundation models could do and operate and working quite closely with a sequence of design partners to really nail out what are the high ROI relative to AI usage cost applications and use cases that are universal regardless of whether you know you're a professional service firm on the Luigi La Corte: Hmm. Andriy Mulyar: and east side or running a traditional like GC. So a lot of what we do today, a lot our product form factor is built around out of these learnings as we sort of kind of entered a new evolution of the company about a year and a half ago. Hyper focused ⁓ on solving the most critical and problems that slow down the design and construction of physical assets. And frankly, what we realized over course of of that six to eight month kind of build period was that it's all around project coordination. of the build out and I think kind of the impetus for this conversation was a recent launch of AECbench, which is kind of an open source benchmark measure, it's a measuring stick for AI agents on project coordination tasks. A of that came out from our learnings from that too. Luigi La Corte: That's a fantastic background. interesting note about how this podcast came together, Mike and I had a disagreement on LinkedIn ⁓ whether or not LLMs could quote unquote, ⁓ estimating tasks, plug play off the shelf. And obviously took the position that they couldn't and Mike was, you know, along and took the position that it could. you know, maybe Mike has changed his perspective a little bit on this, but Mike Powers: you Luigi La Corte: I do think there are inherent limitations in LLMs. And you go into document understanding, like civil recognition, there's only so much that these LLMs can do. And I tee this up because there are a group of people who believe that LLMs will completely subsume, let's say the takeoff coordination estimating space ⁓ out the box. An example of this may be people pointing to certain advertisements from a company called Perplexity, which is saying, we can do estimates for you. Now to do estimates, you need to have cost data, you need to be able to do symbol recognition, you need to be able to do area understanding, these are completely different problems. As part of what we wanted to do here was kick off a process, which I'll hand over to Mike, to see how perplexity does for a project that Mike has a ground truth for. And I think it'd be an interesting experiment as we continue to dig into your background and obviously in NoMix capabilities. I'm obviously expecting this to fail, partially because I've seen the results. ⁓ but we love your perspective on that divide, Andre, and maybe, maybe even commenting on why certain people think it works versus your obviously in-depth understanding of where it fails. Mike Powers: Yeah. So let, before you go into that, let's kick it off just because it takes time. So I think everyone saw this kind of floating around LinkedIn. ⁓ So Perplexity is using some, guess we consider GEO, ⁓ generative, like engagement ⁓ marketing. So it created this web portal, says turn Blueprints into accurate estimates within minutes. ⁓ So I'm actually gonna go ahead and choose a file. I think the first thing that it actually does that's interesting, ⁓ there's a limit on the file size to 50. megabytes. So I had to reduce this down pretty large. ⁓ But as I go through this actual workflow here, it actually doesn't do anything. It just takes you to their chat interface. ⁓ So all those steps are for it. very generative. It's very marketing-led, ⁓ which is understandable. They're super early in the process. So ⁓ I actually went ahead and just manually did this. So I created a prompt that says, your mechanical subcontractor preparing a bid proposal for this project. So I'm going to head and launch this one. I also did the same thing for Claude and Claude just released 4.7. It's going to launch that one. And then also going to launch one in ChatGPT with Extended Pro. These will create proposal outputs with the prompt and then I'll actually have it run through and do a bit level exercise. So just setting the background again, this is going to take time. So I'll set it right in the background, but I'll pass it back to you all as it kind of cooks. Andriy Mulyar: Yeah, so I think one thing Luigi said was, ⁓ He doesn't believe that language models are sort of like on the right trajectory to be able to talk solve a task is both diverse across disciplines, right? There's hundreds of different tools and solutions out there for every single unique discipline. You know, there's electrical takeoff software that has product market fit. There's mechanical takeoff software. And the reason you need this is because I think in the previous generation of software constructed ⁓ very based, obviously human engineered. had to have, you had to carve out all the artifacts and aspects of every single discipline to be able to do a task like a takeoff. I actually kind of disagree with you, Luigi. I think we'll reach a point. It's a ⁓ much harder task than task like, let's marking up a drawing relative to a set of rules or a standard doc, because a takeoff is a multi-step that involves complex like calculations, like exhaustive counting, where if you mess up, you mess up the end result. It involves not just ⁓ a, oftentimes not sufficient to purely look at the drawings or plans, ⁓ even though kind of like the intent to go in perform it. ⁓ All these kind things put together make it something that if you go in and kind of go to one of the out of the box agents out there, so the agent that powers ChatGBT, so it has access to web search it has access to maybe a code sandbox to write some code to perform a calculation. or you go into Claude like Mike just did, you're getting basically a very nerfed version of these systems that don't do a good job at running for a long time, especially in domains and complex tasks. ⁓ From what we've been seeing at least, it's more of ⁓ a question how much are you willing to experiment with longer range running tasks that might burn you a couple of hundred dollars of AI spend, right? ⁓ What is actual human labor that goes into performing a takeoff? right? Potentially you're spending thousands of dollars of either billable time or company time to perform a takeoff task. Is it warranted to spend $200 of AI time or $300 of AI time to do this? The answer is probably yes, but you're going be hard-pressed to find a system out there that lets the average internet user accidentally burn $200 before seeing an output to really test out the capabilities of this thing. Andrew Zukoski: Can we, yeah, just I have a question. What do you mean you're building custom models? Like, is that, ⁓ the major, the foundational models spend billions of dollars trading a model from scratch. What is a custom model to you? Andriy Mulyar: Yeah, so whenever you're building an agent, you're kind of... layering on three things. You have the underlying foundation models. Like what is the actual driver model that is doing the tool calls, bringing information into context to go in and make a decision. So in this screen we're looking at right now, this prompt ⁓ into Claude, this is running, the actual driver model that's being running here is Opus 4.7. ⁓ That base model, you can pick whichever foundation model provider you want. You can go with OpenAI, you can go with ⁓ Gemini, you can go with GPT from OpenAI. They all can go in, call tools, bring information to context to solve a task. ⁓ The next layer of information ⁓ is the actual tool that the model has access to. So for example here, Claude clearly has access to the ability to write a bash script, write some code, execute some code. That's a tool that not every single ⁓ language model harness, so agent, has access to. And that's going to give it differentiating capabilities to be able to solve harder tasks than a different agent. can solve. And then finally, you have the environment which the agent operates in. If the agent is, for example, marking up a PDF, you want the agent to be able to produce markups that maybe are compatible with review software, like Bluebeam Review. So it has to operate in a sandbox environment that matches the environment that a human lives in. All these three levels are places where people who are building agents, especially agents for the built environment, have to think very carefully about tinkering and configuring and adding custom capabilities to be able to sort of close the capability that's native to foundation models because they haven't seen in their training distributions or their training data sets, you know, many takeoffs, for example. I think we're talking about takeoffs here. So what we do very specifically is we work in that second layer, the tool layer. Our custom models can go in and take a drawing, like a spec PDF, and transform that into a form by going in and performing various layers of information extraction, things like creating aggregate counts of symbols that allow for the underlying foundation model that then makes a tool call to not have to say, okay, I'm going to then spend a bunch of time now trying to figure out how to accurately count all these 97 symbols across 50 sheets. It's going to have that information pre-computed with another system that's more rules-based. And then these kind of tools allow it to be a lot more accurate, a lot more consistent, and spend a lot less cost to achieve that end outcome. So our domain-specific models, they're something called parsing models. So they go in and ⁓ they basically go in and take a drawing, take every single detail, every single table, every single legend, ⁓ structure it out of the document, and then provide it as a interface for agents to use to be a lot more effective, so basically ⁓ to generate a higher quality ⁓ set of actions while also using less tokens, so also being cheaper. And we find in our end, especially, so nine months ago, this was basically critical to make anything work at all, ⁓ because the foundation models were frankly absolute garbage when it came to the multimodal inputs that govern any kind of project coordination tasks, especially including drawings. They're getting a lot better now. This is one of the reasons that we build benchmarks internally, because we want to know when we can turn off our expensive domain-specific models and rely more on the foundation models as we deliver our offering. ⁓ But hopefully that gives you a little bit of a better picture of when any company out there say they do domain-specific models in a world where you have billions of dollars of investment going into foundation models, they likely mean that they're operating and injecting custom tools or custom environments into the underlying agent that they're using to deliver that outcome and that task at a higher quality and most likely in less time with less token budget. Mike Powers: So just like step out and even simplify that a step further. Cause I think a lot of like listeners on this call ⁓ are like seeing, know, cursor, sorry, seeing anthropic, seeing the names open AI. So there's LLMs, which are these large foundation models that have, ⁓ you know, have been pre-trained with billions of dollars to create different metrics and different knobs internally to kind of next guest tokens. in ⁓ a very, very simple as possible as your kind of next guessing tokens, right? So in your position, your perspective, you're not changing that next guess token. You're taking these foundation models and then you're almost like layering on top the ability to go plug into a blue beam and use blue beam as a tool ⁓ to do X, Y, and Z. You're structuring down data and saying, hey, we're gonna name this file schedule. So when you talk to call a schedule as opposed to the foundation model going through the thousands of pages. Again, it's going to know exactly to go to schedule in a very simplistic way. Andriy Mulyar: Yeah, 100 % and it's not just us, any agent builder building something that works is doing something like this. The biggest thing that we differentiate on is that our tools that the agent is using are actually custom AI models that do a first pass before the foundation model even touches the data to allow it to have a much higher quality interaction. Mike Powers: Yeah, yeah, yeah. Luigi La Corte: So Andre, if you go back, let's go back to the original premise of how we started this. know, we, I have a belief that let's say LLMs currently as they exist could not do something like a takeoff very accurately. It sounds like you disagree. I just want to understand your disagreement. Do you disagree that an LLM by itself out of the box? would not be able to because what you're describing obviously is a harness. The harness that you've built within Nomek AI uses different models. Maybe you're using a visual language model or maybe using custom models or some kind of computer vision model. But doesn't that agree with my point that LLMs as is cannot out of the box do a takeoff accurately. Andriy Mulyar: ⁓ okay. 100 % agreed. If you're just going in and sampling from a model that you give it a bunch of text and it generates out the next half word, 100%, it's not gonna be able to do a takeoff. But I don't think a single human ⁓ in this room has, ⁓ I personally have it in three years, interacted with a raw LLM. You're always using something like ChatGPT or Claude, which is actually an agentic harness that's been hyper-tuned to give you happiness to use it so you come back. Luigi La Corte: Yeah, that's a fair point. So what is the Nomek AI product? I mean, obviously we went to your page, we see your products page. It sounds like you have a few products. You have the harness, you have custom models, you have APIs. If let's say the CIO or CTO of a general contractor wanted to improve their quality assurance process and they said, Andre, help us. What exactly would you be plugging into their system and how would you do Andriy Mulyar: For sure, yeah. ⁓ Our core IP at Nomek is our agentic harness ⁓ that ⁓ produces ⁓ outputs and works with designers, engineers, pre-construction teams to deliver ⁓ tasks and work output that is acceptable in quality, so it doesn't require a bunch of rework or redoing the task ⁓ across a broad range of project coordination tasks. So on the design and engineering side, our agent can go in and automatically review a drawing set against a standards doc, so an internal standards doc, a client standards doc, international codes and standards that our agent has access to, and not produce a table of like, here are the problems, it'll actually mark up the drawing in the exact same markup formats where you can drop it into Bluebeam and go look at it in the markup bar. That same agent harness that we build can also go in and review a submittal for you. It can also go in and integrate if you're integrated with a project delivery software, so on the horizontal side, know, something like a Bentley, ⁓ or on the kind of general purpose side. ⁓ Autodesk Construction Cloud, pull information automatically from the APIs of your setup project in those systems, run those operations, and push that information back in with human approval. So our core product and our core capability is that we're building an agent that can perform design engineering and pre-construction tasks across a wide range of different tasks that happen across project stages and life cycles. So 30%, at 60%, at 90 % on the design and engineering side. after IFC when you're going in and doing progressive takeoffs on the pre-construction side and all the coordination that follows that. ⁓ But that's kind of our core IP in technology. Our actual product and platform ⁓ is a platform that allows for ⁓ design and engineering firms and design and engineering teams at GCs to go in and orchestrate those agents ⁓ extremely effectively. So we have the agent, our actual platform and product allows allows your teams to onboard to collaborate with that agent. Give it, you you can log in, give it a drawing. Within two minutes, you're getting markups ⁓ pushed back to you on that drawing that you can export out, keep using somewhere else. And most importantly, we're deeply integrated with ⁓ all of the places where your ⁓ source of truth and project delivery state sits. So on design and engineering side, all the project delivery software, ACC, Bentley, A-Connex, all the way down to the contracting side places like field wire and Procore. This is really important because for a human to be able to accept an agent's output, that agent must start work in the places and in the formats that humans start work. So if it's a gigantic submittal package that comes in in a zip file, guess what? Like that agent better take that zip file. ⁓ And it has to end work in the same outputs and formats as the human will accept it. So if that means PDF markups, it has to generate PDF markups. It can't generate a table. If that means an Excel file, with 10 sheets formatted with the right headers for every single schedule that exists in the drawing if you're performing a takeoff, it better do that otherwise you're going to make the human redo work. We focus really aggressively in making sure that ⁓ we meet all those conditions because that's what I truly believe is needed to allow for ⁓ the skilled professionals in the built environment to be able to collaborate and work with agents effectively. Luigi La Corte: Fantastic. Andrew Zukoski: So Andre, you've got a pretty strong academic background in computer science. Are you familiar with Rich Sutton's Bitter Lesson essay? This was here. I'm going to share my screen and take over. ⁓ Andriy Mulyar: Yes. Mike Powers: start with. Andriy Mulyar: I I read that the first time when I was like 16 or 17. Andrew Zukoski: Yeah, great. And like you probably could have because it's just like, it's just text here. Like, ⁓ and so I'll, I'll, I'll just summarize it quickly. ⁓ and then I'm really interested in your take on it how it applies like to what you're doing in nomic. But the, bitter lesson of Rich Sutton is that all that matters is scaling applied compute that there's a, this long, long history of humans taking a like human researchers taking some capability in a computer and saying, like, ⁓ if I teach it about this domain, if I inject my special knowledge on top of this generalized technique, I will be able to build something that will solve the problem and will always like be the best. And you need my like specialized ⁓ expertise to be embedded in the system. And like the example, the example he gives is chess, right? Where people in building chess engines for probably literally 70 years. And they eventually won when they abandoned anything about chess inside the code base. And it was this very deep search place. And then Google did this with AlphaGo as well, where there's no code that understands the Go rule set. It's all learned, and it's all this scaled compute. I know a few other companies in the the AEC space where circa 2022 or three or four, whenever it was, they were using models that they have trained themselves, like that they had built to understand code or understand contracts or understand drawings, like whatever it is. And then it became clear that chat GPT like was better or was very quickly going to be better. they, they switched over to like leverage these foundational things. And so I'd love to hear your take. Andriy Mulyar: Yeah, so this is by the way one of the most cited papers when it comes to how you should go in and kind of, if you're building a product or you're building some sort of technology over top of the latest wave of tech, how do you build that correctly so that you don't end up on what I said earlier, the wrong exponential, right? And what Richard Sutton is saying in this piece is essentially the only thing that you can bet on, which has been consistently true across the 60 year history of and leaving the research space and entering ⁓ general population's consumption ⁓ that everything you do needs to be hyper aligned with the idea that every couple of years the power of systems, whether it's processing power on CPUs or processing power on GPUs is going to improve. And you to build methods that scale in quality as computing power scales. So an example of one of these methods is deep learning, which allows for you to in, give a large set of tasks, a data set. You go in and plug in a really big model. You plug in bunch of really big connected GPUs over top of that model. You spend a bunch of time burning a bunch of electricity. And then out comes out a model that can go in and perform the tasks that you gave it as input. And this is one of the reasons you've seen insane of this technology is because it's ⁓ a for computers learning to perform actions like humans learn actions that scales really well the more compute you inject into it. The bigger data center you build, the higher quality the end model that you get. The more tasks you put into it that cover distribution of tasks you want the model to be able to perform. So when I distribution tasks, a task might be do a takeoff. go in and answer an RFI accurately ⁓ to a folder of project files. Mark a drawing relative to like a standard. Those would be examples of tasks. As you in and produce more of those tasks for the model to train on, you can go in one year just to that model and then as computing capabilities improve, two later if you just perform that same retrain, you're going to have a higher quality system that comes back out. That's what Richard's system is saying here. Andrew Zukoski: OK, and so for Gnomic, leveraging these foundational models, and so you can plug and play them. And so as new foundational models come out, your system gets better. ⁓ But I think the harness layer. That layer also moving pretty quickly, right? Andriy Mulyar: Exactly. Yeah, so, well, this is part of it, right? It's like, whole reason everyone's talking about this word harness right now is because I think everyone has deeply internalized the phenomenon that Richard Sutton's piece here is trying to describe, which is if you're going in and fighting against this idea that with the same data set years later on a much larger computing regime is gonna produce better AI models, you're on the wrong exponential of where this technology is going. The reason we go in and we've basically built our product all around an agenda harness is because I deeply believe in five years a lot of the work that's going to be delivered is not going to be, you people are not going to be purchasing rent in SaaS systems and paying monthly rental fee for like a seat or a license to be able to work with technology. They're often times going to be purchasing services of software. ⁓ the end outcome coming out of the technology. And that's all based around agencies actually working on the task distributions that exist in various verticals in various industries. ⁓ the where people maybe take the wrong lesson sometimes from Richard's piece here is that it doesn't mean that you don't need individuals hyper-focused on specific domains. It's only a commentary around... building systems that scale as computational capability scale, which has always been a truth the last 60 years. It's still really important to that these computers are not to replace us, they're here to empower us and us deliver the that we're doing every day better. On the software engineering there's more software engineers out there since AIs came out, not less. ⁓ Even though software ⁓ AI is to go in take a lot of tasks that software engineers do, do it end to end. people are getting a chance to do more and being able to do more in shorter periods of time. kind of seeing, frankly, exact same thing. Some of larger partners on the design and engineering side, they're able to go in and tackle more work because they're having agentic that can go in and review a submittal in five minutes as opposed to eight hours. The tasks that humans hate are oftentimes the places where agents can do a really good job at producing first passes that humans can then spend time It's really important to, ⁓ think, differentiation between Richard is saying don't build domain-specific systems, it's saying build systems that scale as compute scales. But systems need to understand very deeply where humans start work and where humans end work for humans to find them useful and accept them into their daily work streams. Andrew Zukoski: that like, ⁓ that we even coded the workflows ⁓ of AEC, could argue poorly into ⁓ software. trick is it's in ⁓ all ⁓ systems you mentioned. It's in the ACCs, it's in the Bluebeam, right? Like Bluebeam is you understand and mark up drawings, right? And so if that's the front end, for the workflow. I guess the most challenging version is we like, why do I need Nomek? Like if over time, over time Bluebeam, they bought Formic, Forma, whatever it was. ⁓ Formis, yes, Formis, yes, sorry. Haven't had enough coffee. of those F names. And work is still gonna happen in Bluebeam ⁓ the... Luigi La Corte: Firmness. Mike Powers: Yeah. Andrew Zukoski: most differentiated capability are remain the foundational models. Like everybody's calling into the same foundational models. Like, why do I, yeah, why do I need Nomok? Andriy Mulyar: Yeah, So everyone's calling into the same foundational models, but everyone is most certainly not using the same harness around the foundational model. And as I said, our core IP is not the fact that we were clever enough to use 4.7 two days before somebody else. It's the fact that we've spent over a year building a agentic harness that can capture and perform a lot of the task distribution that governs a lot of the tasks that designers, engineers, pre-construction teams do across project coordination, for example. That's our core IP. And the reality is that, one, teams in existing software oftentimes aren't working on this and aren't focused on it. The other is there's a lot of constraints that exist in those systems that prevent for this technology to really to, I think, full fruition in those. Andrew Zukoski: This is what you mentioned earlier about the sheer amount of electricity that's necessary to to ⁓ these tasks. OK. Andriy Mulyar: No, no, I mean like, just like guardrails in place when you're working with a software that's 30 years old, that kind of hinders innovation to happen at these organizations, right? Like, we go in and you can kick off, for example, one of our Nomic agents via an API. In Bluebeam, if you wanted to go in and in a comment markup, kick off an agent that would go in and generate more markups, you could do that via our agent API. And ⁓ the reason you might about this is because... ⁓ Andrew Zukoski: ⁓ okay. Andriy Mulyar: It's very hard to replicate the kinds of capabilities that we've built into harness because it requires a lot of iteration and frankly a lot of data work to be able to do well. One of the biggest things, I would say, our competitive advantage at Gnomic is that we take benchmarking to heart. When we released that ⁓ AEC benchmark, which basically like 200 tasks that we have seen in real world environments that people have to perform every single day across drawings, across project documents, across specs. Luigi La Corte: Hmm. Andriy Mulyar: and really codify that into a label data set that we work with our agents to be able to replicate so that when agents are actually deployed into production, ⁓ humans finding them valuable. We have much larger internal data set ⁓ in the that we haven't released. That's not part of our open source release. And the reason this is because if you want to go in and you want to iterate on building a better agent, you need to make sure you're not regressing in the capabilities that you delivered last time. It's kind of like a measuring stick, like every time you go in and release your software. And our biggest differentiator, the reason we're able to ship so fast and reason we're able to maintain such a high quality is that we spend a lot of our resources, like I have structural engineers on staff literally going in and hand doing right now purely for our agents to be able to measure themselves in their takeoff abilities. And that's a really big differentiator too. Luigi La Corte: Hmm. think part of the tension that Andrew's getting to is in Rich Sutton's piece, you could take away that the only thing that matters is scaling compute and what the LLM providers are doing really well is scaling compute. Should we just work with them? But it's not mutually exclusive to Andre's point, which is, yes, compute matters. They benefit from that compute, but the harness is very important. There's been a lot of research that's come out. It's very anecdotal, but... chain for example, Lank Smith has shown that changes to harness can materially impact performance across various benchmarks. Let's say on SWE bench, the software engineering bench. So harnesses are very important and for people who are listening who don't know what a harness is, ⁓ effectively ⁓ chain of prompts and and context that you give the underlying LLM. Let's say an LLM has this context which is all of the spec information extracted or structured, let's say in JSON. It now knows it can call a tool, read spec versus let's just do a regex for various terms. It makes it easier for to hit certain results. so Andre is saying with their harness and with additional tools, he can benefit from the scaling compute and disproportionately provide value to customers compared to OpenAI. Obviously Turner, for example, spent a lot of money with OpenAI. I think people who are construction executives are thinking like, well, should I go domain specific or should I just go with OpenAI? OpenAI is gonna give me the best bang for my buck. They're going to scale compute faster. I actually don't need Nomek or I don't need join or any provision. It really depends. And like, I obviously believe that if you give agents certain capabilities, it's going to be better, materially better than just an out of the box, not say undifferentiated, but unspecialized, doesn't have access to special tools or context as they to Nomek. And then obviously what Andre is saying is, Where these people do the work is very important, which is why you're not just saying, Andre, we just do custom models. You're also saying, no, I want you to work in Bluebeam, because you obviously know that the difference isn't just capabilities, it's also where your customers are doing the work. Andriy Mulyar: Yeah, mean, it's not just that's the difference. That's like 80 % of what it takes to actually get humans to collaborate and accept the outputs of agents. Like 20 % of the difficulty of what we've done at Gnomic is build this system that can like mark up a drawing and it's producing accurate markups relative to like specs or relative to like a ⁓ that somebody has for like our drawing review capabilities. 80 % of it is the change management that we have to do at every organization that we work in to train people up. Andrew Zukoski: We got it. Andriy Mulyar: to have them understand how to use these systems. We're operating and we're software and to an industry that doesn't ship software to end clients. They ship really complicated, highly risky physical assets that require years to build and construct, ⁓ and it's a very industry in a human-centric space. There's absolutely no reason why anyone should be trusting these systems out of the box. ⁓ and trusting the fact that these things will work. And a lot of the time that we spend, and I would say 80 % of the challenge and growth circle that we have as an organization that we're figuring out and how to sort of make work, is that change management that organizations are going through as this new generational wave of software happens. Andrew Zukoski: distribution and onboarding. Yeah, can we go back? Can you pop open the blog post for the benchmark set? There's a graphic there I want to ask you a question about, but maybe while that's happening, I'll just sing the praises of more power to you for putting the effort in and then publicly releasing something quantitative about performance of these systems in the AEC domain. I think that's commendable and sorely needed. If we zoom, can we just zoom into that top graphic? Basically the like, like to the right is good and green is nomic and then it's better than open AI and it's better than anthropic, right? ⁓ Andriy Mulyar: It's better than a specific, very popular general purpose harness that OpenAI has around their models. ⁓ Andrew Zukoski: OK, so it's better than ⁓ the codex harness, and it's better than Claude. The Claude code harness. ⁓ that's interesting. ⁓ Andriy Mulyar: Yes, cloud code harness, yep. Mike Powers: Because Andrew just has to pause on that to clarify, right? Because for everyone listening, like, Andre could be using the same LLM as the foundation, but his harness on top of that is doing these unique tool callings that is increasing the efficiency. So if you just blanket hit just the LLM on the very, very basic one, might be 50%. And then using Anthropic's Claude code, it goes from 50 to 57, uses OpenAI. plus codecs goes from 60 to 62. so just like very much simplify that. Like there's still like this one point that's 50 % if you just like one shot at it to a very like LLM. And then there's these agent harnesses that are increasingly increasing that efficiency and tool calling. Andriy Mulyar: Yeah. Yeah, couple of caveats as well for people looking at like, obviously when you go in and you, behind this release we have a paper, we have a code base, we have a big data set of like literally hundreds of drawings with like ⁓ markups of engineers over top of it, ⁓ some middle reviews that happened over top of it. But like what you're looking at is a very abstracted out sort of like set of bar charts that partially, you know, our marketing team had to go in and have some opinions on. So this is why things are very simplified here. But I want to peel back a few ⁓ observations. ⁓ in what we do here, we actually explicitly say that the main agent that's driving, that LLM that's actually driving ⁓ the capabilities, in this case, in all these benchmarks that we ran, is actually, think it's Claude 4.6 with our set of harness and tools. And the whole point of this benchmark and this release, from the gnomic side, apart from releasing the dataset, is what we were trying to convey is that the most important thing about building an agent that can operate in the built environment is not the underlying LLM. It's the harness. It's the set of tools in the environment that it operates in that gives it that set of capabilities and that extra boost in performance that oftentimes is a step function boost in a human being able to accept and work with its outputs. Because oftentimes what you'll get is you can go in and let's say stick a drawing into ChatGPT. it'll happily go in and give you a table of markups. Guess what? The false positive rate's gonna be so high it would've been faster if you did it yourself. That's the kind of step function change I'm talking about. And these numbers, these scores, these reward scores, might seem like they're very small, but even a couple points boost is oftentimes a difference between a human accepting the output and actually finding it, accelerating them, versus it being slop that they have to go in and rework anyways. ⁓ So that... that clod code, that anthropic clod line that you see. Another objection that I've kind of heard in the past and had to explain before is, Andre, you're evaluating an agent that's supposed to go in and do ⁓ design, engineering, and pre-construction work against an agent that's supposed to be writing code. That makes no sense. So it's called clod code, right? It's called codex. One of the things that has been observed across verticals and across domains is that these agent harnesses are not just harnesses for writing code. They're harnesses that are actually general purpose ⁓ agents that can go in and solve many tasks across many domains. Code is just the medium through which they're able to go in and store memory, store operations, to be able to solve those longer-running tasks. This is why it's a very fair comparison to be able to take the latest version of Claude code, that's a big harness, obviously powered by Claude as the LLM, the latest version of Codex, big complicated harness, obviously powered by GPT, and compare them against our custom harness, which ⁓ is obviously specialized for ⁓ design engineering construction tasks. Andrew Zukoski: Okay, so, great, so this is really benchmarking the harness, if I'm understanding that right. Andriy Mulyar: Exactly. Anytime you're a benchmarking agent, you're only benchmarking the harness usually, which includes the LLM. Andrew Zukoski: OK, now, so you've been doing this for a while. As new models come out from Anthropic, does the clod bar move to the right? Andriy Mulyar: The clod bar is going to move to the right ⁓ every single time there's a new LLM release. Every time somebody ships ⁓ a new change to the clod code harness, ⁓ there's going to be a change. This is obviously like a benchmark at a point in time. That's the problem with benchmarks is that to say something and be able to analyze it, you have to snapshot a point in time. But the whole point here and the whole point of any company that's building an agent in this space is that as those foundation models ship improvements, ⁓ Ideally they're building their harness in such way where they can instantly capitalize on those improvements themselves too. Luigi La Corte: Hmm. Andrew Zukoski: OK, and so the statement is like, the new model comes out, Claude moves from 57 to 75. At the same time, Nomek would move from 70 to 90 or something. Like, Nomek will always, because it has a domain-specific harness, it will always exceed these horizontal harnesses. Andriy Mulyar: Yep, and that's the claim that we really tried to, we read these things called ablations, means we take away things from the harness and see how that impacts performance. It's very well explained in the actual paper that accompanies this blog post. And you can literally see as you take away tools, quality starts degrading ⁓ in the actual end agent until oftentimes you are much worse ⁓ than the harnesses that power. CodD code or codex. But the same thing with the CodD code and codex harnesses. One of the biggest things we identified, and I'm to scroll down here for a second, is that one of the reasons they, in many subtasks, so like we broke down the benchmark to cover tasks, project coordination tasks that you can do in a single sheet. Project coordination tasks that require context across a whole drawing. So you might have to navigate across sheets and across disciplines, coordinate across disciplines for example. And then project coordination tasks that require navigating an entire directory of, let's say, project data. So ⁓ a file of specs, a project manual, a drawing, and maybe various revisions of that drawing. And then maybe you also have, let's say, like a textual RFI come in, or like a zip file containing a product submittal that comes in, and having the agent solve tasks at that level of granularity too. ⁓ And this is all to say is that As you peel away tools and capabilities, you can really see a of a ⁓ monotonic shift in the model's ability to perform sets of tasks that exist in each one of these kind of task categories. Andrew Zukoski: It's funny, the thing that's jumping to mind for me is weather prediction. ⁓ Or it's another domain where we figured out how to scale compute. It was by simulating weather patterns at greater and greater granularities. ⁓ And there are these awesome curves. I tried to pull it up, but I couldn't find it, of just the improvement in accuracy over time, or reduction in error over time of the compute models. And the way that they were used was in a given year, the human plus computer was at a certain point. And a few years later, the human interpretation wasn't necessary to achieve that level of accuracy anymore. ⁓ like the just computer on its own had gotten better and exceeded the previous sort of like domain specific, like application of it. ⁓ but the humans still provide an edge on top of that, that like, you know, domain specific interpretation and understanding and massaging of, of the like computed outcome, like continues to provide an edge. And so you fast forward, yeah, the computer is better than the entire like domain specific plus generalized tool was, but Like, if you layer that specificity on, like you're writing a better curve over time. Okay. Thanks. I think I get it. Obviously not. Yeah. Mike Powers: As we take a step out of this, so I know one of the, I know I always talk about Harvey and LaGora, similar to the customers you saw and into the AEs space, like they bill by the hour. So from their perspective and like your pitch to them from a business model perspective, how is that resonating with them? it like, cause like, hey, I bill a thousand hours for this project. and now you're telling me it can theoretically do in 100 hours? Does that change their person? What does that look like for you? I know one thing that I always try to tie to is what's the actual outcome? Yeah, I would love you to unpack that, because that's what people want to hear here is ⁓ what's hey, it's great that this works, but now what? Andriy Mulyar: Yeah. Thanks for asking that question. Yeah, often. Exactly, right? What is it like? Assume this is not like marketing, ⁓ it's vaporware, these things actually work. What does this mean for the reality of like how humans operate and how these industries operate? And I think the analogy you make with like legal organizations is very apt. A &E, ⁓ architecture and engineering, ⁓ these These are oftentimes consultancies, professional services businesses, whose traditional business models revolve around making a margin across billable hours that then they resell to the teams that they're teaming on on physical ⁓ design, and and also performing like consulting on repairs and inspections to municipalities. And of the question Mike getting to here is, what does it mean when you a piece of software that makes one of those humans more efficient at their job, or even, know, for so many tasks, you know, turns a task, turns a job that would have taken eight hours into a job that would have taken five minutes with 10 minutes of human verification and checking time. What does that mean for the entire business model here? Does the business model collapse? This is actually, and I get this question a lot, and I always remind people that like, software existed before 2022 that made people more efficient. People had to... just entered this question way before AI existed as well. It's obviously more amplified now. What does it mean to create in a consultancy so you deliver your work output with less cost of goods but still retain that same margin? It means that people have to shift their business models to milestone-based pricing, where at the 30 % mark, you get a milestone, a lump sum payment that is not tied to keeping a record sheet of hours. The problem with this is that the timeline to which software develops and moves is much faster than the timeline, ⁓ 10-year timeline to sign a contract around designing a asset and delivering a physical asset. And especially, for instance, with the US government, a lot of the ways they only do contracts right now, ⁓ because the that they're working with are so large and have such large time scales, you're contractually obligated to go the hourly for how you structure those contracts. ⁓ It's a very big reckoning that's happening in this space. Every company is dealing with it. I have this conversation literally every single day. that's why I'm glad you asked about it. But end outcome is you need be billing for the outcome, not billing for the hours that it takes you to get to the outcome. Because it makes you unaligned with what's happening on the ground in terms of... the output when your clients will deeply understand that your cost basis for that outcome is changing. will just trust you less. And ⁓ people ⁓ one... ⁓ Mike Powers: Is that is anyone doing it though is like anyone like are there any owners receptive to an A &E firm saying, hey, we're going to charge $10,000 to do X and how what does that go up to? Because there's obviously iterations and there's changes to drawings and the owner wakes up one morning and says, I want a pink wall instead of a blue wall. Yeah, curious to unpack that. Andriy Mulyar: Yeah, yeah, 100 % is happening. It's been happening forever in, let's say, like the inspection world, right? like A &E, they don't just work on brand new. new artifacts brand new constructions. Inspections have always been like per out-time based delivery. in these subsets of business is where you're often times seeing technology and innovation grow most rapidly because there's the most incentive to basically automate work because it's pure that you're making. And a lot of new assets that being delivered. The firms are going in and from the ground structuring structuring the contracts because they're able to explain to their clients that this is actually the incentive structure that most aligns them. At the end of the day, the ⁓ end can oftentimes be the same or maybe a bit less for the end client. The margin or the amount of profit being made by the firm can be kept relatively constant, but the important part is that... that the business model aligns with the actual realities of how that work is being executed and done. ⁓ Mike Powers: So yeah, just a pushback on the inspection. In the inspection comment, there's a one zero outcome, right? Like, hey, it's passed the inspection or not, and then here's the remediation. One thing I've heard from specifically, I work a lot of mechanical design engineers. And with mechanical design engineers, it's like... You can, they've tried to charge like, want to charge 30 cents a square foot for my design services. but it always goes back to, there's, there's mechanical engineer on site. There's structural engineer on site. There's a acoustics engineer on site. There's this architect that's overarching all of this. And it's very, challenging for them to have a cost based sum price whenever else around them has a billable hour. And this kind of relates downstream even to like RFIs and submittal reviews where like. an owner ⁓ only paid for one submittal review for one product line. And if there's more, they're actually going to charge more and there's always like this line incentive. So I'm curious, like in that ideal state for you all, is it like, could you sell Nomic to an owner and be like, Hey, you, all these firms have to use Nomic because of X, Y, and Z, and then also change the business model for those AE firms. Andrew Zukoski: Okay. Mike Powers: I know I was trying to like, that's one thing where it's like, I'm just, we're trying to reconcile as like, Hey, like this is great. If I can automatically do it, automatic takeoff on a PDF, which the whole underlying issue of that is like someone's already done a takeoff in a model, in a BIM model or whatever. And then they have to like abstract it and do it out. But like, like, what is that pitch and what is that conversation like, and how are you seeing value drive to that? Andriy Mulyar: Yeah, so we're already with owners and operators. We work across the A, the E, the C, and the O. Our focus and heavy market penetration right now is on the A-E side ⁓ because where we that's where a lot of the project coordination issues stem up. And that's where a lot of the value sits. But GCs with vertically integrated design teams or operators that are to deal with the mess of owning the asset afterwards ⁓ get a lot of from this tech, too. I don't have the answer to the question of how do you, every single design discipline, in every single case, make transition happen. It's an extremely hard problem. This is one of the oldest industries in the entire of humanity that has always charged, in some sense, per hour. right? it is now only in the past 40, 50 years that that cost basis has been significantly challenged, frankly, by technology in a major way, or maybe in the last 120 years, right? And then I think the entire sector has been coming to grasp in various ways. In some places it's easier, and in some places, like you were just mentioning, right? Especially when you have a lot of people on the ground in person, it's a lot harder. But there are definitely of businesses. Like the very concrete answer can give you is it's very mixed bag where there are firms that will continue having to bill by hour for substantive work probably forever that's just how it works and their can be gone in and moved to milestone based outcomes lump sum based payments. Luigi La Corte: Andre, do you have a hard stop at the end of the hour? Andriy Mulyar: I can go for like an extra an extra five minutes, but I do have a ten o'clock call Luigi La Corte: because I know we still want to show the output from at all. Okay, great. ⁓ Mike Powers: do it. It is live finally. It only took 40 minutes to do the open AI one. Yeah, I'll share it real quick. Andrew Zukoski: Anyway... Luigi La Corte: Okay, you have it ready to go, Mike? Okay, great. Mike Powers: So yeah, this again, this the whole rep and I will share this after as part of the link. ⁓ And so I'm looking to dive into it. But overall, like it was the same project. No, yeah, I'm just kidding. Just be a smart ass. ⁓ Same project, same document, same prompts across the three different models. Again, these are the going back to Andre's point, like these are the ⁓ foundation models, the LLMs plus the harness toolings that they have in their chat interface. So you obviously can see there's a Luigi La Corte: Can you zoom in? Mike Powers: you know, a 50 % spread on pricing. ⁓ Perplexity was the quickest. Perplexity is in three minutes. It missed a lot of things. ⁓ It also missed what we can go into. ⁓ OpenAI came in the highest, Cloud came in the lowest. ⁓ That's all screwed up. And if you actually normalize pricing, it actually gives a recommendation at the end, which is this Flatbush company, which is, Flatbush was... think Flatbush was open AI. Luigi La Corte: Where's the ground truth, Mike? Do you have this against the ground truth? Mike Powers: I do not right now. This was a random project that I found. ⁓ But the cost is in range. The $350 is a good cost per square foot for this type of project. ⁓ There was just big misses. One ⁓ of the agents, one of the subcontractors included the owner furnished water source heat pumps, which was like a $60,000 Delta. ⁓ So it's interesting. Again, it does pull up. Luigi La Corte: I see. Mike Powers: gaps in the analysis. We'll share this with the link after so we can review it in more detail. But I think the biggest blaring kind of open piece is one perplexity was in the middle, but there was $200,000 spread between the lowest and highest bid on the same set of documents. And I don't know any general contractor that's buying. projects on a 50 % spread. This is a very small project, so you can imagine as this grows, those ranges get even larger. And just to reiterate the point that leveraging foundational models and their foundation harnesses is not the best practice. And you really have to control the conversation, control the harness that you're building to get as close as possible to what you're looking for. But it's interesting data point for sure. Andriy Mulyar: Yeah, this is really cool. Luigi La Corte: Andre, when you show people the output from Nomek, can you give us an anecdote of what the impact looks like? Maybe not in the abstract, but as concrete as possible. Andriy Mulyar: Yeah, so some customer quotes, I think, that get people most excited when they hear them reiterated. What are the tasks that people just really hate doing? ⁓ tasks that don't give people dopamine throughout their day. They're oftentimes the same tasks that happen across when you're doing lots of coordination work or when rework is having to be happened in those communications. So, for on submittal reviews, we've taken four to five submittal packages that would have taken an engineer mountains of time that they would have to be, you know... Deallocating from another project being able to do them in five minutes ten minutes within another ten minutes of human verification time That is insane ⁓ not only savings, but frankly a mental load savings You didn't go school for ten years to become an engineer to spend all your reviewing submittals. That's an example Another big is on just getting to the work you can't get to On the fixed amount of hours you have allocated to that project or the fixed amount of time you have allocated dedicated to the project. One of the really big advantages of AI is the fact that these systems can be a lot more exhaustive than humans when operating over top of 5,000 page project manuals interspersed with a 300 page drawing set, for example. And that's another place where I think a lot of the aha moments in value comes from is that when these systems catch things that you as a human just frankly don't have the bandwidth in the amount of times you've dedicated to the project to be able to execute on. It's cool when the agent goes in and does a markup like, It's kind of a sexy thing when you see the agent marking up a PDF or something like that. That's not typical output that you see from an AI system. that's where the real ROI comes from. You're able to deliver a higher quality output when you have an agent working alongside a designer or engineer or a pre-construction member than you would a member working by themselves, oftentimes in a faster amount time, which has implications to the business model and the underlying business. Luigi La Corte: What are you most optimistic about in terms of the problems that you hope to solve with Nomek? Andriy Mulyar: I think one the things I'm really excited for is the shift that's going to be happening in the next couple of years across every single industry, where as these systems improve in quality and improve in consistency and the general public starts trusting them. Obviously, there's a lot of trust being built in daily workflows. Everyone's using some agent to go in and run research tasks, figure out what restaurant they want to go to. As it starts getting more embedded in professional lives, I think you're to start seeing the business models of the companies that are operating. shift to being more outcome based billing as opposed to you're buying a subscription of a bunch of seats into a product. You're going to expect that a company can go in and you're going to be able to pay $200 for a drawing review and get that drawing review out as opposed to paying some amount of licensing fee to go in and have access to a product that can deliver drawing reviews for $200 of AI spend because of the pressures that are just going to happen in the market. We're already seeing this in the legal space, kind of the professional service industry that I think is most similar. to how AE firms operate from the business model perspective. You have companies coming in like Crosby, which are competitors to Harvey and Legora, which are, they're not selling you a platform, they will literally sell you a contract review for $100, and you pay $100 for that contract review. This is just the reality, I think, of where all software is headed. People are going to expect that software to be built by outcome. So I'm really excited personally to sort of figure out that transition as we have to meet the reality of that. Luigi La Corte: Mm. Andriy Mulyar: of that changing landscape too. Luigi La Corte: Yeah, so true. I think it was tried, poorly. I see some iterations of this. I there was an A16Z funded company. I think they were called Palazzo that tried to do AI enabled services, but I think realized how challenging the base documents were. But doesn't mean that it's not going to happen. It's just that was also three years ago. OK, I think we should wrap here. Andre, we're very grateful for you joining. Thank you so much for opening up and telling us about Nomek. ⁓ I'm sure our are very excited to double click and see what you're up to. Really appreciate you spending the time with us and and Mike and Andrew fun as always. So thanks everyone. Make sure to rate us on Spotify subscribe to our newsletter. Thanks everyone ⁓ Mike Powers: Thank you. Thank you. Andriy Mulyar: Thank you everyone.