Latest / Real Estate Meets AI / A Conversation with Arpit Gupta (NYU Stern) and James Scott (MIT)
Transcript
- Sandeep Ahuja: what I love about today's conversation. I've got an economist who used AI to read every zoning code in America. And the guy who tracks 8,000 prop tech companies and can tell you why the Empire State Building was built faster in ⁓ than anything we build today. Arpit Gupta from NYU Stern and James Robert Scott from MIT are coming at the AI and real estate question from a completely different angle and completely different angles to each other. James Scott: you Sandeep Ahuja: And that's exactly why this is going to be so good. So let's go. up, we've got Arpit Gupta, Associate Professor of Finance at NYU Stern. You probably know him from his research on office real estate apocalypse. The paper puts a $556 billion price tag ⁓ what remote did to commercial real estate. But what you might not know, that he's now teaching AI and finance. and building his own alum tools to analyze zoning codes. Arpit, welcome to the show. Arpit Gupta: Thanks so much for having me. Sandeep Ahuja: Amazing. And next up, we've got James Robert Scott, who's the director of Real Estate Transformation Lab at MIT. James was literally the first employee MIT's Real Estate Innovation Lab when it was launched. ⁓ he's built what's probably the most comprehensive map of prop tech in the world, ⁓ over 8,000 companies. He just co-authored a major report on AI in industrial real estate. James, it's so wonderful to have you today. James Scott: Sunday, thanks so much for having me. I'm really looking forward to this. And I like the way you opened up the conversation. was almost like you were pitting myself and our pit against each other. And I was like, no, no, no, no, no, we're going to get on really well. This is not going to be one side. But yeah, exactly. Arpit Gupta: Hahaha. Sandeep Ahuja: We're friends! We're all friends here. Well, Arpit, I have to start with your Twitter handle. Arpitrage. That's Arpit plus Arbitrage, I'm assuming, right? Arpit Gupta: Right, yes. Sometimes people say it was an armpit rage or something like that, but punning off of the financial arbitrage thing. Sandeep Ahuja: That's what I thought. So tell me about your journey from math undergrad University of Chicago to publishing in the American Economic Review about the office apocalypse to now teaching AI to MBA students. I would love to hear about that journey. Arpit Gupta: Absolutely. So I think really the true line is trying to use data to understand the world. When I was graduating college back in 2009, we were in the midst of the financial crisis. And I can already see back then that real estate was a really big part of the economy and it wasn't working very well. And we were just beginning to get all these great new data sets to understand the financing of housing, the way housing is located all across the country. And just really interesting data sets that allowed you to explore more what's going on in connection to its importance with the whole economic landscape that we all live in. So I really pursue it a PhD to try to disentangle, understand those drivers a little bit more. got a job as a professor ⁓ at NYU Stern where I've been for 19 years. And the time I've been largely exploring real estate, household finance, and urban economics and how these things interplay, influence the world. The way I connect to AI. ⁓ is I realized that so many of the real estate topics I was interested in had a large textual component to them, specifically regulations, that I could use these LLM tools to help me understand. And so I started to develop pipelines and ways to source all that information and understand it in ways that would deliver new insights about the ways in which housing regulation ultimately impacts our built form and the way we live. Sandeep Ahuja: the common thread really across everything is your fascination and love for data and maybe ⁓ actually specifically for you, large ⁓ data and you know, of the dance that they're doing see what, okay, what are they actually us? Because ⁓ more data doesn't necessarily mean we know more. More data sometimes can just more data. we still know about the same or less. that's super interesting. Arpit Gupta: Yeah, absolutely. Just one example to your point about how more data isn't always helpful. One of the big struggles I've had, I would say, in my career is trying to get historical data because we can often get more cross-sectional information about, ⁓ here's another market or here's another part of the country. We can kind of get some more information there, but we're often limited in our ability to get information from the past. And so that's another dimension in which I started to use alum tools more recently is to explore the depth of information that we can get historically. which kind of expands our data in a different way. It's not necessarily big data, but it is really informative data because it's covering something we don't understand very well. Sandeep Ahuja: makes sense. right, well, James, you came from a different direction, I think, right? You were in commercial real estate, ⁓ to MIT, and then became the number one employee of what became the real estate innovation lab. ⁓ James Scott: it's, no, no, it's much more convoluted than the way you put it. The ⁓ you said it, there was a lovely trajectory there that just all made sense. It was exactly the exact opposite in ⁓ of ways. was actually, I was a lawyer for many years, almost nine years ⁓ I moved to the US back almost 11 years ago. And when I came here, I wanted to move into something the lines of real estate private equity at the time. And ⁓ then found myself at MIT. And through... Sandeep Ahuja: ⁓ dear. James Scott: My time there, is back 11 years ago, ⁓ came across some incredibly brilliant, inspiring who were showcasing this whole new world of technology within an industry was possibly not ready accept it at that stage and ⁓ has found itself many challenges with adoption innovation. But they kind of opened my up to a whole new concept and idea. And through their... ⁓ insight, I joined up and started working for them on different projects ⁓ as student and ⁓ then after my trajectory quite dramatically over the years in that ⁓ I started getting in a lot of hardcore research within ⁓ that ⁓ specific to point that ⁓ MIT came to and asked me to build out a class. I now have three classes that I'm part of MIT. I ⁓ run the real estate lab. I'm one of the. directors of the actual program within MIT at the Center for Real Estate and our master's program. And yeah, it's one of these situations that if you'd asked 10 years ago, was this where I was going to go, ⁓ not sure would been able to even imagine it. But a lot of things happened in that time ⁓ that helped facilitate and technology. was obviously advancements within the tech itself. COVID had a huge factor within that. and then trying to help facilitate and drive that innovation through an industry which finds it sometimes very hard to accept and to deal with. It's one of the more interesting areas. It's often the business school environment where you can showcase certain plans outlines of how to integrate new technology within an industry. work very, very well and they have worked well for many, many years, but real estate is such a nuance and such a niche area. I realize I said that in a European way, but ⁓ unique. oftentimes, mean, find it with a lot of tech and a lot of finance individuals, I'm not that they look down on real estate, but they think it's much easier business. But what they don't realize is actually far more convoluted and far more difficult ⁓ in order drive change and navigate change within it because of the essential things like Arpit Gupta: Awesome. Sandeep Ahuja: Mm. James Scott: has something like stakeholder complexity, which doesn't make it easier. Sandeep Ahuja: And the nuance of the uncertainty, Like everything is physical, everything physical needs to be built. was on a call recently where you we were talking about how the construction costs went up 34 % in the last quarter. that's just one of the many factors that we're playing at. So I couldn't agree, I couldn't agree more. what I was at was what was it like building a research lab from scratch? But what I basically pulled away was that it was incredible. It was different, not what you expected. ⁓ James Scott: It was a, no, it's a, it was, you know, I don't get me wrong. I was to take some incredibly hard work that had been done with people that I'd worked with specifically, yeah, Andreas Shigoo back in the day with the real estate innovation lab. And I was able to build on that a little bit with certain relationships and go, but I went to the, in a sort of a much more of a, a sort of an industry perspective as opposed to a research perspective in lots of ways in order to try and help drive change and innovation within real estate. When I say real estate, a vast amount of different tasks, areas, from the and construction to leasing to the ⁓ investment, property management. a wide, wide, range of individuals within that. to then obviously sit on top my favorite group of individuals who are real estate developers out there. Because trying to, if you can... If you can change their mind, you can change anybody's mind. Sandeep Ahuja: That's probably true. All right, so let's do this thing that we do on the show, the AI meter on a scale of one to five, where one is AI verse, we're actively and five is AI forward that we're leading charge. Where is the real estate industry right now? And James, you literally track thousands of prop tech companies. So you first, what's number and why? James Scott: That's really hard. That's like saying the music industry and we want to find it by different genres. I'll have to just pick different areas because if you look at the financial markets and investment and owner operators at a high level, They're at the five level across the board and have been for a long time. They've had their toes in this and they're navigating it with spending significant amounts of money to try to build it. And then probably on the flip side, you have something like facility stroke property manager, which have found it tougher and harder to, to, to really discover the true return on investment for a lot of the products that have been out there in the marketplace. But that's again, rapidly changing. Like, mean, that's another part of this that, it's hard, you mentioned sort of work that I, I've done in previous years, but everything in the last two years. kind of has changed so much that ⁓ and advancement of capability ⁓ something that we've never really seen before. And understand that has been a really, really tough process. it's still, it's not a case of it's on pause right now and you can kind of catch up with it. You're at a place where something new coming out again that can mean ⁓ from an capability or from. a quantum capability or something else that's coming on the stream. So trying to kind of keep up with all of this is as tough a job I think that anyone could possibly have. have moved to the ground around the airport and the instructions you have reached Central Parking. ⁓ Thank you. ⁓ Sandeep Ahuja: Yeah, yeah, that makes sense. question for you, Arpit. what's interesting about what James said is that ⁓ it's really in the last two years where kind of reinvented a lot of it, like the knowledge is changing, it's all transforming. So same question to you, Arpit. Where are at and why? Arpit Gupta: I'll maybe give a simpler answer. Maybe I'll just say two. Just because I'm comparing real estate to other sectors and verticals that I'm tracking and looking at. You again, software, for example. That's a clear five. There's ⁓ tons software code that's now completely AI written. The whole industry has really taken to it. And if you're now seeing some companies have to pull back a little bit like Amazon and saying like, okay, we're not going to vibe code everything. We'll have some senior engineers review the code before. Sandeep Ahuja: I'm Arpit Gupta: we deploy to AWS so doesn't crash all the time. But that's like the level that they're at that they have to like tone it down a little bit relative to what they were able to deploy it out. And lot of errors in finance do seem to be adopting it very rapidly. We're seeing that across both public markets as well as private markets, where everything from deal sourcing to extracting information from documents to thinking about investment theses, to thinking about connections and associations of narratives between numbers and text. Real estate, I think, lags behind a little bit for a few reasons. Of course, the industry is a little bit more conservative than some of its other counterparts, but it also confronts a variety of physical real-world challenges, whether that relates to ⁓ construction properties, maintaining and operating them, or about the challenges ⁓ in marketplaces that don't necessarily have the same level of price discovery. So ⁓ all generates very unstructured data and workflows that are often associated with multimodal data. So you might have images and PDFs and Excel documents, the document base is very scattered. And the challenge is to kind of take that very messy workflow and find ways of integrating it, which is just a little bit easier in other sectors than in real estate. Sandeep Ahuja: That's fascinating. So basically, if I pull back, you're both not saying things that are too dissimilar. mean, sure, there's parts of, James, what you said, the industry that are definitely on the cutting edge on the on the finance equity, you trying to figure out the deal structure, perhaps standpoint, that's much further ahead. But on the whole, like when you come down to the nuts and bolts of actually build buildings, like building buildings or, you know, kind of putting occupants in it. James Scott: Yeah. Sandeep Ahuja: We're kind of at that level, like somewhere between two and three is kind of what I heard. James Scott: Yeah, I'll ⁓ add to it a little bit. you think about it, it all makes sense in certain respects in that, you know, if you're dealing with something like high level finance where the data itself is far more freely available and you have the ability to utilize it in a different way, plus also something that you rarely see in the real estate world across no matter what sector it is, ⁓ sort of level of ⁓ R &D, whether that's budget or looking to try and do new things. kind of see where that can go. But then in other areas of real estate, you don't necessarily have that data capability to be able to extract it. It's too nuanced. It's too siloed, like from different directions. Or it doesn't necessarily coincide with the concept of what big data really is with something else, which is coincides with the budgetary restraints that something might have. Like the construction industry, I wouldn't be going completely off piste here to say that they work off margins of anywhere between. two and a half to sort of four and a half, five percent. It doesn't leave an awful lot of wiggle room for R &D to get things wrong. there's not a huge amount of incentive within their organization to drive change and take on. ⁓ However, light the you just made there with the price points, which were already at a place where a lot of real estate development didn't really work from a construction perspective already to where they've already gone even in the last, what was it? was again, stat even since January. 12.5 % increase in construction costs. you utilize some sort of new technology or capability in order to drive efficiency and reduce costs or reduce some sort of time component within this, we're going to see an awful lot of stuff being built in the near term. Sandeep Ahuja: Couldn't agree more. Yeah, please. Arpit Gupta: I just want to add one caveat to that, which is I think the data center rollout. So that's not AI in the sense of people in the industry using AI as a tool to achieve whatever objective there is, but it is a picks and shovel story of how real estate is meeting and accommodating the AI buildup in the rest of the economy. the scope and scale of that data center buildout is massive. know, the latest numbers I saw suggested that we're... James Scott: ⁓ Arpit Gupta: doing more construction volume now in data centers than we are in office buildings. So it's just my former interest. So that is one area in which that real estate sector has moved very rapidly is to accommodate the physical expansion that is going to be necessary and the rest of the AI economy. Sandeep Ahuja: It's interesting you talk about data centers. was listening to podcast from, I want to say 20 BC where basically ⁓ were breaking down sheer amount of bets that have been taken with the limited amount of proof that we have thus far that those are going to convert into revenue. And the point that either way this gentleman was making in the podcast, which I roughly agree with, is that it's not like it's not going to pay off and that AI is a fad or short term thing. It's certainly ⁓ but the time within which ⁓ it off or which it works ⁓ is a real question then when you pair that with all the... energy challenges that are happening, where folks are really starting to back against either way hyperscalers and this that and the other. ⁓ all these more micro data centers sprouting up that I've heard and more of. ⁓ It's such an evolving world and that all still doesn't take into account. Well, ⁓ as the gets better and models get better, I would hope our need for energy improves as well because it would really suck if it doesn't. Not disagree, just a side there, that's a whole other world. So our pit, obviously you've built, in your most recent research and we, talked a little bit about it ⁓ and it me really excited because I have an architecture background. So walk me through ⁓ zoning AI research tools that you built together. What was the you were trying to solve? Arpit Gupta: Absolutely. So this is joint work, I should say, with Alex Bartek and Dan Milo, who are climbers of mine in this project, where our big picture objective is to try to create a data set that puts the zoning regulations and housing regulations into structured information across all cities in the US. That's sort of our North Star where we're trying to get to. And we're not there yet, but we're making a little bit of progress along the way. we're starting out is we began with a target of answering about 20 or so questions about the zoning code that previous researchers had thought was interesting to examine in localized areas. So they had done this analysis where James is in Boston, looking around the local Boston area ⁓ answering a series of granular questions about like, what's the minimum lot size? What is the requirement for a public hearing when you have new construction? Is multifamily even allowed? Are townhomes allowed? So trying to understand various aspects of use regulations and some bulk density requirements and lot sizes, setbacks, things like that, as as process regulations that pertain to how and when housing construction can happen. How is it allowed? So we began with that initial set of questions, but she kind of back tested based on when humans had already responded to in areas. And we up the process to answer those questions across really broad sample. of all American municipalities. So this provides us for the first time information on what the actual codes say in a harmonized data set across the U S and to do so we use something that is now I would say one of the most dominant AI applications in business that I've seen. So if you're just new to AI, you're thinking, okay, how can I use AI in a way that's a little more deep than just tossing some documents into chat, GPT while we're using something called retrieval augmented generation or rag. So this is a architecture. that allows you to pull specific documents and figures from a corpus of text. The way it's gonna be relevant for us is, you know, we have a document, we want to answer a question about, you know, what is the minimum log size requirement that can be done in a particular area or district or something like that. And you can, if you want to, just toss that document to the JAT GPT and it'll give you a response and, We kind of found that that simple thing was like, let's say like 60 % accurate, right? So I think for most people in the industry, they're probably like, well, that's not really good enough for me. I kind of want to a ⁓ much higher level of trust in the data. And where that's coming from is like when you throw in these like really long contexts into AI ⁓ and you have a sort of search through the whole span and you don't really give enough guidance on how to exactly answer the question you have. It kind of flounders a little bit. So we built a lot of tools in. again, through this RAG architecture, which basically allows us to find the needle in the haystack a little bit easier. So we're kind of like zoning in on a part of the document that's going to be most likely to respond to the answer. We get a answer in a much higher likelihood. And then we have a little bit of an audit trail to help us think back and examine whether or not that that was a true citation for the fact that we want to establish. So all of that kind of gets us to accuracy rates, say more like 90%. of higher on some questions and lower on others. for us as researchers, that's sort of a good stopping point. can say like, all right, we have a ⁓ enough data set. We build that out across the country so we can have, and this data is all publicly available. Also, you can find it on my website ⁓ that a lot of people to kind of search and look at this and use it for research or other purposes. And it just ⁓ one example, think of the potential that we can ultimately get to by taking all this unstructured data and beginning the process of making it more structured so that we can better understand. the regulations and rules that we live with. Sandeep Ahuja: Yeah, that makes a lot of sense. if I was to pull out the two things that I think, knowing a little bit about this world as an architect, because we need zoning data all the time, right? Like we're working with developers. They'll be like, here, I'm trying to build something on my site. ⁓ first thing you have to do is zoning research. Like, OK, what can you even build? You need a variance. What's possible? What's not possible? This, that, and the other. So instead of obviously a old school going to your Municode website and doing it manually, that's maybe two years ago. So no one does that hopefully now. The step two, you're saying some people will throw that particular PDF into chat, GPT, Claude, whatever their choice of chat AI tool. And then it'll pull out information, but it's neither traceable nor very reliable. So as an architect, I can't put my reputation on the line and be like, Well, that's what you can build because I'm 60 % sure that's not good enough. So what you have built is significantly more accurate and traceable, which is again, for different types of research, different types of professions, different types of work, a very critical element. And that's basically what you're pulling out. that a fair, simple representation in very layman words of what you just described? Arpit Gupta: Thank Absolutely, and if any of your listeners are interested in moving along that chain, very good place to start is Notebook.com, which is a product by Google that provides this Rack functionality in a more usable interface. So what you can do with that tool is upload your favorite zoning documents or housing regulations or building codes or whatever is the source of document that you're interested in, and then just query that database just as you would chat to you or anything else. And the difference is that the Google notebook LM product is going to just provide a little bit more verification of the specific source of the document because it's basically doing rag on the backend in order to give you a little bit more of an accurate query. Sandeep Ahuja: That makes sense. Well, that's amazing. James, you've been working on some pretty wild technology, or at least what I would call wild technology over at MIT. Obviously, I read about the heat leaks in buildings, the tech tracker, I'm sure it's continued to even evolve based on what was even written based on this research. ⁓ would love to get from you one concrete example of where AI changed what was possible. The coolest, most exciting thing that comes to mind. James Scott: So again, I come at this in a very different way these days because I probably come at it more from an industry component, more ⁓ a research component. And that's just been an evolution of a lot of the things that I work on. There's some incredibly brilliant things that are out there. call it products that are out there. But they only really work from a research I've seen this. variations of this in different localities and it's so insightful and you can see like I'm like a when I start using it I'm like Bart Simpson like when he gets the keys to the city I start kind of dribbling a little bit like because I get so excited but then you kind of there's another element of this which is industry and they oftentimes because development is so localized and most developers if you were to think of them in your head as an individual they're generally you know, they've been through two or three cycles, they're a little bit older, a little bit wiser in, you know, they've seen ups and downs, but they don't generally, you unless you're talking about ⁓ large scale, you know, national one, but even within the national ones, they have localized teams and they have localized teams because there are certain nuances, regardless of the information that ⁓ required to understand and relationships that are required with cities and government officials as much as anything else as well. And they know they're, as I call it, they're shyer inside out. And The next evolution for me, looking with AI, and we're starting to see this now, is whether it works or whether doesn't, but it's a really interesting concept. Something like artifacts in AI, which takes this element that Arboret has just taken. And then when you have a city like Boston, where there is a blanket law of an FAR of one, regardless of what the entitlement is, but it's so that you have to, when you go for a variance, you have to deal with the city at everything that you actually do. Well, that kind of takes a little bit of having a capability and understanding out of it because you're now dealing with individuals and you're dealing with people, with whoever they are at a city level or at a government level. So what artifacts have done is they've taken every single meeting minute that can be within an environment with all of the different people and all of the decision makers and have been able to extract all of the information and more importantly, a sentiment analysis and understanding of what each individual wants and demands when you go for a particular variance. at any particular site. And so from that can build out and extract a profile ⁓ ⁓ that individual but more importantly, what is demanded ⁓ from personal perspective in order to achieve that variance. ⁓ And I see can be an extremely tool to work for being able to take that and even though they'll have good relationships with people. You can, you people can have bad days, people can have, you be in bad moods, but to have a really good understanding of what specific individuals who you might be dealing with on the other side of the development process, what they want, what they need and what they demand with everything that they do. ⁓ that, that's a really nice way to start neutralizing AI in order to achieve quicker and faster development. And so ⁓ instead of finding yourself butting heads ⁓ the, you know, the side along way. knowing what they need and not that you won't have a good idea in your head, but having it there with true data, like being able to back that up. And I think that's a really nice way to move a lot of development forward. Sandeep Ahuja: That's a really thoughtful of putting it. what saying, basically, is that yes, AI for data, yes, AI for cut and dry black and white things, that's ⁓ slow hanging fruit. Duh, like we're obviously going do that. ⁓ it's humans that are ⁓ super complicated. the that's really moving the needle is utilizing, you know, to better decipher humans actually want because a lot of these projects have ⁓ these, layers and many, many, many and this, that and the other. actually listening, like properly, truly listening is where maybe AI can help. James Scott: Exactly. If I was to frame it in a sort of an illustrative way, it would be like there are poker players that are brilliant online because they have the data available and they can run the numbers in their head. And then there are brilliant poker players in person because they can read people and they can understand what the person's kind of motives are and what they're trying to achieve. And this moves it from being and having that capability online to be able to do it actually in the room. I think that that's where I kind of how I see it, which I think it could be extremely valuable to a lot of people. Sandeep Ahuja: I agree entirely. This actually reminds me of a project. It was a public project that we recently were working on. And was a really relevant building and a lot of people from the community had a lot to say about it. ⁓ ⁓ obviously it would be time consuming and too expensive to have an architect actually read every single one of those. hundreds and thousands of comments and decipher an opinion on what that means from purely an architectural level. to ⁓ AI it, so to say, ⁓ was a significantly better move to make sure actually we heard everyone. And sometimes it can help us hear people better not. So I don't disagree ⁓ at all. helps with data and it helps with people. Love it. So last thing, obviously you both talking to ⁓ real estate professionals every day, developers, investors, operators who are to figure out what to do with this AI moment. some are excited. And I'm sure some are not excited and most are probably somewhere in between. ⁓ ⁓ the one piece of advice you'd give to someone navigating this change right now? We'll kick it off with you, James. James Scott: I'm going to be really, really unfair here and I'm going to both sides of the fence. ⁓ you know, there isn't a business school, there isn't a consultancy for anyone else that isn't putting structures in place for ⁓ large whether they're real estate or not, ⁓ in to implement AI within into their workflow and their systems and their people. ⁓ But have to say, and this is not my view, I'm going to steal this view. you all those structures that they're implementing all make complete sense. If it was as simple as that and you just implemented this six stage process with implementation, then it would be very easy and everybody could just go about that. But culture and location and how you do business, add a significant amount of nuances to already the most nuanced industry that I've ever seen that I know of out there. And then I saw this really, really interesting play on this about two weeks ago by KP Ready, who I'm sure a lot of people get the newsletters from. And he was outlining, when it comes to something like what we do with Excel is, somebody joined organization, there was no real formal Excel training program for all of the individuals. while you might have it for in certain aspects, like a Wall Street prep or something like that, but what he was trying to get at was that the C-suite, they might use Excel in a... completely different way from let's say a pro forma analysis and sensitivity analysis and capabilities. Whereas somebody on the marketing team might use it for constructing lists of individuals and have some sort of CRM system that they might kind of use it distribution point of view. And then you might have somebody ⁓ working at the reception desk who's using it to keep tabs on who's coming in and out ⁓ a building, maintenance or something along those lines. And so he was trying to showcase was that here is an incredible tool that's utilized in significant different ways by different tasks by individuals throughout an organization. ⁓ really how they came about having an understanding was they sat down and started using. ⁓ they did that and they kind of taught themselves how to get the best out of it what they needed to get it done. ⁓ so is that maybe the better way ⁓ going out there and showcasing, you something like a Claude or a chat GPT ⁓ about people within an organization, giving them an opportunity to try and figure this out of how it can be utilized for each one of their... you know, the tasks that they need and have necessary on their day to day and letting them build out a little bit of confidence and reporting back with some sort of understanding of what worked and what didn't work. Now in saying all of that, I'm going to flip it again, you still need a process, you still need accountability, you still need governance, you still need all of that within that. And so somewhere in there, it all makes sense. And when somebody gets it all right, they'll write the book and everybody will follow and it be a bestseller. But that's the reality of everything that that means going through. Sandeep Ahuja: Yeah. Arpit Gupta: I love James's point about kind of combining the flexibility with sort of framework for how to adopt it. I think one thing fits within that, which I think every organization or firm considering adopting AI should do, is establish a of internal benchmarks. Which is to say, the way AI adoption works in any domain is the researchers establish specific benchmark. which relate to how capable AI is going to be at some series of typically technically defined tasks. you know, again, for our project on regulation, we had our own human benchmark, which is how well the AI did at reading the regulation relative to this human counterpart. But in your firm, you can't just rely on those technical benchmarks because you have very specific business needs. So think about what those needs are. Think about what you need to do as a firm organization. What are your benchmarks? some existing human point of reference or you know whether it's a look here's a pro forma and we do this analysis and we get this IRR from this deal and we kind of leave the spreadsheet so how well is the AI going to do at matching the human performance in that benchmark so you have a series of these corporate benchmarks and then just test and see how every new model release does at this series of benchmarks you know maybe you're going to find for your business and use case actually you get better mileage out of Claude maybe for you it's actually a chat CPT for you, even though the model has this technical capability, like when you integrate it with your Excel and your files, it actually doesn't work. You know, that's all something you can discover, but only if you have a rigorous process for determining ex ante, like here's a set of skills that we do that we kind of need to measure and benchmark. And then you have a process for evaluating how different model releases, how different approaches are going to do with that. James Scott: Everything that I could say is 100 % correct. The problems that I've seen in recent times with the real estate industry, and this is not me giving out about it in any way, but you're asking individuals to do something that they've never done before and is very much out of their comfort zone. A lot of executives within large real estate organizations, they just want three bullet points. They want three things. Okay, this is what we do when we implement it. Okay, now we've got an AI strategy. It takes a lot more effort than that. It takes a lot of self-reflection. It takes a little bit of company therapy to understand how your workflows really operate, understanding where the ⁓ are, where the challenges are. ⁓ from that, then having that implementing of a situation, it's not just training or ⁓ kind AI literacy. It's about figuring out we find the greatest level of efficiency that can be created utilizing this technology. ⁓ And that, all what I've just said there requires effort and work. And it's not simple to ask people to do something like that when they have so many challenges on a day to day already within an industry that's, been rough three years, except you went in data centers, ⁓ right? it really has, it's been a rough three years for this industry. ⁓ And got so many fires to put out and they're dealing with so many other issues. And now they're trying to be told to reassess and recalibrate how they even do their job. ⁓ It a big ask, and it's a simple task either. Sandeep Ahuja: Yeah, I mean, to me on many levels, you all are saying basically different versions the same story or like kind of different directions or ⁓ puzzle pieces ⁓ the same puzzle wherein, yes, it's Yes, you have to do it holistically. no one person or one company has it all figured out. So for all that are listening, ⁓ if anything, this be... ⁓ reassuring y'all are definitely two of the smartest people about this and to hear that directly from from you who are seeing it on the very cutting edge who are on the forefront of what's happening out there I hope that's reassuring to all the listeners that you're probably not behind you're probably where most everyone else is if you're in the middle of trying to figure it out because that's what it sounds like where most everyone else is With that, I will say this was such an incredible podcast. Art Pit, James, thank you so much for keeping it real, giving us your opinions. so thoroughly enjoyed it and I know so will our audience. I so appreciate your time. Arpit Gupta: Thanks so much for having me. James Scott: Sandeep, thanks so much for having us. be really great to hang out. Arpit Gupta: You too.