Latest / The Neon Show / How to Solve AI's Biggest Problem | Atin Sanyal, Galileo

Transcript

How to Solve AI's Biggest Problem | Atin Sanyal, Galileo · The Neon Show

• 0:00
  SPEAKER_01: The cost of bad data in AI systems is the next 10 years.
• 0:05
  SPEAKER_01: 99% of the world's data does not live in tables.
• 0:09
  SPEAKER_01: The question is, how do you get data that's ready for models?
• 0:13
  SPEAKER_01: Trust is a big problem in machine learning and it is unsolved.
• 0:16
  SPEAKER_01: Even simple models like XGBoot, the output of a model, which is a decision, how do you know whether it's good or not?
• 0:23
  SPEAKER_01: Kind of became one of the leads and architects of Uber's entire AI fleet.
• 0:27
  SPEAKER_01: Michelangelo, Uber's AI platform, which hosts thousands of models, became a brand in the AI infraspace.
• 0:35
  SPEAKER_01: When software has become tokens, what is the final definition of a great product?
• 0:40
  SPEAKER_01: That's a great question.
• 0:41
  SPEAKER_01: I was figuring that out on my way here, and the day before that, and the week before that.
• 0:45
  SPEAKER_01: That's pretty much all I think about.
• 0:51
  SPEAKER_00: Hi, this is Siddhartha Lualiya, your host at Neon Show and managing partner at Neon Fund, a fund that invests in some of the best enterprise AI companies, start from India and building globally like Atomic Works, Spot Draft CloudSec.
• 1:04
  SPEAKER_00: Today I have with me Atan Sanyar.
• 1:06
  SPEAKER_00: Atan, welcome on the Neon Show.
• 1:08
  SPEAKER_01: Thank you so much for having me.
• 1:09
  SPEAKER_01: I'm super excited to chat.
• 1:11
  SPEAKER_00: Atan, your journey has been so inspiring, right?
• 1:15
  SPEAKER_00: You have been a researcher, right?
• 1:16
  SPEAKER_00: Came from India, came to do your master's here.
• 1:19
  SPEAKER_00: Then you were a part of the Michelangelo team at Uber, right?
• 1:23
  SPEAKER_00: And then you started Galileo.
• 1:24
  SPEAKER_00: So there is very similarities, Michelangelo, Galileo.
• 1:27
  SPEAKER_00: Did the name for Galileo get inspired from your project at Uber?
• 1:32
  SPEAKER_01: That's a great question.
• 1:34
  SPEAKER_01: So the story goes that I was at Uber working on Michelangelo, and I had met my co-founder who Vikram?
• 1:44
  SPEAKER_01: Vikram, who Vikram was at Google, and he was working on like um finance systems like Google Pay, and then did a bunch of AI work.
• 1:56
  SPEAKER_01: And um he sent me this WhatsApp of a diagram he drew on a whiteboard of a dashboard, you know, so we saw charts, etc.
• 2:08
  SPEAKER_01: And at the top, he just wrote Galileo.
• 2:11
  SPEAKER_01: And I asked him, What is Galileo?
• 2:13
  SPEAKER_01: Like, uh, did you just already come up with the name of the company?
• 2:16
  SPEAKER_01: He's like, No, I just threw in some similar medieval character like Michelangelo.
• 2:21
  SPEAKER_01: And uh was like, I love the name, so it kind of became a thing.
• 2:25
  SPEAKER_01: And uh, but turned out Michelangelo, the real person, died the same year that Galileo was born.
• 2:32
  SPEAKER_01: And I found that out two years later.
• 2:34
  SPEAKER_01: So funny coincidence.
• 2:35
  SPEAKER_00: And which was your first job after UCLA?
• 2:38
  SPEAKER_01: So I joined a very early post-acquisition Siri team at Apple, and it the team was about 30-35 people, still the founders were there, and um I ended up, it was a long story to join that team, but I ended up working there for uh uh a good number of years, about five years, where I worked on the early knowledge graphs that Siri was built on.
• 3:01
  SPEAKER_01: And uh it was kind of in the era where of early deep learning models were being experimented.
• 3:06
  SPEAKER_01: It was still very expensive to run it in production, uh, but I kind of saw the whole arc of how do you build knowledge graphs and language learning systems without these models, and then saw models kind of become a thing.
• 3:21
  SPEAKER_00: Got and and then how did uh you know Uber happen for you?
• 3:26
  SPEAKER_01: So Uber was an interesting opportunity that that came to my plate.
• 3:31
  SPEAKER_01: Uh at Apple, I had worked towards the end of my stint at Apple.
• 3:35
  SPEAKER_01: I was working on some streaming machine learning-based uh systems, more from an infra standpoint.
• 3:41
  SPEAKER_01: And uh Uber was starting out this team.
• 3:45
  SPEAKER_01: Uh, it was supposed to be the foundational AI team, and they required uh engineers who had worked on uh scaling streaming systems because streaming ML was very new.
• 3:57
  SPEAKER_01: This is back in like 2016, 17.
• 4:00
  SPEAKER_01: So I decided to talk to them, and they were very interested in getting me to solve that particular problem.
• 4:06
  SPEAKER_01: And then over time, I kind of became one of the uh leads and architects of Uber's entire AI fleet.
• 4:13
  SPEAKER_01: So, Michelangelo, for those who don't know, uh, is Uber's AI platform, which hosts thousands of models across demand pricing, ETA prediction, any AI feature that you're using across any Uber's product today is powered by that platform.
• 4:30
  SPEAKER_01: But back then it was very zero-to-one.
• 4:32
  SPEAKER_01: We barely had any uh any models running.
• 4:35
  SPEAKER_01: And Uber's machine learning uh fabric was just a bunch of data scientists training their own models.
• 4:42
  SPEAKER_01: So it seemed like a really exciting opportunity.
• 4:44
  SPEAKER_01: And I also love that Uber was one of the first systems that truly worked on very mission-critical machine learning because you're putting two people inside an enclosed box, and there's you know safety issues and um uh a lot of new challenges that the world had not seen before the uh the the um uh what's it called, the the marketplace economy had had come.
• 5:09
  SPEAKER_01: So it seemed like a very exciting technical challenge.
• 5:12
  SPEAKER_01: But over time, Michelangelo kind of became a brand in the AI infraspace.
• 5:17
  SPEAKER_01: Even today, I get emails and in mails from folks talking about uh you know how they're still using the Michelangelo Blueprint to build their internal AI system.
• 5:28
  SPEAKER_01: And uh just the name is enough, at least in the community.
• 5:31
  SPEAKER_01: It's I feel it's the most underrated team because it was a team of absolute superstars, some of the best in the world AI experts.
• 5:38
  SPEAKER_01: And it gave me the foundations on how to think in systems and how to really build world-class software.
• 5:45
  SPEAKER_01: So I owe a lot to my experience at Uber, and uh, it was kind of the foundations of not only me personally starting Galileo, but also learning how observability and evaluations is such a big problem in machine learning.
• 6:02
  SPEAKER_01: So even though the workflows have changed today, the the fundamentals still remain the same.
• 6:07
  SPEAKER_01: The problems that people are facing on how do I know whether the output of a machine learning or an AI system is good or bad.
• 6:14
  SPEAKER_01: They were the same challenges we were trying to solve.
• 6:16
  SPEAKER_01: And the but the mission criticality of it is higher than ever because I think it's the first time in human history that AI has become the face of the product.
• 6:26
  SPEAKER_01: A user is directly talking to an AI and working with an AI.
• 6:30
  SPEAKER_01: So every output of an AI system is mission critical.
• 6:34
  SPEAKER_01: And with agents doing actions and making critical decisions, the stakes cannot be higher.
• 6:42
  SPEAKER_00: And and how did uh you and Vikram zero in on the idea of Galileo?
• 6:48
  SPEAKER_01: That was a long story.
• 6:49
  SPEAKER_01: So we uh were discussing on what to build.
• 6:53
  SPEAKER_01: Uh, Vikram was certainly very gung-ho about um starting a company, and uh I was deeply in the AI space.
• 7:00
  SPEAKER_01: So I knew that AI is the future, and I was very excited to build something in that.
• 7:05
  SPEAKER_01: Uh, he also saw the same sort of vision.
• 7:09
  SPEAKER_01: Uh, we initially started off working on uh some data infrastructure for machine learning systems.
• 7:17
  SPEAKER_01: And while we were ideating and building, we realized that trust is a big problem in machine learning and it is unsolved.
• 7:23
  SPEAKER_01: So we started building a trust layer of every machine learning workflow.
• 7:28
  SPEAKER_01: How do you uh quantify whether an output of an ML model is good or bad?
• 7:35
  SPEAKER_01: And one thing led to another, and I created a prototype.
• 7:39
  SPEAKER_01: We met a professor at Stanford, Chris Ray, who we knew through one of Vikram's friends uh who had recently sold a company that Chris Ray was involved in.
• 7:49
  SPEAKER_01: So I showed him this little demo of a Jupyter notebook where you the user publishes a model, and before the model gets productionized, it would do these bunch of checks.
• 7:58
  SPEAKER_01: And Chris Ray saw it and he's like, this is a great idea.
• 8:01
  SPEAKER_01: We'll do it for language models.
• 8:03
  SPEAKER_01: And my only question to Chris was, what the hell is a language model?
• 8:06
  SPEAKER_01: Right?
• 8:07
  SPEAKER_01: Because LLMs were not a you know a word back then.
• 8:11
  SPEAKER_01: And of course, I knew NLP systems and stuff, but uh um this idea of running language models at scale was so alien and so new.
• 8:20
  SPEAKER_00: This is 2021.
• 8:21
  SPEAKER_01: This is early 2021, so late 2020.
• 8:25
  SPEAKER_01: Uh uh, but it seemed like unstructured data AI is the future.
• 8:31
  SPEAKER_01: And I realized that because even in my time at Uber, we were slowly productionizing more and more deep learning models.
• 8:39
  SPEAKER_01: I still remember a moment at Uber where we productionized the first deep learning model, and we saw that it's kind of giving the same results as a an XG Boost classifier, but it's so prohibitively expensive with GPUs.
• 8:52
  SPEAKER_01: And uh, so we realized at least for some classical use cases like demand pricing and those kind of tasks, it's not really needed.
• 9:00
  SPEAKER_01: It's like using a fancy pen when a pencil would work.
• 9:04
  SPEAKER_01: But over time, the cost of running these models in production went lower and lower.
• 9:10
  SPEAKER_01: And that was kind of my main uh aha moment, where uh it was clear to me that the next 10 years, 99% of the world's data does not live in tables, in cells.
• 9:23
  SPEAKER_01: Uh, it takes a lot of machinery to get data into a clean format.
• 9:28
  SPEAKER_01: Uh, and um for me, it was a personal realization because we had worked on the feature store at Uber and we had built the world's first feature store, and we literally a feature store for uh for the audience, especially, is think of it as a database of data that's ready for models.
• 9:49
  SPEAKER_01: The question is, how do you get data that's ready for models?
• 9:52
  SPEAKER_01: Uh, it's it's a long, um, arduous process of doing ETLs on raw data, which is very messy, very raw.
• 10:02
  SPEAKER_01: And you set up these pipelines which converts that raw data into very specific um ML data, which is ready for a model to consume.
• 10:12
  SPEAKER_01: Uh, features are uh essentially the data points that go into a model.
• 10:18
  SPEAKER_01: An example of a feature, say in Uber's case, would be the number of rides taken by a user in one week.
• 10:24
  SPEAKER_01: Now that value is changing as the days go by.
• 10:28
  SPEAKER_01: So there's this whole rolling aggregation that you have to do to constantly keep that feature updated.
• 10:34
  SPEAKER_01: And because a stale feature, if it goes into a model and a model makes a decision on that stale data point, it's the wrong decision.
• 10:42
  SPEAKER_01: So a feature store is essentially this whole data management system which abstracts all this away.
• 10:47
  SPEAKER_01: You simply define a feature and then you you you give it some configurations and it just makes it ready for you, and you can just consume it via API.
• 10:59
  SPEAKER_01: So this idea of building this uh you know data ready for machine learning systems really blew up.
• 11:08
  SPEAKER_01: Uh, but I realized that uh the the cost of bad data in AI systems is catastrophic.
• 11:14
  SPEAKER_01: It leads to wrong decisions, and those wrong decisions percolate to downstream systems, and you end up with an answer that the user sees that makes absolutely no sense.
• 11:25
  SPEAKER_00: Got it.
• 11:26
  SPEAKER_00: And let's say when you heard in uh about 2021 about language models, can can you share in now that they are popular, but to our audience, what does language of model mean back then?
• 11:37
  SPEAKER_00: How do you understand to a layman?
• 11:39
  SPEAKER_01: Absolutely.
• 11:40
  SPEAKER_01: So uh funny enough, the the definition of a language model hasn't really changed much over the years.
• 11:48
  SPEAKER_01: So to dumb it down, uh a language model is uh a machine learning model.
• 11:55
  SPEAKER_01: And uh machine learning is of course, uh when I say a machine learning model, it is essentially a deep neural net.
• 12:03
  SPEAKER_01: Uh so all these models like OpenAI, uh OpenAI's models, anthropics models, if you shine a magnifying glass on them, they're essentially a deep neural network uh which is which has very specific architectures behind the scenes.
• 12:17
  SPEAKER_01: But what they really are are any deep neural network is essentially a memory machine.
• 12:22
  SPEAKER_01: And the whole process of training a model is making the model remember patterns in the data.
• 12:28
  SPEAKER_01: But um uh data to a model is just ones and zeros.
• 12:32
  SPEAKER_01: So you can feed it language, you can feed it images, and it makes no difference to a model.
• 12:37
  SPEAKER_01: It will just remember the patterns and make predictions based on that.
• 12:40
  SPEAKER_01: So a language model, in the in the more modern sense, is what used to be called as a seek-to-seek model, which is uh the output of this neural net is essentially one token generation after another in sequence.
• 12:57
  SPEAKER_01: How do you define it?
• 12:59
  SPEAKER_01: Okay, a token is a uh for simplicity, it think of it as a word.
• 13:04
  SPEAKER_01: Uh so if you have a sentence, uh say how are you, the three tokens or four tokens in them are how are you, and then the question mark.
• 13:13
  SPEAKER_01: Um, that's the simple definition.
• 13:15
  SPEAKER_01: In reality, uh what a model does is it breaks down these words into subtokens and multiple tokens, but that's you know that that's essentially machine learning language, which is abstracted away from the user.
• 13:28
  SPEAKER_01: Um, but that that's the reason why you see uh any LLM basically spit out one token at a time.
• 13:36
  SPEAKER_01: And that's why you see the streaming.
• 13:37
  SPEAKER_01: It's essentially at each decision, at each token, it's making a choice of what token to spit out next.
• 13:46
  SPEAKER_01: And that choice is essentially a probability distribution of all of vocabulary.
• 13:52
  SPEAKER_01: So think of the entire English uh dictionary.
• 13:55
  SPEAKER_01: That's whatever, uh half a million to a million tokens.
• 13:59
  SPEAKER_01: So at each decision point, the model is choosing what's the next best word, but based on the previous word using techniques like attention and a few other uh concepts.
• 14:10
  SPEAKER_01: Uh, but it's really just choosing one word and the choosing the next best word to say after each previous word.
• 14:16
  SPEAKER_00: So, where did this insight come from?
• 14:18
  SPEAKER_00: That hey, uh obviously you worked at Michael Angelo, that observability uh for these large language models and AI in production could be the future.
• 14:30
  SPEAKER_01: So I realized that AI has always had a measurement problem.
• 14:34
  SPEAKER_01: And this goes far before language models were even a thing.
• 14:38
  SPEAKER_01: Even simple models like XGBoos, decision tree classifiers, it was always hard to quantify good or bad.
• 14:45
  SPEAKER_01: The output of a model, which is a decision, how do you know whether it's good or not?
• 14:49
  SPEAKER_01: Except in the previous world, when we used these simpler models, the input to them was uh uh these features that we were talking about.
• 15:00
  SPEAKER_01: And often it's easy to do certain basic correlations to really dig into why a model chose a particular uh decision.
• 15:09
  SPEAKER_01: The whole ballgame is different with language models because language models are uh these token spewing memory machines, which is essentially there's so much randomness and probability in them.
• 15:22
  SPEAKER_01: It's very hard to say that this was the final answer based on that.
• 15:27
  SPEAKER_01: You know, why did the model say what it did?
• 15:30
  SPEAKER_01: It's a hard problem, and that makes observability of these newer systems even harder.
• 15:35
  SPEAKER_01: But that was the original problem that we were chasing.
• 15:38
  SPEAKER_01: Like, can we dive deeper into these models and really understand why a model chooses the next word to say?
• 15:44
  SPEAKER_01: And a lot of our initial algorithms that we had built in the early days of Galileo was essentially using math and statistics to quantify uncertainty.
• 15:55
  SPEAKER_01: So, uncertainty uh is another technical word, but it's really to in simple words, it's how sure was the model that the next word is the right thing to say.
• 16:06
  SPEAKER_02: Okay.
• 16:07
  SPEAKER_01: So and there's some signals you can get from these models.
• 16:09
  SPEAKER_01: So we took that, the the seed of that idea and built some algorithms to give you whether, you know, how certain was the model in its answer.
• 16:19
  SPEAKER_01: And from there, we've done other more advanced methods to really quantify hallucinations and some of the more advanced issues we see.
• 16:27
  SPEAKER_01: But it really comes down to this whole idea of uncertainty.
• 16:31
  SPEAKER_01: How do you quantify uncertainty in these models?
• 16:33
  SPEAKER_00: Who are your first customers and what's the first use case that you solved for the first 10 customers?
• 16:39
  SPEAKER_01: Yeah, so given we started before LLMs were a thing, a lot of the workflows, especially in the enterprise, was driven around fine-tuning uh some of the smaller language models.
• 16:51
  SPEAKER_01: So for those who've only learned of language models after Chat GPT, they wouldn't know about the erstwhile models.
• 16:58
  SPEAKER_01: But these models were already there.
• 17:01
  SPEAKER_01: In fact, even before GPT 3.5, you had GPT 2 and GPT 1, which were essentially similar architectures as ChatGPT, the modern Chat GPT, except the number of um the size was much lower and smaller.
• 17:16
  SPEAKER_01: And in in the AI world, we measure the size of the models through what we call the number of parameters.
• 17:22
  SPEAKER_01: So if you ever hear people say, Oh, I have a 500 billion parameter model, they are basically saying they have a really large model.
• 17:29
  SPEAKER_01: So coming back to the workflows, folks were working with much smaller models back then, typically in the 100 to 400 million parameter range.
• 17:39
  SPEAKER_01: So very small size.
• 17:41
  SPEAKER_01: It's so small that in theory you could run them on CPUs.
• 17:44
  SPEAKER_01: Okay.
• 17:45
  SPEAKER_01: They would be slow because it's sequential.
• 17:47
  SPEAKER_01: Um, but the GPU infrastructure also needed to serve these models was much less uh uh resource intensive.
• 17:56
  SPEAKER_01: Uh and the workflows back then were primarily around contact center AI and uh entity detection, and these were this is what we call tasks.
• 18:05
  SPEAKER_01: So entity linking, entity detection, classification, these were standard language modeling tasks that enterprises were doing.
• 18:13
  SPEAKER_01: And our first product was uh essentially trying to make these uh the model decisioning of these tasks much higher in quality.
• 18:23
  SPEAKER_01: And we realized that the main determinant of that is the data that you feed into the model.
• 18:28
  SPEAKER_01: Because the model is really just an equation.
• 18:31
  SPEAKER_01: There's not much you can tune to it except a few hyperparameters, but that's not the real fuel that you need to improve the quality of the model.
• 18:39
  SPEAKER_01: Uh, and our first customers were mostly um uh innovative startups and mostly in the contact center AI space or similar thereabouts.
• 18:50
  SPEAKER_01: I remember our first big customer um was a very large bank in the US.
• 18:56
  SPEAKER_01: And they had many contact center AI teams.
• 18:58
  SPEAKER_01: They were also building their native chatbots.
• 19:02
  SPEAKER_01: But this was again in the previous era before LLMs, and all those teams have now shifted to LLMs uh and they continue to be our customers because we've moved with them.
• 19:13
  SPEAKER_01: And and how did the product evolve since then?
• 19:17
  SPEAKER_01: We've had a very interesting journey with our product given we've seen the entire arc of language models for the last six years.
• 19:26
  SPEAKER_01: We uh originally had this product, which uh essentially was a data scientist tool.
• 19:36
  SPEAKER_01: And data scientists are folks who train and evaluate models and uh they really understand the depths of these uh model architectures.
• 19:44
  SPEAKER_01: And the main thing they actually work on, though, is curating data.
• 19:48
  SPEAKER_01: Because at the end of the day, you're not really uh uh a scientist sitting building new model architectures.
• 19:54
  SPEAKER_01: Most data scientists are essentially data curators, is our realization.
• 19:59
  SPEAKER_01: So data was the biggest leverage, biggest bottleneck.
• 20:03
  SPEAKER_01: And we built a platform that would essentially hook into these model training jobs and detect issues in your data using the same sort of statistical modeling techniques.
• 20:16
  SPEAKER_01: Uh, and that became quite a popular tool, which many of the data scientists in our early customers really loved because they saw 60 to 70 percent improvements in their model quality in a single shot simply by using our product, versus it would take them.
• 20:32
  SPEAKER_00: Because your product could produce better quality of data?
• 20:35
  SPEAKER_01: No, so uh well, yes.
• 20:39
  SPEAKER_01: Uh but the main thing our product did was detect issues in the user's data, including label issues, including data issues, in a single shot.
• 20:50
  SPEAKER_01: So you could essentially just grab all the data that you have, pass it through our system, and we would tell you exactly what the high-quality data you should train your models with.
• 21:01
  SPEAKER_01: So it became this pass-through filter, uh, which saved weeks and months of work that data scientists were originally doing.
• 21:08
  SPEAKER_00: But what were data scientists at these enterprises back in 21-22 working on such cutting-edge problems?
• 21:14
  SPEAKER_00: Like they were trying to train their data like models using clean data?
• 21:18
  SPEAKER_01: That's a good question.
• 21:20
  SPEAKER_01: Uh, there were some cutting-edge teams who were working on not the generative type of models.
• 21:26
  SPEAKER_01: That's a whole different uh part of our story.
• 21:29
  SPEAKER_01: And we did get into generative model even before ChatGPT, uh, because there were far and few teams who were doing these sort of pre-Chat GPT generative model but uh use cases.
• 21:42
  SPEAKER_01: But those use cases were far and few.
• 21:44
  SPEAKER_01: The main use cases were kind of these models acted as behind the scenes decision makers.
• 21:50
  SPEAKER_01: Uh, but the the traditional chatbot systems usually had a business logic and code that would power them.
• 21:58
  SPEAKER_01: And that code would use the Decisions from the models, but the model's output would never reach the user.
• 22:05
  SPEAKER_01: And then there were maybe 1% of our customers were doing seek-to-seek modeling.
• 22:13
  SPEAKER_01: Now, seek-to-seek modeling is the token generation modeling, which today is known as LLMs.
• 22:19
  SPEAKER_01: But they were very far and few because this whole use case around, hey, I'm going to use a small model and fine-tune it and make it spew out tokens to generate a sentence.
• 22:31
  SPEAKER_01: The quality of the sentence generation was very poor.
• 22:35
  SPEAKER_01: In fact, you can go to Hugging Face and try out some of these older models.
• 22:39
  SPEAKER_01: And I assure you, for anyone who's only worked with ChatGPT, they'll be shocked.
• 22:44
  SPEAKER_01: In 2020, I think at OpenAI, they had done a lot of experiments on sort of having step function increases in the parameter size.
• 22:53
  SPEAKER_01: That's what led to these models actually constructing coherent sentences.
• 22:58
  SPEAKER_01: But for the vast majority of the world, these generative models could barely talk.
• 23:04
  SPEAKER_01: They were like a baby.
• 23:05
  SPEAKER_01: They would say some random English sentences, which would not make sense.
• 23:09
  SPEAKER_01: And there's a lot of old videos on Twitter you can see.
• 23:12
  SPEAKER_01: But those were not our primary use cases.
• 23:15
  SPEAKER_01: That's why it got really exciting when ChatGPT happened for us, because we've spent all these years working on language model research.
• 23:23
  SPEAKER_01: How can we keep our research DNA but apply it to the newer cutoff models?
• 23:29
  SPEAKER_01: And now we actually have real businesses who are very excited about using this technology.
• 23:35
  SPEAKER_01: So it was around 2023 or late 2022, early 23 is when we are like, man, this could be a really big business.
• 23:43
  SPEAKER_00: So observability, uh you got this insight with the launch of Chat GPT in 2023 that it could become a very large business.
• 23:54
  SPEAKER_01: That's right.
• 23:56
  SPEAKER_00: But what about the enterprise adoption?
• 23:58
  SPEAKER_00: How how fast or slow were these enterprises from 2021 till 2026 early, right?
• 24:05
  SPEAKER_00: On on like if you have to lay out the adoption curve?
• 24:10
  SPEAKER_01: Yeah, that's a great question.
• 24:12
  SPEAKER_01: So I think my observation just looking at uh the our customers, our users, has been that there's in many of the enterprises, especially FSI, you know, banks, telecom, they have AI centers of excellence, and many follow this hub and spoke model where you'd have one AI team that would build out the tools and the technologies for other others to consume.
• 24:40
  SPEAKER_01: Yeah.
• 24:40
  SPEAKER_01: And so that model still existed back then.
• 24:43
  SPEAKER_01: It was just that the AI centers of excellence um were filled with very uh niche-skilled data scientists who likely had PhDs and were very good at unstructured modeling.
• 24:55
  SPEAKER_01: Because a lot of uh AI back then was essentially based on structured data.
• 25:00
  SPEAKER_01: And uh while there were those teams, the use cases were very different, at least for us.
• 25:05
  SPEAKER_01: And we only wanted to work on unstructured AI use cases, be it uh language, voice, images.
• 25:12
  SPEAKER_01: Uh so a lot of those teams were very specialized teams, uh, often research labs that would you know hire PhDs only who have worked on this stuff.
• 25:20
  SPEAKER_01: And it was a very niche skill.
• 25:23
  SPEAKER_01: Um, that I think led to teams moving slow in general.
• 25:28
  SPEAKER_01: One, it's hard to hire these people.
• 25:30
  SPEAKER_01: And two, a lot of these people would want to join a Google or some big tech sort of fang-like company who would pay more.
• 25:37
  SPEAKER_01: Uh, so there were all these challenges.
• 25:39
  SPEAKER_01: Ever since Chat GPT, I've seen the adoption of AI in enterprises increase its velocity manifold.
• 25:49
  SPEAKER_01: And the reason is because the niche skill has been commoditized.
• 25:54
  SPEAKER_01: Now you have a black box, and the whole point is how do you build a software and infrastructure layer around this black box that can make it more predictable, you can control it better.
• 26:04
  SPEAKER_01: That's uh that's not a modeling problem, that's a software engineering problem.
• 26:09
  SPEAKER_01: So, banks and telecom companies and healthcare companies, while their engineering and innovation has stepped up in its velocity, the main bottleneck now is compliance, security, and just general resilience.
• 26:25
  SPEAKER_01: How do you build reliable AI?
• 26:27
  SPEAKER_01: Which is the hot question, and that puts us in the eye of the storm as a company.
• 26:32
  SPEAKER_00: And uh, you know, uh Luna one was your first model uh that you built.
• 26:39
  SPEAKER_00: Why did you build a model in the first place?
• 26:43
  SPEAKER_01: Yes, so our Luna story is fascinating.
• 26:46
  SPEAKER_01: So for those who don't know Luna, Luna is a small language model, yeah, which is specifically designed to solve the evaluation problem in AI.
• 26:57
  SPEAKER_01: So, how do you know whether you know an output is good or bad, or an input has some PII or any security issues?
• 27:04
  SPEAKER_01: So these are very specific uh uh tasks which Luna is designed to solve for.
• 27:10
  SPEAKER_01: So by default, the size of the Luna model is much smaller.
• 27:15
  SPEAKER_01: It is orders of magnitude smaller than, say, a general LLM, which is out there.
• 27:20
  SPEAKER_01: Uh, in fact, our latest Luna models, which are some of our largest, they are the they typically range from one from one to three billion parameters, which is minuscule compared to some of the larger foundational models out there.
• 27:33
  SPEAKER_01: Um, Luna came into the picture when we realized that LLMs as judges, which was the traditional way that people were using to evaluate, and that has a whole history of it, um, they don't scale.
• 27:48
  SPEAKER_01: They don't scale in production, they don't allow you to do full scale observability, which means intercepting every single input and output of these AI systems.
• 28:01
  SPEAKER_01: They simply choke.
• 28:02
  SPEAKER_01: They are very expensive and they're highly unoptimized to solve for the latency and cost problem.
• 28:09
  SPEAKER_01: So we took this problem and kind of came back to the drawing board saying that, hey, how do we distill all this intelligence and reasoning abilities of LLMs and bring it down to a much smaller particular set of tasks?
• 28:22
  SPEAKER_01: Essentially converting these writers, like these LLMs are basically token spewers and writers into calculators.
• 28:29
  SPEAKER_01: Got it.
• 28:30
  SPEAKER_01: That's the difference between uh that the main difference between Luna and a general LLM is that Luna is optimized for a very limited space problem and it solves that really well, versus uh LLMs are optimized for general reasoning and uh general uh generation capabilities.
• 28:49
  SPEAKER_00: Got it.
• 28:51
  SPEAKER_00: So so stepping back, you know, um uh one of the core areas for for you also became during the course uh that uh latency has been a big issue, right?
• 29:02
  SPEAKER_00: And for you working for the future also until now, how did you attack this problem of latency?
• 29:09
  SPEAKER_01: We saw it firsthand that when you make a call to an LLM, yeah, it takes many seconds.
• 29:14
  SPEAKER_01: So 15 to 20 seconds, 15 to 20 seconds, sometimes even higher if you're in thinking mode and you're so you know, trying to or deep research takes or deep research takes, of course, minutes.
• 29:24
  SPEAKER_01: And uh we knew that if enterprises and businesses are gonna adopt this technology, they're not gonna sit and uh wait for for for for 20, 15, 20 seconds, yeah, especially for observability, where time is of the essence in observability.
• 29:41
  SPEAKER_01: Latency is one of the top three things you solve when you build an observability system.
• 29:45
  SPEAKER_01: The whole point of observability is detecting an issue right when it happens and signaling it back to the user.
• 29:54
  SPEAKER_01: So you there's this is a solid problem in traditional software.
• 29:57
  SPEAKER_01: Yeah, SRE has been doing it for the decades.
• 30:00
  SPEAKER_01: SRE's have been doing it for decades.
• 30:02
  SPEAKER_01: The template is out there, there's many unicorns and billion-dollar companies which have you know come out in the observability space.
• 30:09
  SPEAKER_01: Splunk, one of them.
• 30:10
  SPEAKER_01: Splunk is one of them, yes.
• 30:12
  SPEAKER_01: And um uh so it has always been a critical mission critical problem.
• 30:18
  SPEAKER_01: But the key factor there is every observability system has to be low latency.
• 30:23
  SPEAKER_01: Yeah, because latency is the number one killer.
• 30:25
  SPEAKER_01: Yeah, uh, there's no point of observing something when it's already done and it's been 20 seconds.
• 30:30
  SPEAKER_01: So that's the key insight that we found as a team that hey, these LLM systems are pretty nuanced and complex.
• 30:37
  SPEAKER_01: The outputs of these are very hard to quantify objectively, whether they're good or bad.
• 30:42
  SPEAKER_01: So you have to somehow use this technology itself to do evals and observability.
• 30:48
  SPEAKER_01: And we wrote a whole paper on this, on how you use other LLMs to judge the outputs of LLMs.
• 30:54
  SPEAKER_01: But the uh the simple paradigm of making another LLM call to evaluate a different LLM simply won't scale.
• 31:02
  SPEAKER_01: So that was kind of the the aha moment for us to come back to the drawing board and really figure out how do we build low latency systems in this era.
• 31:13
  SPEAKER_01: And that is one of the things that has become the topmost problem for you know the Fortune 500 and beyond.
• 31:21
  SPEAKER_01: Every leader is figuring out how do I productionize my agents today.
• 31:26
  SPEAKER_01: And I'm not gonna fly blind.
• 31:29
  SPEAKER_01: So no one's gonna put an agentic system, especially if they do actions and tasks, they're not gonna do it without any kind of observability.
• 31:36
  SPEAKER_01: And if the entire observability playbook fails, that worked for traditional applications, it fails in the new era.
• 31:43
  SPEAKER_01: What tools do I have to observe, observe these agents?
• 31:47
  SPEAKER_01: Uh, that's the kind of that's the key gap that we fill.
• 31:51
  SPEAKER_01: And that is the problem that we've been working on, uh, especially focusing on production, because production is a bit of an infrastructure problem.
• 32:00
  SPEAKER_02: Yeah.
• 32:00
  SPEAKER_01: Uh Luna is, of course, one of the key ingredients.
• 32:04
  SPEAKER_01: And I think there's two parts to Luna's innovation.
• 32:07
  SPEAKER_01: There's the modeling side, which is how do you tame the intelligence of LLMs into smaller models and use small models efficiently.
• 32:17
  SPEAKER_01: But a vast majority of the performance games and latency gains that we see is on the infrastructure side, which are these uh AI platforms and LLM inference platforms that allow us to host Luna.
• 32:30
  SPEAKER_01: So, for example, we've built our own in-house LLM inference engine, which is designed to host small language models.
• 32:38
  SPEAKER_01: So they're not designed to host the larger models.
• 32:41
  SPEAKER_01: We've done tons of optimizations, including techniques like low-rank adaptation that allows us to run many Luna models under a single foundational base.
• 32:52
  SPEAKER_01: So you need only a single GPU.
• 32:54
  SPEAKER_01: But NetNet, it allows us to build these evals which can run at breakthrough latencies, 100 milliseconds and below, which has not been seen by the world.
• 33:05
  SPEAKER_01: And they are actually capable of evaluating a complex end-to-end interaction between a human and an agent.
• 33:12
  SPEAKER_01: So that was the key unlock for Galileo.
• 33:15
  SPEAKER_00: And if you have to explain, because evals have become really popular in basic uh you know terminology, what are evals and why they have become so popular?
• 33:23
  SPEAKER_01: Yes, so eval stands for evaluations.
• 33:27
  SPEAKER_01: And uh an evaluation is essentially a metric or a signal that you need to judge whether the output of an LLM is good or bad.
• 33:40
  SPEAKER_01: That's the simplest definition.
• 33:41
  SPEAKER_01: The question is how you do it.
• 33:43
  SPEAKER_01: Um, of course, eval spreads into input.
• 33:48
  SPEAKER_01: There's all different components to an uh an LLM request, especially if you're building an application for the app developers out there, they would know that in order to build even a simple application that uses LLMs, there's some sort of a database lookup, you'll have to set up a vector store, uh, and you'll have some system of record.
• 34:09
  SPEAKER_01: So there's many different components, and one small issue in any of them can lead to a bad output.
• 34:16
  SPEAKER_01: So the whole discipline of evals is how do you prevent bad outputs from going back to the user that can be catastrophic for a business.
• 34:28
  SPEAKER_00: And uh you have been a big proponent of small language models, right?
• 34:32
  SPEAKER_02: Yes.
• 34:32
  SPEAKER_00: Why is that?
• 34:33
  SPEAKER_00: Whereas uh, you know, today uh the earlier the uh the gap was that uh LLMs are hallucinating, right?
• 34:42
  SPEAKER_00: And they can't be as accurate as small language models, but the way LLMs are getting better every day, they'll outpace LMs.
• 34:51
  SPEAKER_01: That's a very interesting point and an interesting thought.
• 34:55
  SPEAKER_01: Um you're right that LLMs are not only getting smarter, but they're also getting cheaper to deploy.
• 35:03
  SPEAKER_00: Yeah, yeah.
• 35:03
  SPEAKER_00: Every year the cost of uh uh LLMs is reducing by 80%.
• 35:08
  SPEAKER_01: That's correct.
• 35:09
  SPEAKER_01: So they are reducing by 80%, and these models are getting smarter, but there's a few factors at play here.
• 35:16
  SPEAKER_01: One is, of course, latency.
• 35:19
  SPEAKER_00: And and I assume latency they'll also solve uh with time, right?
• 35:23
  SPEAKER_00: Uh now, even if you throw complex queries to Clot today, uh the results are usually under a minute.
• 35:30
  SPEAKER_01: No, that's uh that's totally fair.
• 35:32
  SPEAKER_01: So all these factors are going down, but despite that, there's some nuances in the in the economics of this, which you'll see that it's despite these improvements, the the challenges around evals will still remain.
• 35:46
  SPEAKER_01: The first reason for that is the more the usage of LLMs in the enterprise and the more Gen AI adoption, uh the infrastructural challenges will keep growing.
• 35:59
  SPEAKER_01: Because even if the per you know unit latency reduces, more traffic means you're just uh subjecting the system to more tokens and just more data.
• 36:11
  SPEAKER_01: And all that really brings the system down.
• 36:14
  SPEAKER_01: That's one.
• 36:15
  SPEAKER_01: Uh number two is agent interactions are getting more and more complex.
• 36:19
  SPEAKER_01: So even at a unit level, if if say you apply some magical engineering and solve the problem of, hey, I reduced a 10-second, 15-second complex generation to one second, agentic interactions will compound.
• 36:33
  SPEAKER_00: Yeah.
• 36:34
  SPEAKER_01: Because it's not about a single request.
• 36:37
  SPEAKER_01: And the software that we are building, the new era of agentic software, they're gonna get more and more sophisticated, more and more nuanced.
• 36:45
  SPEAKER_01: That will just keep widening the gap.
• 36:47
  SPEAKER_00: Yeah, you are right.
• 36:48
  SPEAKER_00: The the kind of complexity they're following every day is is increasing.
• 36:53
  SPEAKER_01: Exactly.
• 36:53
  SPEAKER_01: And the complexity of the use cases are increasing, and really the the the thing that LLM providers are optimizing for, the number one thing they're optimizing for, is general reasoning and making the intelligence of these models better.
• 37:12
  SPEAKER_00: Uh what do you mean by general reasoning?
• 37:14
  SPEAKER_01: Uh, every new model that comes out, you publish new academic benchmarks and you see that you know it's it's able to achieve a higher score on a particular benchmark.
• 37:24
  SPEAKER_01: And that just means that it's able to solve general purpose problems better.
• 37:28
  SPEAKER_02: Okay.
• 37:28
  SPEAKER_01: And that makes sense for the hyperscalers.
• 37:32
  SPEAKER_01: And even if they're you know getting into enterprises, which we've seen a lot of hyperscalers are solving for enterprise use cases, uh, the the general techniques that they're the innovations that they're doing on the modeling side, uh there's a bit of innovators, uh the little bit of uh, what do you say, like push and pull between um, hey, I want to solve a very specific banking use case, uh, but I also want this model to be you know much better at other things.
• 37:59
  SPEAKER_01: So you're gonna throw more data into it, you're gonna have more fine-tuned, and yes, you're gonna increase the size of these models.
• 38:06
  SPEAKER_01: But despite that, there's many variables which are not in the control of these LLM providers.
• 38:12
  SPEAKER_01: Yeah.
• 38:13
  SPEAKER_01: One is the data.
• 38:15
  SPEAKER_01: You have no control over the use cases, and data is gonna change, and it'll be like it's like a fingerprint, right?
• 38:21
  SPEAKER_01: It's different for uh different people.
• 38:24
  SPEAKER_01: So one of the things that we do with SLMs is not only is it small, so the net economics of it is just disproportionately smaller, right?
• 38:33
  SPEAKER_01: It is uh it is so uh cheap to run a single inference on an SLM versus an LLM.
• 38:39
  SPEAKER_01: So that gap will always be there.
• 38:41
  SPEAKER_01: We are kind of solving the opposite problem of hey, let the models get better, let the cost of intelligence reduce over here.
• 38:49
  SPEAKER_01: It is in our favor because some of the base models we are using are getting more in.
• 38:54
  SPEAKER_01: What we are doing is we are adding a layer on top of it which leverages the specificity of the customer data, and that becomes a very big competitive mode.
• 39:04
  SPEAKER_01: Um, and then there's the question of privacy.
• 39:07
  SPEAKER_01: There's a lot of hesitation on sending very key proprietary data.
• 39:10
  SPEAKER_01: It has always been the point.
• 39:12
  SPEAKER_01: And even though there's a lot of cloud adoption, uh, the fear of AI is 10x, the fear of putting data on the cloud.
• 39:21
  SPEAKER_01: People have a lot of trepidations around what is are they training on my data?
• 39:26
  SPEAKER_01: And I don't want to send my data to an AI.
• 39:28
  SPEAKER_01: It's such a black box.
• 39:30
  SPEAKER_01: So that fear is gonna be there.
• 39:32
  SPEAKER_01: Hopefully, it will subside over the next maybe 10 years.
• 39:36
  SPEAKER_01: But even today, 10 years after the you know, the whole cloud revolution, there's still questions around, you know, people still talk about data privacy and uh there's all these compliances, etc.
• 39:46
  SPEAKER_01: So there's all these different motions at play.
• 39:49
  SPEAKER_01: We feel like if you really focus on the fundamental problem that, hey, evaluations for language models is a very task-specific, constraint problem, and it does not require you to use uh, you know, general purpose reasoning for that.
• 40:06
  SPEAKER_01: And then there's fine-tuning you can do on the proprietary data, you can achieve really high accuracy with very low resources.
• 40:14
  SPEAKER_01: I feel like that's a headwind that will always rather that's a tailwind that's always in your favor, no matter how cheaper the models get, there will always be better models.
• 40:26
  SPEAKER_01: Uh, but one thing is a fact that there will be more AI usage over the next many years.
• 40:35
  SPEAKER_01: People are adopting it, just the numb net usage of these AI systems is going crazy.
• 40:41
  SPEAKER_01: We already know uh OpenAI and Anthropic, their revenues have just catapulted in the last few months and it just goes up.
• 40:50
  SPEAKER_00: Anthropic is at 40 billion dollars of run rate, right?
• 40:53
  SPEAKER_01: Right, and they they they they were valued at uh 18 billion from you know maybe a year and a half ago, and from there they're uh almost a trillion dollars.
• 41:01
  SPEAKER_01: Almost a trillion dollars, and and the revenues have kind of seen the same level.
• 41:05
  SPEAKER_01: So this will keep going up.
• 41:07
  SPEAKER_01: Uh the technology is very real and everyone realizes it.
• 41:10
  SPEAKER_01: Yeah, that'll increase the traffic.
• 41:12
  SPEAKER_01: Uh so it's smart to think about evals and observability in a much more constrained way, and how do you solve it cheaply?
• 41:21
  SPEAKER_01: Because it's the cheap and efficient uh uh ability of these SLMs that can really help it scale no matter how much the AI footprint is.
• 41:32
  SPEAKER_00: So you are saying uh there's a huge opportunity building first of all in SLMs for domain specificity, and then building evals for those uh SLMs.
• 41:44
  SPEAKER_01: Well, I'm actually mostly referring to evals and observability, but but you're right.
• 41:48
  SPEAKER_01: There's uh there's many instances where uh you can actually train and fine-tune an SLM and uh use that for say generational capabilities.
• 41:57
  SPEAKER_01: Uh, but uh for me that's uh it's a non-issue in the sense that, hey, you can continue using newer models, and you should, because they are genuinely better and they will get better and better at doing more advanced things.
• 42:12
  SPEAKER_01: So you should not come back to the drawing board and use smaller models for those.
• 42:17
  SPEAKER_01: If it suits you for your use case, then perhaps, but that's a decision you should make.
• 42:22
  SPEAKER_01: Uh, but generally, if you're building any kind of general application system that's solving real problems, say CRM or uh you know any kind of end-to-end workflow, you should try to disrupt it with these newer models.
• 42:35
  SPEAKER_01: So, my my contention is go ahead, use the best models.
• 42:39
  SPEAKER_01: But observability is a bit of a different ballgame because it has a different set of challenges.
• 42:44
  SPEAKER_01: It's an infrastructural problem and latency and cost and quality.
• 42:49
  SPEAKER_01: Those are the three things, and they are a bit of uh opposing forces with each other.
• 42:53
  SPEAKER_01: Uh, it's like this triad of you know, low cost, low latency, high quality.
• 42:59
  SPEAKER_01: It's you can only get two of the three if you use traditional techniques.
• 43:02
  SPEAKER_01: So uh it's very important to you know find that equilibrium where how can you leverage high accuracy of evaluations and observability at low cost and low latency.
• 43:14
  SPEAKER_00: God.
• 43:15
  SPEAKER_00: And and today, let's say if your revenue is X, how much of it comes from observability versus evals?
• 43:22
  SPEAKER_01: Um so it's kind of the whole online offline uh disparity, right?
• 43:27
  SPEAKER_01: Like uh um, and this has been there since in since the dawn of machine learning.
• 43:32
  SPEAKER_01: And one of the key problems is online offline parity.
• 43:35
  SPEAKER_01: That hey, I built my agent in my experimental uh environment and now I put it into production and it's not behaving the way I tested it.
• 43:44
  SPEAKER_01: You know, it it worked on my local host, is the is the famous saying.
• 43:48
  SPEAKER_00: The open cloud works on my machine.
• 43:50
  SPEAKER_01: Exactly.
• 43:50
  SPEAKER_01: Exactly, exactly.
• 43:52
  SPEAKER_01: It worked on my machine, is the last thing they said before it all blew up.
• 43:55
  SPEAKER_02: Yeah.
• 43:56
  SPEAKER_01: So it's a common problem.
• 43:57
  SPEAKER_01: It has been there for years.
• 43:58
  SPEAKER_01: It's Only a bigger problem with agents because they are harder to evaluate.
• 44:04
  SPEAKER_01: So coming back to your question, there's evals.
• 44:07
  SPEAKER_01: Evals is kind of the foundations of it, but uh offline evaluation is what we call it, which is the whole experimental setup to get to a first version of your agent that you are confident of.
• 44:21
  SPEAKER_01: And then there's online observability, which is how do you measure in real-time in production, either uh across your enterprise or at your client's destination.
• 44:32
  SPEAKER_01: Exactly.
• 44:32
  SPEAKER_01: Exactly.
• 44:33
  SPEAKER_01: Uh, one of our realizations, and and and we were early to realize this at Galileo, is that online and offline observability are two parts of the same coin.
• 44:43
  SPEAKER_01: In fact, it's not a point-in-time workflow that you do, they are this connected flywheel.
• 44:49
  SPEAKER_01: And I'll explain to you what this means.
• 44:51
  SPEAKER_01: So, say you're an app builder, you build an agent, and you test your agent on a local host.
• 44:57
  SPEAKER_01: Uh, what you should do, what is good practice, is define the behavior of your agent.
• 45:04
  SPEAKER_01: What all do you want it to do well?
• 45:07
  SPEAKER_01: And what does good mean?
• 45:08
  SPEAKER_01: What does bad mean?
• 45:09
  SPEAKER_01: Define them in the form of evals, is what we call it.
• 45:13
  SPEAKER_01: So, an eval is this entity which is giving you a quantified output of good or bad.
• 45:20
  SPEAKER_01: Uh, so you build a bunch of these evals and let those evals fail.
• 45:23
  SPEAKER_01: In fact, you build the evals before you build the app.
• 45:25
  SPEAKER_02: Yeah.
• 45:26
  SPEAKER_01: Uh that kind of becomes that's why people say that, hey, evals is the new uh weapon for product managers because they are the ones who are defining the app's behavior.
• 45:35
  SPEAKER_01: So there's this notion of evals-driven development in the offline setting where an app builder is building their app and let the evals fail.
• 45:43
  SPEAKER_01: Let the product managers or the builder themselves set up these evals and let them all be red.
• 45:48
  SPEAKER_01: And as you build, let them auto-turn green.
• 45:51
  SPEAKER_01: And once they're all green, you know that you have some quantified sense of hey, this is a version of the app that I'm confident of.
• 46:00
  SPEAKER_01: Then you ship it to production.
• 46:02
  SPEAKER_01: The production will likely meet some of the eval criteria you had, but it will very likely meet newer criteria which you did not test on.
• 46:10
  SPEAKER_01: In Galileo, we call them unknown unknowns.
• 46:13
  SPEAKER_01: So your agents will fail in nuanced ways where you didn't expect it.
• 46:17
  SPEAKER_01: That's where the discipline of catching these unknown unknowns and creating new evals and bringing them back to the offline testing environment, that's what creates the flywheel.
• 46:28
  SPEAKER_01: And there's many other elements to this flywheel where you have to collect the data and look at the traces and evaluate them, perhaps manually through a subject matter expert, but you can also use offline judges to evaluate them.
• 46:42
  SPEAKER_01: And these are workflows that we are seeing already in banks, in telecom companies, in consumer goods, healthcare.
• 46:51
  SPEAKER_01: They are all starting to build these individual workflows, but it's all disconnected.
• 46:56
  SPEAKER_01: That is the problem.
• 46:58
  SPEAKER_01: And that is specifically the problem that we've solved at Galilee.
• 47:01
  SPEAKER_01: Okay, how do you build this connective tissue of doing evals, then measuring new evals or unknown unknowns, which are fodder for the new evals, and then you do more testing and then release, and then catch issues that are false positives in your evals and have human feedback in the loop.
• 47:21
  SPEAKER_01: And then as you scale your agent, you know these evals, which are LLM judges, they will not scale, then collect the data as the foundational training data for these SLMs, and then you compare the SLM performance with the LLM judge performance, and then once you see parity, then you productionize the SLM, and now you have full-scale observability at throwaway cost.
• 47:42
  SPEAKER_01: This is this whole loop.
• 47:44
  SPEAKER_01: This is Galileo in a nutshell.
• 47:45
  SPEAKER_00: And can you name me uh like uh without sharing the name of the customer, like one one industry or one specific customer from an industry where you saw like you know uh agents in production that that are doing autonomous tasks across various uh verticals or various various use cases using Galileo?
• 48:10
  SPEAKER_01: That's a great question.
• 48:12
  SPEAKER_01: Um, certainly I can say uh the bad news is that the reality is it's still very early.
• 48:18
  SPEAKER_01: Yeah, even in the enterprise, folks are building simple agents.
• 48:23
  SPEAKER_01: Uh the enterprise kind of evolved from building sort of knowledge worker workflows back in 2024, 2025.
• 48:30
  SPEAKER_01: What is a knowledge worker workflow?
• 48:32
  SPEAKER_01: Like internal tools that make employees more efficient.
• 48:34
  SPEAKER_01: So having, you know, kind of like uh a company's like Glean, right?
• 48:39
  SPEAKER_01: Yeah, internal search for everyone.
• 48:40
  SPEAKER_01: Internal search.
• 48:41
  SPEAKER_01: So that's been kind of the hot use case for most companies.
• 48:44
  SPEAKER_01: But that was an old hot use case.
• 48:45
  SPEAKER_01: It's an old hot use case.
• 48:46
  SPEAKER_01: So now we've evolved into one, how do you make these use cases more agentic?
• 48:50
  SPEAKER_02: Yeah.
• 48:50
  SPEAKER_01: It's still internal tools, but they do more things than just QA lookups, uh, like creating a graph or you know, creating a presentation.
• 48:58
  SPEAKER_01: That's the new sort of internal efficiency workflows.
• 49:03
  SPEAKER_01: But uh there are a set of customers who are sort of living on the edge.
• 49:07
  SPEAKER_01: Of course, you can put beta stamps and stuff in your product, which they do uh for AI features.
• 49:13
  SPEAKER_01: Uh, but a lot of the the some of the more cutting-edge companies that use Galileo, uh, the use cases I've seen are um uh these these agents that take you know use tool calls and take action and get data from different APIs and really surface much more nuanced actionable information to the user.
• 49:34
  SPEAKER_01: Uh, and that's really baked into the product.
• 49:36
  SPEAKER_01: Um, without naming a customer specific example, there's a sales intelligence platform that uses Galileo for observability.
• 49:43
  SPEAKER_01: And the the platform, what it does is it gives you uh uh sort of enterprise customer intelligence.
• 49:49
  SPEAKER_01: It's mainly for uh go-to-market teams to discover customers.
• 49:54
  SPEAKER_01: And they've built this beautiful QA chatbot-like system where the interface is still language, but it's doing a lot more things than just answering questions from a database.
• 50:04
  SPEAKER_01: It's going into um, you know, using various APIs and open source CRM tools to get information from you know about the you know what employees are doing and the you know what they're posting on social media, all this very interesting information.
• 50:22
  SPEAKER_01: And you can auto-create these sort of sales battle cards and you know, save them and bookmark them.
• 50:29
  SPEAKER_01: And it's really easy to use and very intuitive.
• 50:31
  SPEAKER_01: I really love that product.
• 50:32
  SPEAKER_01: But behind the scenes, they're running all these autonomous agents.
• 50:36
  SPEAKER_01: Uh, so this is one example.
• 50:37
  SPEAKER_01: Other examples are like AI SRE.
• 50:40
  SPEAKER_01: That's another very hot use case on detecting root cause RCA issues for traditional applications.
• 50:48
  SPEAKER_01: Uh, so they use us as sort of their monitoring their agents.
• 50:52
  SPEAKER_01: Um, the most cutting-edge use cases I've seen, they use our agent control uh product.
• 51:00
  SPEAKER_00: What's an agent control product?
• 51:01
  SPEAKER_01: So agent control uh is a recent product that Galileo launched.
• 51:07
  SPEAKER_01: And as the name says, it is to control agents or provide a control plane for you know seeing the agent what the agents are doing at a particular point in time.
• 51:17
  SPEAKER_01: It is open source.
• 51:18
  SPEAKER_01: So you can Google uh agent control Galileo, you'll find the open source repo.
• 51:23
  SPEAKER_01: And what it this does is uh it's meant to design these policies, which you can apply to specific steps of the agent, and those policies will either shut down the agent or make it reroute to a different tool call.
• 51:39
  SPEAKER_01: And uh what I've seen is it makes the net net agenc experience a lot more reliable and stable.
• 51:45
  SPEAKER_01: Of course, there's that balance you want to find between, you know, it should not be too unstable and stodgy where it says I don't know to everything.
• 51:53
  SPEAKER_01: Uh, but uh it's this control plane that allows you to uh control these agents.
• 51:59
  SPEAKER_01: So some of the more cutting-edge use cases are these long-running agents which are doing longer workflows, like end-to-end.
• 52:05
  SPEAKER_01: Think of any enterprise workflow, whether it's a financial workflow or a CRM workflow, they're attempting to do those with agents end-to-end.
• 52:13
  SPEAKER_01: And the agent control serves as the control layer for essentially observing what's happening at each step.
• 52:19
  SPEAKER_00: Uh, your customers include Reddit, Airbnb, PNG, Comcast, and six of the Fortune 50, right?
• 52:25
  SPEAKER_00: How did you build your GTM?
• 52:27
  SPEAKER_00: And especially like, you know, um uh all of us like coming from India are hardcore engineers.
• 52:34
  SPEAKER_00: So we are not really taught sales back in home.
• 52:37
  SPEAKER_01: Absolutely.
• 52:38
  SPEAKER_01: So I came to the US in 2011 to do research, was in a research lab at UCLA, uh, did my master's there, and like you said, I was a hardcore engineer, uh, loved solving problems, hard problems.
• 52:53
  SPEAKER_01: Um so spent 10 plus years in big tech before starting the company.
• 52:58
  SPEAKER_01: Yeah.
• 52:59
  SPEAKER_01: And I feel like I had to do a lot of unlearning of uh certain habits which I had built while working in big tech because not all problems are Apple and Uber level problems, especially in the enterprise.
• 53:12
  SPEAKER_01: So, first six to eight months of our journey was talking to hundreds of people, and this is in an era where we did not have granola and note takers and you know voice agents, it was all just taking notes in a diary.
• 53:28
  SPEAKER_01: Uh, and we would meet people in in uh in coffee shops here in New York, we would fly.
• 53:35
  SPEAKER_01: So a lot of talking just to really figure out what the problem was.
• 53:39
  SPEAKER_01: My leverage was that I had built AI systems at scale before and truly had uh built some of the in the foundational infrastructure that defined the MLOps industry itself.
• 53:53
  SPEAKER_01: So I brought that to the table.
• 53:55
  SPEAKER_01: Uh, what I had to unlearn was uh how do I build something very simple and very uh narrow that solves a very specific workflow problem.
• 54:07
  SPEAKER_01: Uh and that is something which I learned over dozens of conversations which we had.
• 54:12
  SPEAKER_01: So most of our first year was spent just doing this.
• 54:15
  SPEAKER_01: Yes, we would build some prototypes here and there, but it was mostly throwaway.
• 54:19
  SPEAKER_01: Once we re-figured out that really the problem of trust in AI is a big problem.
• 54:27
  SPEAKER_01: And correlating that with hey, it's uh it is very evident that more and more AI systems in the next three to five years are going to be uh unstructured AI-based systems.
• 54:40
  SPEAKER_01: It really gave us the conviction of the specific problem that machine learning and AI is a garbage in, garbage out problem.
• 54:47
  SPEAKER_01: So we have to solve the garbage in problem.
• 54:49
  SPEAKER_00: So for example, let's say uh imagine I am a potential customer, right?
• 54:55
  SPEAKER_00: Uh, who would my ICP be in your earlier days?
• 54:58
  SPEAKER_00: It would be the CTO, the CIO, or VP engineering in an enterprise.
• 55:02
  SPEAKER_01: So you are the potential customer of Galileo?
• 55:04
  SPEAKER_01: No, yes, the potential customer of Galileo.
• 55:07
  SPEAKER_01: So in our case, our ICPs were the VPs of engineering, uh, or rather VPs of data science back in the day.
• 55:15
  SPEAKER_01: It was a role that apparently is a diminishing breed.
• 55:18
  SPEAKER_01: Uh, but we would go to these advanced AI teams, mostly at larger uh financial companies and telcos, and we would go to Capital One and talk to data science teams there and really ask them like, what are you doing on a day-to-day?
• 55:34
  SPEAKER_01: And what is the key problem?
• 55:36
  SPEAKER_00: Got it.
• 55:37
  SPEAKER_00: And let's say uh initial screening is done, you're you're you're figuring out the problem.
• 55:43
  SPEAKER_00: Now uh now I'm potentially like the 20th customer of Galileo.
• 55:47
  SPEAKER_00: How would you your pitch change now?
• 55:50
  SPEAKER_01: Uh that's a good question.
• 55:52
  SPEAKER_01: I think it uh it it the pitch difference depends on the maturity of the product, of course, but also um the maturity of the the the landscape or the two you know the the the this area is moving so fast that you know between customer one and customer ten and customer twenty, just the the entire wording of the pitch has changed.
• 56:16
  SPEAKER_01: Uh but what has not changed about the pitch is this core problem of uh hey, I'm building AI systems and agents, and I don't want to fly blind.
• 56:29
  SPEAKER_01: How do I measure these AI systems effectively?
• 56:32
  SPEAKER_01: So give me a system to measure good and bad.
• 56:37
  SPEAKER_01: So that has not changed.
• 56:39
  SPEAKER_01: And more than us telling the user, it's the users who tell us that, hey, I'm looking for a way to quantify good and bad in my data.
• 56:47
  SPEAKER_01: And this thing has not changed since day zero of our company when we started early 2021.
• 56:52
  SPEAKER_01: It's still all about how do you quantify good and bad, even in the era of agents.
• 56:56
  SPEAKER_01: It's the same pitch where we say, uh, our pitch is we are fortunate enough to have a product which almost needs very little selling as far as the problem is concerned.
• 57:07
  SPEAKER_01: Because everyone will tell you that, hey, I'm building AI systems, I need a way to measure things.
• 57:12
  SPEAKER_01: I'm not going to productionize it, I'm flying blind very clearly.
• 57:14
  SPEAKER_01: The question is how.
• 57:16
  SPEAKER_01: I think the challenge is how.
• 57:17
  SPEAKER_01: But it comes back to us being engineers and deep in the systems thinkers, uh, which is a great problem to have.
• 57:24
  SPEAKER_01: So our main challenge is hey, how do we build a good product that can solve this measurement problem?
• 57:30
  SPEAKER_00: And uh you recently switched roles from CTO to CPO.
• 57:33
  SPEAKER_00: What's been uh uh behind that transition?
• 57:37
  SPEAKER_01: Yeah, so uh I worked on the technology uh of Galileo.
• 57:42
  SPEAKER_01: I literally built the first version of Galileo in the earliest days uh during our pre-seed days.
• 57:48
  SPEAKER_01: And I've always been a builder and engineer and uh led many teams in engineering, always been in the engineering space right from the beginning, especially in the AI space.
• 57:58
  SPEAKER_01: The new era of software, people are figuring out what it looks like with agents, with gen AI, with tools, and it's just the beginning, right?
• 58:09
  SPEAKER_01: Uh OpenClaw was one blueprint that was published, and it just blew up because it said that here's one step that I've taken at how to build a long-running agentic system, and uh it's only a blueprint.
• 58:24
  SPEAKER_01: OpenAI, uh OpenClaw, my apologies, has uh many flaws.
• 58:30
  SPEAKER_01: And anyone who's worked deeply on the system, it is a brilliant uh idea that's been put out, and it is the the creator of OpenClaw is absolutely fantastic.
• 58:41
  SPEAKER_01: That said, it can be maneuvered in many ways, and I have seen that personally as I've built out you know on my you know personal uh local host laptop.
• 58:51
  SPEAKER_01: I've tried a lot of things and I've tried to move things, uh, really break open claw in many ways, and I've seen that it's while it has its flaws and rough corners, it's the first blueprint of what an AI agentic system could look like.
• 59:06
  SPEAKER_01: So, yes, so um I've played a lot with uh newer Gen AI systems, and I've always been on the cutting edge.
• 59:15
  SPEAKER_01: The form factor, people are still figuring out of what product in the new age of AI looks like.
• 59:22
  SPEAKER_01: And there's uh some ideas that yes, you need still some sort of a browser form factor.
• 59:29
  SPEAKER_01: Uh, some of it is automatable, there's agents in the loop, there's the CLI form factor, which is hot again, and you have cloud code and other sort of CLI applications, there's MCP connectors where you have a chat box.
• 59:41
  SPEAKER_01: So it's all sort of taken the world by storm that hey, traditional enterprise applications, which used to be a dashboard which would show a chart with squiggly lines on it, all that era is over.
• 59:52
  SPEAKER_01: Now it's the era of actions and intelligence.
• 59:55
  SPEAKER_01: So I took on the product uh hat, I wore my product hat because I'm a deep sort of engineer and engineering leader.
• 1:00:04
  SPEAKER_01: I've come from this space, I at least understand it from silicon up.
• 1:00:08
  SPEAKER_01: So it's a good uh role to take in an era where builders are the kings and queens.
• 1:00:17
  SPEAKER_01: And uh, me being just having that builder DNA, I think it helped us uh do more experiments, uh, think about what good product looks like.
• 1:00:28
  SPEAKER_01: Because even the observability platform, it is a product and it needs to be built out really well.
• 1:00:34
  SPEAKER_01: We we we have this internal term we use called umami, and umami means you know something that just feels so good.
• 1:00:41
  SPEAKER_01: Uh, and in an era where you can build in you know a website and a web application with a Postgres database behind in 30 seconds, software is just commoditized, software is just tokens, right?
• 1:00:56
  SPEAKER_01: Code is tokens.
• 1:00:58
  SPEAKER_01: So, how do you build high-quality product in the new era?
• 1:01:01
  SPEAKER_01: Was a key problem that we had to solve, sit, take a step back, think about outside of all the machine learning and AI innovations we do, what is the new product?
• 1:01:10
  SPEAKER_01: That was the key question.
• 1:01:12
  SPEAKER_01: I took on that role to try to answer that question.
• 1:01:15
  SPEAKER_00: So, when software has become tokens, what does what is the final definition of a great product be?
• 1:01:22
  SPEAKER_01: That's a great question.
• 1:01:24
  SPEAKER_01: I was figuring that out on my way here, and the day before that, and the week before that.
• 1:01:28
  SPEAKER_01: That's pretty much all I think about.
• 1:01:30
  SPEAKER_01: I can certainly share my I guess my learnings till now.
• 1:01:34
  SPEAKER_01: Uh, it is um, there's no uh no one knows the right answer per se.
• 1:01:40
  SPEAKER_01: Uh, it is what I've realized that is that you have to meet people and workflows where they are, because it's not a clean sweep, clean canvas.
• 1:01:50
  SPEAKER_01: Right?
• 1:01:50
  SPEAKER_01: There's software is pretty hairy if you really look at what goes on behind the scenes.
• 1:01:57
  SPEAKER_01: There's a lot of traditional software that's already in place which solves really hard problems.
• 1:02:02
  SPEAKER_01: So we have to fit into that and then evolve the form factor.
• 1:02:06
  SPEAKER_01: Uh, the easiest thing to disrupt is what a good UI looks like in the new era.
• 1:02:12
  SPEAKER_01: And I can certainly tell you about um in the observability and eval space, what is the problem that any observability tool really solves, whether it's AI or not AI?
• 1:02:26
  SPEAKER_01: It's the hey, what's wrong question.
• 1:02:29
  SPEAKER_01: Like you ask a system, hey, what's wrong, and then you take 50 steps to figure out what's wrong.
• 1:02:35
  SPEAKER_01: Uh traditionally, we have solved that through statistics, through charts, through drift, through all kinds of mathematical paradigms that we put into a software and that shows up like a chart, and that's what developers do.
• 1:02:49
  SPEAKER_01: But now you're in the era where you can ask a question and get an answer, and there's all these steps in the middle which requires reasoning.
• 1:02:56
  SPEAKER_01: You don't have to build custom business logic for that.
• 1:02:59
  SPEAKER_01: And the primary interface for this question, hey, what's wrong, can be voice and can be language, text, because those are the primary modes of human interaction.
• 1:03:09
  SPEAKER_01: So it all starts with that, and then from there you trigger this workflow where your little agents who are the little workers, they do their things and they bring you the results.
• 1:03:20
  SPEAKER_01: So what's in between is still to be figured out.
• 1:03:25
  SPEAKER_01: Uh, what I know is that the current form factor has to fit into the existing form factors, which is some sort of a browser, some sort of an application.
• 1:03:33
  SPEAKER_01: There is a case to be made for a natural language-based interface, whether it's voice and or or language.
• 1:03:42
  SPEAKER_01: Certainly, a lot of voice-based interfaces are also getting really popular.
• 1:03:46
  SPEAKER_01: We're seeing this movement of voice interfaces becoming the standard as the quality of you know voice models uh are getting better.
• 1:03:53
  SPEAKER_01: You're talking to one of my friends uh later on who is uh uh running a pioneering company in voice.
• 1:04:01
  SPEAKER_01: And uh a lot of these businesses are doing really well for this reason that voice quality is so good.
• 1:04:07
  SPEAKER_01: So, in the end, we'll enter an era where we'll talk to software and software will do work for us.
• 1:04:12
  SPEAKER_01: And everything in the middle is a means to an end.
• 1:04:15
  SPEAKER_01: Whether it's a blue button on the top right, or whether it's a chat chatbot or chat GPT-like interface, it will likely be some sort of a hybrid where a button makes sense, we don't want to replace it.
• 1:04:26
  SPEAKER_01: Uh, but ideally, maybe 10 years down the line, or perhaps five to ten years down the line, uh, if we can build a system which is completely headless, where you just talk kind of going back to Iron Man, right?
• 1:04:38
  SPEAKER_01: Where you just talk to the walls and they answer you and they get stuff for you.
• 1:04:41
  SPEAKER_01: That's the ideal thing.
• 1:04:43
  SPEAKER_01: And they and then what take taking it one step further, actually fixing issues for you.
• 1:04:48
  SPEAKER_01: Um, so we are going towards that trend, but it's gonna be meeting the world where it is and taking it's like ship of thesis, replacing one brick at a time.
• 1:04:57
  SPEAKER_00: And uh you know, last of all, uh big congratulations uh on the exit, uh right.
• 1:05:05
  SPEAKER_00: Our monumental life journey, you know, and wirestone, the life of a founder, right?
• 1:05:10
  SPEAKER_00: Uh and getting acquired by let's say a Fortune 50 company uh like Cisco, which has been a leader for the last many decades.
• 1:05:18
  SPEAKER_00: Uh right.
• 1:05:19
  SPEAKER_00: So, how did the the marriage with Cisco happen?
• 1:05:22
  SPEAKER_00: Can you share the process?
• 1:05:23
  SPEAKER_00: Whatever is shareable.
• 1:05:24
  SPEAKER_01: Absolutely.
• 1:05:25
  SPEAKER_01: Um so uh the uh you know the work with Cisco that we were doing even before uh we got acquired by Cisco was super interesting.
• 1:05:36
  SPEAKER_01: Uh so we really met them much before today.
• 1:05:41
  SPEAKER_01: And uh the the relationship kind of evolved into that it really makes sense for for them to acquire us.
• 1:05:48
  SPEAKER_01: Uh we started maybe a year and a half ago working on uh certain interesting sort of future facing problems on how do you define agent to agent interactions and you know in the era.
• 1:06:00
  SPEAKER_01: Of when A2A and these protocols.
• 1:06:03
  SPEAKER_01: Well, MCP is one stab at solving a very specific problem of agent interactions, which is tools selecting the right tools and uh sort of automating that process.
• 1:06:14
  SPEAKER_01: There's a lot more to be solved.
• 1:06:15
  SPEAKER_02: Yeah.
• 1:06:16
  SPEAKER_01: It's a drop in the ocean of the overall set of interactions.
• 1:06:19
  SPEAKER_01: So, anyway, there were some open source efforts, uh, specifically agency that we partnered with them on and did a bunch of other sort of similar things.
• 1:06:27
  SPEAKER_01: So, we knew the team.
• 1:06:29
  SPEAKER_01: Uh, Splunk, particularly, it was acquired by Cisco a couple of years ago, and they are an absolute beast.
• 1:06:38
  SPEAKER_01: They are kind of running the data systems of the world.
• 1:06:41
  SPEAKER_01: They have thousands and thousands of enterprise customers who use them as a telemetry logging system.
• 1:06:48
  SPEAKER_01: But also over the years, they've built out this amazing observability stack, which is kind of silicon up.
• 1:06:54
  SPEAKER_01: They have observability at each layer.
• 1:06:56
  SPEAKER_01: Uh no, Cisco is traditionally known for uh networking and security, but they have really honed in on application observability in the last decade.
• 1:07:07
  SPEAKER_01: And the gap was agents.
• 1:07:09
  SPEAKER_01: So now, you know, traditional applications will move.
• 1:07:12
  SPEAKER_01: I said, right, software is just tokens now.
• 1:07:14
  SPEAKER_01: So everything is just gonna change very quickly.
• 1:07:17
  SPEAKER_01: So that kind of shows you the sense of urgency.
• 1:07:20
  SPEAKER_01: That's why we are such a clean fit into you know Splunk's uh overall stack.
• 1:07:26
  SPEAKER_01: But not just that, uh Cisco is one of the fewest companies in the world, which is treating observability and security as two sides of the same coin.
• 1:07:35
  SPEAKER_01: They have products such as AI Defense, which is Solving Core Security, they have network security uh companies, acquisitions that they've made, like Thousand Eyes in the past, App Dynamics, uh of course Splunk.
• 1:07:47
  SPEAKER_01: Uh so they've filled out a lot of these gaps that they they had and they've built this beautiful stack on top of their federated data metrics.
• 1:07:56
  SPEAKER_01: Um, so we are sort of their agentic observability and security uh play.
• 1:08:04
  SPEAKER_01: So that's how it kind of happened.
• 1:08:06
  SPEAKER_01: We uh I feel like looking at the five-year journey, going back to the first day, I mean, I can't even recognize the company that it is because the whole world has changed four times over in the last five, six years.
• 1:08:19
  SPEAKER_01: Uh, so firstly, thank you for the wishes.
• 1:08:22
  SPEAKER_01: It certainly means a lot as a founder to see the company succeed and seeing all the employees, all the work that our engineers, our product teams, our GTM teams, all the work they've done, you know, come to fruition.
• 1:08:37
  SPEAKER_01: But it's really not the end of the Galileo story.
• 1:08:40
  SPEAKER_01: We will continue the Galileo story just within Cisco, except now we have the rocket fuel of unimaginable proportions.
• 1:08:52
  SPEAKER_01: So I'm personally super excited.
• 1:08:55
  SPEAKER_01: A company which was literally started in my garage with a little table and there was a heater because the garage was too cold, going from there to hopefully the next five years.
• 1:09:04
  SPEAKER_01: Yeah, Cisco serves 80% of the world's internet.
• 1:09:08
  SPEAKER_01: And they it's just unfathomable the scale at which they operate.
• 1:09:12
  SPEAKER_01: Uh, they're a phenomenal company, they're the top 25 companies in the world.
• 1:09:17
  SPEAKER_01: And to see Galileo be the agentic fabric for any application that's on the Cisco stack, I mean it's a dream come true.
• 1:09:24
  SPEAKER_00: Yeah, yeah, absolutely, absolutely.
• 1:09:25
  SPEAKER_00: Congratulations again, Athena.
• 1:09:27
  SPEAKER_00: Thank you.
• 1:09:27
  SPEAKER_00: And thank you so much again for the podcast.
• 1:09:29
  SPEAKER_00: Thank you for having me.
• 1:09:30
  SPEAKER_00: This was such a pleasure.
• 1:09:31
  SPEAKER_00: I think this was one of the most beautiful conversations uh that I ever had.
• 1:09:36
  SPEAKER_02: Thank you.