Latest / CXOTalk / Advice on Agentic AI: A Top Snowflake Exec Explains | CXOTalk #903
Transcript
- 0:00Everyone's talking about a gentic AI, but most enterprises
- 0:05are ready. I'm Michael Kraigsman and today
- 0:08on CFO talk #903. Christian.
- 0:12Kleinerman, Snowflake's EVP of product, shares hard truths on
- 0:17what it really takes data readiness, governance, AI,
- 0:22economics, and workforce impact. We're the company delivering the
- 0:27AI Data cloud. And what does that mean?
- 0:31The air data cloud, we have two lower stars with two important
- 0:36directions. 1 is we want to help organizations with the entire
- 0:41life cycle of data from the moment data is born or created,
- 0:46whether it's a sensor, whether it's an application or a device,
- 0:50all the way through transformation and enrichment
- 0:54and most important, through analysis and consumption of the
- 0:57data. How do you get true insights?
- 0:59A lot of that data. And at Snowflake, my role, I, I
- 1:02run the product team, a number of functions, product
- 1:04management, product design, technical writing, data science,
- 1:09variety of capabilities. I've been with the company 8
- 1:12years, very happy, very proud of what we do.
- 1:15Christian, we're talking about AI agents today.
- 1:20What can enterprise grade AI agents reliably accomplish?
- 1:28Where AI truly shines and truly has potential in the enterprise
- 1:34is when you make the context of that enterprise available to
- 1:39those state-of-the-art models where now I can go to a chat bot
- 1:44and ask a question and it's not generic knowledge from the
- 1:48Internet. It is based on knowledge from my
- 1:50company, maybe my customers, maybe my products.
- 1:54That is what where I would say the most interesting use cases
- 1:57are coming up today. I will classify them on, on, on
- 2:012 categories. 1 is read type of use cases where I just want to
- 2:05retrieve knowledge. My organization has a lot of
- 2:08documents, a lot of institutional information that
- 2:11has been hard to access until now.
- 2:14AI shy into being able to retrieve the right information.
- 2:17So this is where you see internal assistance, customer
- 2:22support or customer experience assistance.
- 2:24All of those we are just trying to retrieve information in a
- 2:27helpful manner. The other set of use cases,
- 2:30which I think is newer, but they're also happening and
- 2:34they're wonderful use cases, is when these agents can start to
- 2:38take action. They can start to call an API,
- 2:42maybe close a service support ticket, maybe draft and send a
- 2:46response to a customer, maybe build a demo.
- 2:50There's a number of possibilities, but the the most
- 2:52interesting thing is the state of the technology, AI and models
- 2:58I think is far ahead from the types of applications and
- 3:02potential that we have in front of us.
- 3:04Now let's take a moment to learn.
- 3:06About Emeritus, which is making CXO talk possible If you're a
- 3:10business leader navigating change or driving growth,
- 3:14explore executive education programs with Emeritus, a global
- 3:20leader, They offer programs developed in collaboration with
- 3:24top universities designed for decision makers.
- 3:27Like you, there's a. Program tailored to your goals,
- 3:30whether that's AI and digital transformation or strategy and
- 3:35leadership. Find your program at
- 3:37www.emeritus dot org. That's a really interesting
- 3:43point. So you say that the state of the
- 3:45technology is in advance of what we in the enterprise can absorb.
- 3:52Can you elaborate on that point? That's a key issue.
- 3:56These models, AI models can do so much as a function of the
- 4:02context that you provide to them.
- 4:04So the more information I, I make available, I want to, I
- 4:07want to feed the information about our customers.
- 4:08For example, now they can start to reason about this is what
- 4:13happened with a customer that's more traditional descriptive,
- 4:17backward looking type of analysis.
- 4:19But then there's so many more things that I could start
- 4:21thinking of doing in terms of my engagement with my customer.
- 4:25What should I be doing, what others in the industry are
- 4:28doing, ask interesting questions and reason in ways that truly
- 4:33have been unthinkable in many ways for for us today that these
- 4:38models are ready to do so. I I do have a high conviction
- 4:42that the state of application and use cases of these models
- 4:46are still being explored in many areas, are still being pushed
- 4:50forward ahead of additional developments and on the models.
- 4:55And what are the limitations that are driving this this gap?
- 4:59Most of the scenarios where we see projects going wrong or or
- 5:05projects getting shut down, they run through through two types of
- 5:09issues. Issue #1 is correctness of
- 5:12results or correctness of actions.
- 5:15The promise of AI is amazing and to the extent that you can trust
- 5:23the results from that AI, everything's wonderful.
- 5:26But we have heard scenarios of a company rolled out an assistant
- 5:31and internal customer service reps are starting to give wrong
- 5:34answers. That is unacceptable.
- 5:37So I think that that plays an important role.
- 5:40The other aspect is access to the right amount of data or the
- 5:46right context. And when I say right, I mean in
- 5:50terms of security and privacy. We have heard about
- 5:53organizations that try to roll out their own solution in the
- 5:56last, I don't know, 6-12 months and all of a sudden they were
- 6:01giving information to people that wouldn't have had access to
- 6:05that information. So that leads into a security
- 6:08and privacy concern. I would say to the extent that
- 6:12there are solutions that deliver correctness of results and
- 6:17enforces existing governance and security policies, that's when
- 6:21AI shines. And and of course that's a lot
- 6:23of what we have been working for the last 1824 months at Soviet.
- 6:28What is the driver of this gap between the technology
- 6:34capability and the enterprise ability to make use of it?
- 6:39I think it comes down to in many areas, organizations are not
- 6:44ready to provide that context, provide that data in a
- 6:49structured and organized fashion to the AI models.
- 6:53The, the canonical conversation I have with, with many of, of
- 6:57our customers and our organizations is when you ask a
- 7:00question, tell me something about customer XYZ, do you know
- 7:06where to go and answer and, and get information about that,
- 7:09that, that customer in your organization?
- 7:11Because if data is siloed in all sorts of places and nobody
- 7:15really knows what's where, it's going to be a lot more
- 7:18challenging for any AI model to be able to understand that.
- 7:23Whereas what we see, at least the technologies that we've done
- 7:27with Snowflake and other players in the industry are doing
- 7:30similar, is you end up curating what we call the semantic view,
- 7:34which is a translator between business terms and where the
- 7:40answer to certain business terms or certain business questions
- 7:44may lie. That is the type of data
- 7:48governance and data organization and data quality that is
- 7:53prerequisite for organizations to be able to tap into those
- 7:56results from AI. Is this gap of organizations
- 8:01being able to adopt the full capability primarily the result
- 8:08of or insufficient data, or is it something else?
- 8:14For the scenarios that are data-driven, the answer is
- 8:21largely yes. The the most common conversation
- 8:24we have with with organizations these days are how do I
- 8:27accelerate my data quality efforts, data rationalization
- 8:32efforts so that I can have my data be AI ready and, and that's
- 8:38a combination of things. Maybe it's simply a matter of
- 8:41eliminating boundaries or silos between data sets.
- 8:45Sometimes it's as simple as saying I don't have canonical
- 8:49sources of truth for data. If I have five different
- 8:52versions of what my customer list is, it's difficult for the
- 8:55AI to figure out which one is your, your official answer.
- 8:58And, and then the the last one is security.
- 9:02Do I have clear policies on who can access what and how do you
- 9:05get AI to, to enable that? So I would say that that is
- 9:09true. The data governance, data
- 9:10quality is an important prerequisite for leveraging
- 9:16data-driven solutions in, in, in, in the enterprise.
- 9:19There are other use cases where we have not talked about the
- 9:24about them, things like coding assistance that help with
- 9:28enterprise application development and other things.
- 9:30But at least for the majority of the use cases that we spend most
- 9:35of our days, which is enabling organizations to get value out
- 9:38of the data. Yes, there.
- 9:39The short answer is it's about data quality and data
- 9:42governance. What are some of the very
- 9:44practical use cases that you see?
- 9:49I'll do in reverse order since I just mentioned coding assistance
- 9:53and I am completely fascinated by the types of use cases that
- 10:01code generation can provide for any of us these days.
- 10:06Some of you may be thinking, well the coding assistant is
- 10:08only for developers and engineers, but we're seeing all
- 10:13sorts of functions of use cases being able to leverage coding
- 10:18assistance. I'll tell you something that is
- 10:20super exciting to me. Many of our solution engineers
- 10:25that the folks that are like technical at Snowflake that are
- 10:28trying to engage with customers, they can build customized demos
- 10:33for every single conversation that each one of our customers
- 10:37or put the prospective customers are interested in.
- 10:41This is sort of unheard of. The the I've been doing
- 10:44enterprise and and databases for for 25 years and for the longest
- 10:48time you had a canonical one demo, often times something
- 10:53about a bicycle shop or something like that.
- 10:55And no matter who the audience is, you showed up with your
- 10:58bicycle shop them. But that doesn't have to be that
- 11:02way. It's very much easier for me to
- 11:03say, dear coding assistant, I am about to meet with a Consumer
- 11:09Packaged Goods company. These are the priorities for
- 11:12them based on maybe the last four earnings results or
- 11:16earnings reports and how do we show the value of the technology
- 11:21in that context and then the coding assistant can go and
- 11:23generate an example, a demo or a sample that can be used.
- 11:27Now let's quickly hear from. Emeritus, which is making CXO.
- 11:31Talk possible. If you're a business leader
- 11:34navigating change or driving growth, explore executive
- 11:39education programs with Emeritus, a global leader.
- 11:43They offer programs developed in collaboration with top
- 11:47universities, designed for decision makers like you.
- 11:50There's a program tailored to your goals, whether that's AI
- 11:54and digital transformation or strategy and leadership.
- 11:58Find your program at www.emeritus.org.
- 12:04Coding assistance are generating value for functions way beyond
- 12:10what I would call traditional application developers or
- 12:13software engineers. I'll give you one other example
- 12:15on, on, on that line of thinking, pro product managers
- 12:19in my organization now they're not just waiting for, for
- 12:23engineering teams to go and create products or, or early
- 12:27moths or functional prototypes. They're they're doing it in
- 12:31themselves because it's fairly easy for pretty much anyone
- 12:35organization to guide a coding agent into building solutions.
- 12:40So that's on the, on the coding side.
- 12:41And the the other set of use cases that we're completely
- 12:43excited about is it just unlocking the value of data.
- 12:47The trend that we have seen in the last maybe 20 years is data
- 12:54democratization is a trend. Is it something that we want
- 12:57more and more people in a company to be able to get access
- 13:01to what business intelligence did 20 years ago, which was data
- 13:06is not a problem of IT. Data is an opportunity for
- 13:10everyone to inform the day-to-day decisions.
- 13:13I think AI is materially dramatically expanding the set
- 13:18of use cases we we've introduced at Snowflake, something we would
- 13:21call Snowflake Intelligence, which we think of it as the next
- 13:25level of unlock on companies be able to do more with their data.
- 13:29We have a question from Twitter from Arsalan Khan, and he says,
- 13:38right, His question is provocative, but I'll ask it
- 13:41anyway, he says. Will the AI bubble explode when
- 13:45organizations realize the best they can do with AI is chat
- 13:50bots? We're talking about the under
- 13:52utilization of the capabilities really.
- 13:55We already see use cases that go far and beyond chat bots.
- 14:01They the notion of chat bots being AI comes from that first
- 14:06category of use cases that I described, which was the more
- 14:09read only capability. I just want to retrieve data and
- 14:12I think there's plenty of opportunity there.
- 14:16But now that we see agents be able to take action, I think
- 14:22that's very, very different from a chat box.
- 14:24Yes, in many instances the interface is conversational in
- 14:28nature, but that doesn't mean it is a chat.
- 14:32But if, if, if anyone that is listening has used any of the
- 14:37coding assistance cloud code or cursor, we introduce also Cortex
- 14:42code at Snowflake. You quickly appreciate that this
- 14:46is so much more than a chat, but it's not I ask a question and I
- 14:49get an answer is more. I give a command and maybe an
- 14:53hour later, sometimes 4 hours later, I get solutions answers
- 14:58to the the the problem or or the problem that I specified that
- 15:02are typically tasks that would take humans days or or weeks.
- 15:09So I would say now that agents can autonomously take action,
- 15:14follow up on items, reason what we see is is effectively an
- 15:21augmentation of, of human capability that I I think is
- 15:23unprecedented from, from our times.
- 15:26I, I was chatting with an organization a few weeks ago and
- 15:31they have a data team. They have a number of analysts
- 15:33traditional, but they also augmented it with agents AI
- 15:38agent. Actually, they have a name for,
- 15:40for one of the agents and they think of it of, of the agent
- 15:44that's a member of the team. And when a request goes out that
- 15:47goes to the, the, the, the channel in Slack that every
- 15:52member of the team gets to answer questions.
- 15:54The agent is part of that. And believe it or not, it has a
- 15:58lot of productivity, is very able to, to help with others.
- 16:02We do the same thing at Snowflake.
- 16:04We would have a number of integrations with the tools that
- 16:07our employees use on a regular basis.
- 16:10And it goes materially beyond the chat bot.
- 16:13So I, I do think that the, the potential is way higher than
- 16:17many of us see today. I think on a regular basis we
- 16:21see that aha happening from, from individuals, which is a
- 16:26yeah, it's cute. I can ask questions, I can get
- 16:28answers and half of the time are right or wrong.
- 16:31We're we're moving quickly past that.
- 16:33And once we see AI based technologies, agents taking
- 16:38decisions, making actions, making changes in our behalf is
- 16:42truly exciting. Christian, there has been so
- 16:45much hype out there. How can business leaders
- 16:51evaluate vendor claims about autonomy, about reasoning, about
- 16:57elaborate workflow capabilities? How can business leaders look to
- 17:01see what what is real when it comes to AI agents right now?
- 17:06If we all pay attention to a lot of the claims are there, it is
- 17:11dizzying because many of them sound similar, overlapping.
- 17:16It's complex. My recommendation to everyone is
- 17:20go and try the technology. The beauty of this era of AI and
- 17:25and and agents is most products are a couple of prompts away
- 17:31from someone trying it. I think instead of saying I am
- 17:34going to run a 3 month proof of consent on AI solutions yada
- 17:40yada yada, now is a how about we spend an hour with a product and
- 17:45we'll see how good or not they are.
- 17:48We're doing this on a regular basis, just evaluating new
- 17:51models, new products, new applications of the models and
- 17:55quickly you can see whether what they're doing is materially
- 17:59better than the state-of-the-art versus just another quick
- 18:06wrapper on top of the, the, the AI model.
- 18:08So my, my recommendation, because it's hard to, to tease
- 18:11apart the nose is try it. We're in a world where anyone
- 18:15that speaks English can largely use most of the products based
- 18:19on AI. So that's the path forward, my
- 18:21opinion. So basically.
- 18:22You're saying you've you've got to do the empirical testing for
- 18:28yourself. You can't.
- 18:29You can't rely on vendor marketing in order to determine
- 18:33what's real and what will work in your particular environment
- 18:36and use case. Correct.
- 18:38And. And that has been true for
- 18:39certain aspects of enterprise technology for many years.
- 18:44It's only that the cost of the POC is going down materially.
- 18:47I can. I've been doing this long enough
- 18:50that I remember when if I wanted to do apoc for an appliance
- 18:55database appliance you would get on a sign up sheet and they will
- 18:59tell you well your turn to get the appliance shipped to you is
- 19:028 months from now and then you get it for a week.
- 19:06That sounds completely crazy in this day and age.
- 19:09What what's happening with their products is just go and and try
- 19:13it. Usually it's on the spot and an
- 19:16hour later you have a very good sense of what what it is.
- 19:19And then the other thing that has happened traditionally that
- 19:22I don't think it works that well for AI is benchmarks used to try
- 19:27to simulate or model customer use cases.
- 19:31And then you could say, well, this vendor is better on this
- 19:34merit benchmark. So maybe I'll go without a POCI
- 19:37think. Even though there are plenty of
- 19:39benchmarks that are constantly evolving and and vendors
- 19:43leapfrogging one another, I think nothing substitutes
- 19:47leveraging or trying a product for the needs of your
- 19:50organization. You raise a very.
- 19:52Interesting point about the experimentation is that
- 19:57something that is fundamentally different about AI in the
- 20:03enterprise that it lends itself to trying this and trying that
- 20:09and iterating extremely rapidly 100.
- 20:12Percent. In an ideal world that would
- 20:15have been true of all enterprise technologies in the past and SAS
- 20:20applications in the past. But it's harder.
- 20:24It's expensive to go through a quick evaluation of a SAS app or
- 20:29a quick evaluation of invention appliances as a use case.
- 20:32But traditional database in knowledge.
- 20:34What's unique about AI is that in many use cases, the time to
- 20:40create and test a pilot is always semantically lower than
- 20:45some of the other technologies I just mentioned, which opens this
- 20:48door to say, I don't need to go and just read the, the, the
- 20:52vendor claims and decide based on that.
- 20:54I can just try it. And then we hear on a regular
- 20:57basis, I, we, we did this snowflake world tour where we
- 21:00take our, our message to a number of cities, 23 cities.
- 21:05And I heard directly customers like I just did a a test of your
- 21:09technology with two other vendors and this is what we
- 21:11like, which is amazing like that, that that was the the
- 21:15friction used to be so high that that didn't happen.
- 21:17Now it's. Materially.
- 21:20Lower. I'm not going to downplay that
- 21:22the cost is 0, but it's so low that is not unthinkable to try
- 21:283-4 technologies in the period of a day or two.
- 21:31So there's a real. Mindset shift that must take
- 21:35place for people. Managing and leading.
- 21:39AI deployments and when it comes to agents, how how does this
- 21:46approach overlay that the the agent world agents are?
- 21:52The the the best manifestation of AI, which is is the AI
- 21:57models, but they have the ability to reason through some
- 22:01context and be able to invoke some tools or take some action.
- 22:05So I would say at this point, agents is the, the best
- 22:09instantiation of, of what I would say the, the, the AI
- 22:12models and it, it has materially changed the effort needed to go
- 22:19and try one of the solutions. There's also a number of
- 22:23interoperability protocols emerging.
- 22:26MCP is, is gaining a lot of traction.
- 22:31I think no customer conversation I have these days goes without
- 22:34saying. How do you plug into an MCP
- 22:36ecosystem and can I be part of my broader agent strategy?
- 22:41Same things happening with agent to agent protocol.
- 22:44And I think more and more interruptible is going to is
- 22:47going to emerge. But the the notion of companies
- 22:52should much easier Again, I want to be careful then I'll say 0
- 22:56cost, but much easier try this technologies is is is changing
- 23:00how what changing what's possible and I would encourage
- 23:04the companies to change how they think about evaluating
- 23:07technology are there. Limitations of agents today
- 23:12based on the technology, limitations of data and so
- 23:16forth, where the where the boundaries of where we go into
- 23:20the bleeding edge that becomes maybe dangerous.
- 23:24We don't know the. Bounds and then that was a
- 23:26little bit of my comment earlier that the, the state of the
- 23:29technology may be ahead of our applications where where most
- 23:34companies take a, a, a, a slower, more guarded approach is
- 23:41in regard to to the two issues I I was referencing, which is 1
- 23:45correctness of results and security.
- 23:50Privacy compliance guarantees correctness of results does not
- 23:54just apply to did did the agent or the chatbot give me an
- 24:01incorrect answer? The more we talk about agents
- 24:04taking action, it is important that that action is also
- 24:07correct. Like probably one of the last
- 24:12operations that will, that will be fully automated with AI is
- 24:16going to be things like a bank transfer or a wire transfer
- 24:20because you, you want to make sure that those things are 100%
- 24:23correct. But in the meantime, there are
- 24:26all sorts of use cases in, in the consumer world, all the
- 24:29examples are take a reservation to a restaurant.
- 24:33But in the enterprise world is it's, it's, it's simpler.
- 24:35It's like close a support ticket or generate a, a set of
- 24:40questions for this review that I'm about to go to or generate a
- 24:44set of insights for maybe a board meeting.
- 24:48And not not that the stakes are not high, but they're lower than
- 24:52some of the more mission critical use cases.
- 24:55Can you? Talk about the data and
- 24:59governance gaps that can cause problems with agent deployments.
- 25:04You touched on this earlier, but really drill into the data for
- 25:08us. This is in my mind, the single
- 25:11biggest impediment for companies to tap into AI today.
- 25:19And and it comes down to if the data, the states of a company is
- 25:27not well organized or is not broadly accessible, there is
- 25:33little magic that the AI can do. Like I, I, I, I like to say AI
- 25:38is a, is a turbocharged, super capable technology, but it's not
- 25:44magic. If you have two data sets, say
- 25:49with customer lists and they're inconsistent with one another, a
- 25:54great AI solution may tell you, well, I found two lists and
- 25:58according to this list, Michael is a customer.
- 26:01According to this list, Michael is not a customer.
- 26:03They cannot make a judgement call on that.
- 26:05So I would say cleaning up data, having a single version of the
- 26:11truth, which is sort of the, the Holy Grail in enterprise data
- 26:15systems is like for any question, there's only, there
- 26:18should be only one answer, or at least factual.
- 26:20Having that type of rigor matters a lot because all, all
- 26:25of the AI systems are doing is reasoning and retrieving based
- 26:29on the data. But if the data is messy, nobody
- 26:33can help you. I, I, I usually use a limo
- 26:35status. Like if you were to ask the best
- 26:38analyst in your company to answer a question, would this
- 26:40best analyst know where to go, know what to do?
- 26:45Then there's a path for for AI. If the best analyst says no, I
- 26:48just need to go have lots of meetings because all the
- 26:50knowledge is in people's heads. That's going to make it harder
- 26:53for AI to do it. There are promising efforts on
- 26:57AI trying to infer some of that knowledge from the data, but
- 27:01that starts to add additional levels of non determinism and
- 27:05potential risk. We have a.
- 27:07Question from Arsalan Khan on Twitter right now very
- 27:11specifically about the data that is institutional knowledge
- 27:17residing in people's heads. And there's a number of
- 27:21implications of that, including our salon raises.
- 27:27Will people feel we don't need them if they are, if they share
- 27:31that information in? Most technology changes some
- 27:36resistance like that comes and I think the, the, the, the reality
- 27:41is what a lot of this technology can do is enhance productivity.
- 27:46Yes, maybe rely less on tribal knowledge and what's on people's
- 27:49head, but it also frees up individuals to go and work on
- 27:54additional tasks, higher value tasks.
- 27:57I forget who said it, someone said it publicly and now I
- 27:59repeat it almost on every instance.
- 28:01I, I, I get a chance, which is I don't think individuals should
- 28:04be worried about being replaced by AII think they should be
- 28:08worried about replaced by individuals that know how to use
- 28:11AI is because that that's the, the, the, the more real thing.
- 28:15And yes, relying on on on institutional knowledge or
- 28:20tribal knowledge in my head, that's not good.
- 28:22Now I'm going to be be be called at all times of the day, which
- 28:26you can say it's good for independence and job stability,
- 28:28but that's not long term good for the company or for the
- 28:31individual. So how?
- 28:33Should organizations manage and govern all of this unstructured
- 28:39data that not even in people's heads, you know, documents,
- 28:43images, audio, how should organizations deal with that?
- 28:48There's. A broad trend in the industry
- 28:50and, and, and it's not like we're, we're leading much of
- 28:52this, which is how do we capture in what is called semantic
- 28:57models and semantic views and ontologies that business
- 29:03knowledge that often times is either embedded in tools or
- 29:06embedded in, in people's mind. But how do you codify it in a
- 29:10way they can be made available to the AI?
- 29:12So that is happening. We, we started a cross company
- 29:17in a consortium to do interoperability of semantic
- 29:22models. We're, we're very excited about
- 29:24number of companies participating in that.
- 29:26And it's all about how do you take that corporate knowledge
- 29:29and codify in a way that you can more intentionally make it
- 29:31available to, to AI. So I wanted to, to add that to,
- 29:34to the comment on knowledge in people's minds or embedding
- 29:38tools. Now let me shift to, to your,
- 29:41your question on unstructured data, which your premise was
- 29:45spot on. Unstructured data, you don't
- 29:47even have to know. Is it in someone's head or
- 29:50something like that? No, the reality with
- 29:52unstructured data a is the vast majority of data that an
- 29:55organization has their estimates that put it at 90 percent 8595
- 30:00that the specific doesn't matter.
- 30:01But I think most of us would agree vast majority is just on
- 30:07and on documents and logs and emails and all of that contact
- 30:13context, which is truly difficult to retrieve or has
- 30:18been truly difficult to retrieve up until AI.
- 30:23In the world of AI, there's still some amount of OK, you
- 30:27need to go give it the right index.
- 30:28You need to figure out if you're going to do this type of
- 30:32semantic search, approximate how, how important this
- 30:36correctness and citations. But what AI has done for a
- 30:41structured data is as if overnight it turned on the
- 30:44light. Like imagine that we had all
- 30:47this data around us, but we if, if it's dark in the room, you
- 30:51don't even know that it is there.
- 30:53Now you open your eyes and like, oh, there's so much for us to
- 30:57learn as a company. It's incredibly exciting what
- 31:01what AI is doing for unstructured data we have.
- 31:04Really an important question from Sai Panamuru from on
- 31:09LinkedIn who says why are so many projects getting stuck at
- 31:14the proof of concept stage in my mind?
- 31:17A lot of what we saw maybe 12-18 months ago, there was this
- 31:21general sense that AI is so powerful that it's easy for me
- 31:26to just go and roll up my sleeves and build the solutions.
- 31:30I'm going to build my customer support agent.
- 31:32I'm going to build a talk to my database agent.
- 31:36It's not too hard and many of those initial trials produce
- 31:43great results. Like in isolation.
- 31:46I know exactly the question. I'm working with maybe 5 tables
- 31:49in my database and everything works great.
- 31:52And what we've seen I, I would say for sure last year plus is
- 31:58that by the time you try to roll those PO CS or those organic
- 32:02efforts into production, then you get the intersection with
- 32:05reality. And, and reality is harsh.
- 32:08Reality is like, Oh yeah, you tested it with five tables, but
- 32:11we have 100,000 tables and they're not clearly named in, in
- 32:16the, in the, in the POC. They were called orders and line
- 32:18items and customer in, in in the production system is something
- 32:22like table one, Table 2, Table 3.
- 32:24And it's impossible to know what's in there.
- 32:27So, so the, the, the reality operates have a different scale.
- 32:30It's messier back to this data quality concept.
- 32:34It matters on how organized and how discoverable and
- 32:37understandable data is. And the other piece that that I
- 32:40mentioned is correctness and trustworthiness of solutions has
- 32:45stopped many pilots from going to production and it has shut
- 32:49down projects that went into production just because again,
- 32:54they, the POC sounded great, but the real system is providing
- 32:58wrong answers. I don't know, one out of four
- 33:00times, which you can say, well, it's not too bad, but like one
- 33:03out of four times if you're serving, I don't know, 1000
- 33:05customers in a day, that sounds a lot of unhappy customers.
- 33:08I think that's the friction on, on do it yourself type of
- 33:13solutions, which is why ourselves and that's Novic
- 33:17analysis in the industry have seen a lot of momentum and
- 33:20uptake with our solutions. Like I mentioned slowly
- 33:23intelligence because it's harder than it seems.
- 33:25And now customers are ready to say I want to put in production.
- 33:28I don't want to have to roll up my sleeves this.
- 33:31Issue of the leverage of agentic AI also means that when you have
- 33:42errors, there's a magnification of those errors that can grow.
- 33:47And we have a question from Elizabeth Shaw on Twitter, who
- 33:52says models and algorithms have a tendency to drift or become
- 33:58polluted and with a gentic AI that can cause havoc.
- 34:05How should companies address this set of issues?
- 34:08And who should be responsible for this?
- 34:11Inside organizations? Some of these?
- 34:14Trends and problems were there and were visible in traditional
- 34:19machine learning technologies like what when when someone was
- 34:22creating a recommendation engine with a again, predictive ML, the
- 34:28the discipline emerge to how do you do evaluations and how do
- 34:32you monitor for drift? And how do you make sure that
- 34:35you're not over fitting a model and, and some of those
- 34:39techniques are effectively, how do you govern, monitor and
- 34:44maintain non deterministic technologies?
- 34:48Like back in the day when, when everything was 100%
- 34:52deterministic and your balance is above 0 or not?
- 34:55That, that's, that's easy that that will not drift.
- 34:57But as we've got into more sophisticated solutions and ML
- 35:01was the beginning of that and a lot of best practices have
- 35:04emerged. And how do you govern ML into
- 35:06production? Now they're in OverDrive with
- 35:09AI, which by definition, even with all the same inputs, is
- 35:13going to give you a variable outputs.
- 35:17And I think the same set of patterns are emerging to govern
- 35:21AI solutions in production as Snowflake.
- 35:25We've we've enabled our customers to do logging of
- 35:28responses, track evaluations, be able to see that what was
- 35:33intended to happen is delivering the answers that customers would
- 35:38need to deliver. So I would say there's president
- 35:40with machine learning, we need to go and do a lot more as an
- 35:43industry to help organizations cope and put into production
- 35:48with confidence solutions that are inherently very varying and
- 35:52adapting to context. So.
- 35:54On this point of the nature of the technology being non
- 36:00deterministic, how should business and technology leaders
- 36:05measure value, measure return when the cost, the consumption
- 36:09patterns, the productivity gains remain uncertain.
- 36:15It's not the same as putting in an ERP system where you.
- 36:19Know you you have your. Defined order flow, for example
- 36:23in. All instances the benchmark or
- 36:27the evaluation has to be a function of what am I getting in
- 36:30return for what I'm putting in. I'll I'll I'll I'll give you an
- 36:34example. Some consumer level mobile app
- 36:39was trying to make some LLM calls in a critical path of the
- 36:44application just to show a slightly better recommendation
- 36:50that's expensive because the uplift in the quality of the
- 36:54recommendations didn't come anywhere close to the cost of
- 36:59the AI. So I would say that this is no
- 37:02different predictive and unpredictable technologies like
- 37:05what do you put it in and is the delta greater or not than the
- 37:10benefit that you get? So, so that one is, is, is more
- 37:13the exceptional case in many instances.
- 37:16What we're hearing is it's productivity gains for employees
- 37:21and organization. Often times measuring orders of
- 37:25magnetism, a use case that maybe an analyst can take, I'm going
- 37:32to make it up five hours can be done by an agent in 5 minutes.
- 37:38The the ROI in there is so large, there is plenty of room
- 37:43for AI companies to make money, including the all, all the
- 37:48platforms in between. And still the company gets a
- 37:52positive ROI on that. So it's not to say, yeah, let's
- 37:55let's charge whatever for, for AI and, and have companies pay
- 37:59as much. But depending on what it is
- 38:01benchmarked against, in many instances, the, the ROI is so
- 38:05large that there's actually a lot of reason to be optimistic
- 38:09about this. And, and back to the, the, the
- 38:12previous question, the premise of it's a bubble.
- 38:14No, I, I, I think there's a lot of real productivity in here.
- 38:17It's an. Extraordinary comment that you
- 38:20just made that the, the ROI is so large potentially and I think
- 38:27that gets right to the heart of all of this vast amount of
- 38:32investment that's being made in AI right now.
- 38:36But we have a a really interesting question on Twitter
- 38:41from Miles Sewer, and I'm Christian.
- 38:44I'm going to ask you to answer the next questions pretty
- 38:47quickly because we're simply going to run out of time.
- 38:49Miles Sewer says this research shows that much data is
- 38:57immature, unwrangled, and without a semantic layer.
- 39:02It's a market limiter for every software company.
- 39:05And what is Snowflake working on to help with this?
- 39:10Data quality and and and data claiming is get in the way of
- 39:13companies leveraging AI. We have launched capabilities to
- 39:17help extract semantic information from a number of
- 39:20sources, whether it's existing BI tools or query logs.
- 39:24So we're helping organizations extract that knowledge.
- 39:28And also we're partnering with a healthy number of third parties,
- 39:31companies, start-ups that are working on inferring some of the
- 39:35semantics from data. So it's a work in progress for
- 39:38the industry, but it's very exciting what's going on very,
- 39:40very quickly. Sai Panamuru comes back and from
- 39:44his earlier question and he says are there key solutions you can
- 39:48recommend to or advice for moving from moving AI from proof
- 39:54of concept POC to production and very quickly please it's too.
- 39:59Generic of a of a question because depends on what AI and
- 40:02what solution. But if you wanted to do talk to
- 40:05your structure data, I'll plug in price of intelligence we
- 40:10have. A question from Arsalan Khan on
- 40:14Twitter. Folks, keep your questions
- 40:16coming in. We have a few minutes left.
- 40:17We'll try to get through these really fast.
- 40:20Arsalan Khan now wants to shift the conversation to the
- 40:23workplace implications, and he says that performance reviews
- 40:29are a mix of objective and subjective opinions.
- 40:34When AI gets involved, it's more objective.
- 40:38And should AI be used in hiring and firing?
- 40:42And what I'm really interested in is the impact on jobs from
- 40:48agentic AII would think of. AI for people related matters as
- 40:52an assistant, not a decision maker. 2 reasons.
- 40:57One is AI results are still as good as the inputs that you
- 41:01provide to the AI and there's subjectivity in curating and
- 41:05choosing those inputs. And the other thing is we, we
- 41:08also established that there is some variability in the
- 41:12responses that AI produces. So I would say do leverage it,
- 41:16maybe use it for comparison. We were, we were summarizing an
- 41:19interview feedback last night for someone, but it's summarize
- 41:22not replace it for a decision maker.
- 41:24Do you have? Much interest in cause of
- 41:27concern among your customers for the impact of a of a genetic AI
- 41:33on jobs and the workforce. Everyone has the opportunity to
- 41:36do more with their data, which should translate to faster
- 41:42business outcomes, Better Business success, and that's
- 41:45what, at the end of the day, gives prosperity.
- 41:47Jobs will evolve. Jobs are evolving already.
- 41:51But I, I, I, I, I'm on the optimist side that this is a
- 41:55productivity play, massive productivity play.
- 41:58But people will adapt. Jobs will adapt.
- 42:01Companies will do better if they leverage AI.
- 42:04What? Advice do you have for senior
- 42:06business and technology leaders on managing this period of rapid
- 42:11change and it's particularly in relation to agentic AI?
- 42:15Don't. Take on large scale AI roll outs
- 42:21When, when, when you hear we're going to buy 100,000 licenses of
- 42:25XYZ solution and put it out there and it's like, well, do
- 42:29you even know if it works or not?
- 42:31I, I think the better approach in the same way that I earlier
- 42:34said the cost of evaluating a technology is fairly low.
- 42:38I would say find smaller use cases, put them out in
- 42:41production. If it works well, scale it.
- 42:44That's the, the, the mode I think all of us should be in.
- 42:48Technology is advancing very fast.
- 42:49If you try something four months ago and it didn't work, there's
- 42:52a chance that it may be working today.
- 42:53So just stay in that constant state of evaluation trial, small
- 42:58rollouts and scale based on success.
- 43:01So constant. Iteration constant, trying new
- 43:04things because it's all changing.
- 43:07Take what works, put what works into production, and discard
- 43:12what doesn't work, basically. Correct.
- 43:14That's correct. All right, and.
- 43:16With that, we are out of time. A huge thank you to Christian
- 43:20Kleinerman, Snowflakes executive vice president of product.
- 43:24Christian, thank you so much for being with us and taking your
- 43:27time. I'm very grateful to you now
- 43:29like. Likewise, Michael, thank you so
- 43:31much for for having me and I look forward to doing it.
- 43:33I get some time, everybody who watched.
- 43:36Thank you for those great questions.
- 43:38But before you. Go.
- 43:41Subscribe to the CXO Talk newsletter so we can keep you up
- 43:44to date on our upcoming shows, everybody.
- 43:49Hope you have a great. New Year, our next show is in
- 43:53early January and we'll see you again next.
- 43:56Time. Take care everyone.