Latest / The Data Splash / What Data Should You Rely On? (AI + Real-Time Data) - with Amaury Desrosiers, Nimble
Transcript
- Omri: Welcome everyone to the Data Splash podcast brought to you by Upriver. If you're a data engineer, a data leader, or anyone trying to make sense of what is happening in data and AI right now, you're in the right place. Today with me is Amaury from Nimble. We will talk about which data organizations they are actually using and how AI is reshaping that mix, especially when it comes to using external and real-time data. Amaury is the director of solution consulting at Nimble, a platform at the front line of one of the biggest shifts we're seeing in data right now. how to make external web scale data ready for AI usage by companies and the agents they're building. So if there's anyone with a clear view of what is actually going on there, it's him. Glad to have you here, Amaury! Amaury Desrosiers: Thanks for having me. Happy to be here. Omri: So now before we dive into all of that deep stuff, we have a small tradition here in our podcast. We call it the 30 Second Splash. We're going to do quick fire questions and you have to answer them as quickly as possible without thinking. Let's jump in. Yes or no? Is AI a hype? Amaury Desrosiers: No, super exciting time now. Omri: What do you prefer, Snowflake or Databricks? Amaury Desrosiers: whatever our customer is using but Databricks is one of our investors so I have to say Databricks. Omri: Who do you think will win the upcoming football world cup? Amaury Desrosiers: France Allez les Bleus ! Omri: or DBT? Amaury Desrosiers: I'll let Claude code decide ⁓ the best, depending on the use case. ⁓ Omri: most underrated skill in data right now. Amaury Desrosiers: understanding the business case and how the business will leverage the data eventually. Omri: And that's a wrap for our splash. Thank you very much, Amaury These questions did go on the record. So now we can dive into the deep stuff. And if you could, just in two sentences, explain to us what Nimble does and what's your role in that company. Amaury Desrosiers: Sure, so Nimble is a real-time web data platform. So we deploy AI-powered web search agents that browse the live internet, verify what they find, and turn it into clean, structured data, your AI agents and analytics can actually use, right? So I lead the solution consulting team. So we translate basically what customers actually need into how the product can solve it and making sure it works in their environment before they actually buy it. Omri: And today when you're talking to these customers who want to bring in the data, what kind of data are they actually using and what do see them trying to answer? Amaury Desrosiers: It depends, but almost every conversation I have starts with questions that their internal data cannot answer. So a real-time customer wants to know potentially what their top 20 competitors are charging right now across thousands of SKUs and reprice before the day is out. A bank might want real-time due diligence on counterparty, right? News, filling, leadership change, pull this together in a couple of minutes instead of weeks. Or CPG brands want to track every mention of their product across reviews, social forum to catch up on quality issues before it actually become a recall, for instance. So different industries, totally different use cases, but it's the same challenge eventually. The answer is the need lives outside of the company. So on the open web, they need basically this data to solve their pain. Omri: And this is super interesting because I think if we look back at what things, what data used to mean for most organizations, it used to be everything they were able to store inside their data warehouse. And kind of sounds like that's not the case anymore. So do you actually see companies using and what is being taken from the external web data and what is taken from their internal warehouse today? Amaury Desrosiers: So... The mental model we use with customers is like three concentric circles, right? So you have the inner circle, which is a warehouse. So like your sales, your CRM, your product data, that's still the foundation, right? That doesn't go away. Then you have the middle circle, which is kind of the in-structure data that they have. So could be Slack tickets, know, Drive, Jira, transcript. then you have the outer layer, which is the by far is everything outside of the company. So the web, third-party data, public record, social signal, and so on. And the shifts that... the outlier used to be kind of a nice to have. Maybe you you bought a dead asset from a vendor twice a year and now it's like it's a first priority. AI application or consuming it like natively as part of the reasoning loop and people want this as quickly as possible and not you know something deprecated that's six months old. They want to understand what's going on on the web right now to make better business decisions. Omri: And do you still see internal data playing a critical role there? Or is that kind of taking a step back to make space for that external data that can now be consumed in real time? Amaury Desrosiers: So in internal data is still the grand truth, right? That's who you are. So your customer list, your product catalog, your pricing, your performance, your sales and so on. External data is what's happening around you. So you absolutely need both and the magic really happen when you join the two. So the best use case is not, let's pull data from... from the warehouse, so let's put it in data, it's more like, let's enrich single one of our sales data, for instance, with competitor promotion value ⁓ what timing, at specific retailers to understand, if my sales went down, ⁓ could be the root cause behind it, whether specific promotion happening by specific competitors, ⁓ basically understand the full context. Omri: super interesting. And again, I think that is critical to understand like the ground truth component of the data you have internally and then understanding how that relates to everything happening in the world around you and being able to merge the two things together. Now, when we look at external data, especially web data, and I remember that back from my engineering days, it used to be a nightmare, I think, for most organizations because you had to write a lot of connectors, then you always had data quality issues with them. and then you had to go and fix that, understand? And the internet changes on daily basis. how do you see that changing right now with what you're seeing at Nimble, what you guys are doing as well? Amaury Desrosiers: Yeah, so yeah, it used to be a nightmare for two critical reasons. First, connection. Like it was hard, right? So every website is pretty much unique, different layout, different anti-bot capabilities, different JavaScript framework. So building the scraper was potentially like a one month, one week project, but maintaining the scraper was like a four-hour project. You can ask any data engineer, they will tell you stories about waking up at three in the morning because a major retailer redesigned a product page and everything broke. so that was extremely difficult. The second piece is that even if you got the data, typically it came back as soup, right? So, you know, there was a reason why they called beautiful soup. One of the parsers, know, HTML blobs in constant field, for the record missing, you know, specific data points. And so what's changed is that basically we built at Nimble a platform specifically to solve Omri: Yeah. Amaury Desrosiers: both problem end to end. we orchestrate thousands of web search agents on top of our browser infrastructure, own proxy network and an optimization engine that picks the right approach for every site in real time. the agents doesn't just fetch a page, they understand it, they run the JavaScript, ⁓ know, they navigate dynamic sites, find the right field, validate them, cross-check them, and then eventually, you know, deliver directly in the Databricks, Snowflake Azure, whatever data platform, structure that agent dashboard or model are going to be able to to leverage dynamically. So the customer doesn't see any of the complexity. They describe what they want and they just get reliable structure stream of web data directly to their environment. So the maintenance overhead, like all of these disappear. They can just, you know, leverage the data the same way to where it could have been, you know, internal data directly in the table that they're already using. Omri: Yeah, that is honestly, somebody who's like used a beautiful soup framework. That is honestly amazing being able to get data you can trust and don't have to then massage it to actually make it usable. And using all of these new AI capabilities, you're talking about semantic understanding of the websites and so on. How do you see this changing the field for organizations who weren't able to build these pipelines? Do you see this as more as small organizations who didn't have the data engineering capacity can now match what was being done by the most technologically advanced companies? Or is this something else where everybody's starting to use this and it's kind of leveling the playing field? Amaury Desrosiers: It's absolutely changing how it works. If you look at this like in the past decade, it was more like, okay, if you have a team of 20 engineers that could actually try to build this, right? So, you know, if you're like a massive tech company, hedge fund with a big quantity, but a mid-sized, mid-market retailers, forget about it. Like they would buy a static data set, maybe once a quarter and call it competitive intelligence. And now we're seeing, you know, 50, 20 % companies standing up in an agentic workflow that consume live data, web data and production, repricing their portfolio like on a daily or sometimes hourly basis. So they're not building a proxy infrastructure. They're not maintaining, you know, headless brothers, they're not passing HTMLs. They're just describing what they need, you know, in natural language, whether, you know, by coding APIs or, know, through... ⁓ cursor, codecs, ⁓ basically getting structured data directly their environment. So that's a democratization. And while the mode to be the infrastructure, ⁓ now the mode what you actually do data. ⁓ So yeah, definitely democratize data availability. Omri: So interesting what you're saying. So in that case, you're thinking how I'm actually thinking about my data estate and my data usage is becoming the mode. It's not the technological capabilities anymore. It's ⁓ do I actually have? Do I need the analysts to be able to think about what kind of questions I want to ask and then I'm able to solve them, right? Amaury Desrosiers: Absolutely. Omri: And I think like real time external data used to be a buzzword for a very long time. Do you feel like that has finally changed being able to bring in whenever you want fresh data and use that as part of your pipeline? Amaury Desrosiers: Absolutely. Real time is not nice to have anymore. It's really worth the value compounds. Every minute of freshness translates directly into better decisions, faster reaction and outcome. Especially when speaking about retail and CPG, if something was in promotion yesterday or last week, who cares? It's already too late. That's fundamentally like a new dynamic. that really find how people are their stack and what kind of data they need. ⁓ can give you an example. We're working with a ⁓ large ⁓ chain and want to understand ⁓ for much every zip code in the US, what are the offerings from ⁓ competitor places, whether they're large chain like Mom and Pop's shop, ⁓ what of ⁓ menus, kind of offering, what kind of promotion they have. ⁓ And then if you know like, what does it look like at seven in the morning or 9 p.m. or 12 p.m. the offering is going to be different. So it has to be real time. They need to know at multiple times a day at critical points, what is the offering for a thousand restaurant across, you know, thousand of zip code. What are the offering? What is the competitive landscape to basically be able to reprice and make the best decisions. Omri: Yeah, definitely. And we talked about the football World Cup coming up. Again, being able to understand now there is a game going on. Do I need to change my pricing? Like these are the kind of questions now people are going to be asking themselves. Do see that like the companies have the right that I think even right skills inside the company to be able to ask these questions? Amaury Desrosiers: Yes. I think so. think what the beauty of AI is that now you can focus on the business part, on the business question that you ask, and eventually get some help on how you can translate this into data points. And the focus now is more on the business side, and you abstract and get data very simply with Platform like Nimble So yes, I would say absolutely. ⁓ The fact the business team... ⁓ can be engaged much earlier in the decision process and build this potentially on their own is a very big shift. Omri: And moving to the more technical aspect of building all of these pipelines. Now, when you have a lot of internal data and external data, how do you see that changing the way in which pipelines are being built in your customers? Is it still being able, is it still the pattern of storing everything into Databricks breaks or Snowflake and then running queries on top of that? Is it a lot more ad hoc cases where you don't even really need to store it for a long time? How do you see that changing? Amaury Desrosiers: It depends. We see both use case and I think it depends on the use case. see the pattern of, to take the example of retailers, people saying like, you know what, I want ⁓ entire categories from ⁓ Target, Amazon, you say it, like I want it every morning or multiple times a day ⁓ understand the full competitive landscape. But I might have some, so in that case. ⁓ You want something that you fetch fresh and you store it and you basically build trends across time to understand how it's changed, what product might be emerging, trends and things like this. Now you might have some cases like World Cup could be one where it's going to be more ad hoc whether it's understanding which... which company are putting banners advertisement or ⁓ they have specific offers as of specific games or ⁓ on So in that case, yes, it can absolutely be fetched on but ⁓ typically see a combination of really depending on the use case. Omri: So you still see companies relying heavily on their internal data warehouse and kind of using that external data with a batch process to fill it every day with the new data that's relevant. But also that agent fetching, is that also a very common use case you guys see? Amaury Desrosiers: Absolutely, whether it's you know to some specific information, ⁓ hoc or just searching the web to answer specific ⁓ work with ⁓ consulting management consulting, and they specific questions that they need to answer ⁓ for the And you know, they would break down specific questions and use Nimble to search the web. ⁓ and basically gather all the individual points to eventually answer a question like, you know, what's the size of a type two diabetes, the market size for that new type two diabetes drugs, for instance. And breaking down in multiple questions, like, you know, what's the price point of such a drug, like population and things like this, to basically try to, you know, the same way a management consultant would do it and do the research. automating this with Nimble to gather all the raw information and the raw response and basically building a model around it. Omri: Super interesting. And now again, on the technical aspects of like what it means to bring in this data and use it. do you see companies struggling to do when trying to kind of ⁓ the way in which they've worked with pipelines up to now? And when they're trying to bring in new data, like what are the pitfalls you were seeing when somebody's like, okay, I'm used to building a process that runs in this stale updating once every few months data. Now I want to change it to be more real time. Where do you see that going wrong? Amaury Desrosiers: Yeah, what's is that the way people think about pipelines, ⁓ you one connector at a time, right? So that works when you have 10 sources, but ⁓ when falls apart, the web is basically your source, right? Because there's no final list websites to build for. ⁓ And that's you need a different process or way to work. not a connector per site, but a system that works for any website that you might be interested in. And that's the whole idea behind our web search agent at Nimble, is that you have one layer, you can point it anywhere, and you get structured treat your data back in a specific schema. So you don't need to think, know, oh my God, how is it to work if I want to get data from X, Y, and Z websites? Just point it to that specific domain, and we basically can fetch the data directly. So you can think about Nimble as, you know, converting the web into a different API endpoint that you can basically gather data from and automate basically the gathering. Omri: And then do you see most companies actually like having a single raw table for the entire internet or the entire kind of domain they're trying to collect and afterwards like building out the data models for it? Or is it something else you guys are saying? Amaury Desrosiers: Typically it's by use case. So, you know, if you think about CPG retailers, ⁓ they're going to have a schema that's gonna fit, you know, their product architecture, ⁓ you with information, prices, promotion and signatures. If you look at, we have one of customers, they're like one of the biggest beverage companies in the world. And ⁓ basically what care about is where they should send their sales team. So on a daily basis, we're capturing the web the menu of every single restaurant in Japan to understand, you know, we're selling which soda, which beverage at what price. ⁓ that basically the sales team knows, basically their... door they knock on to say like, okay, you're not selling our product or you're selling specific competitor product, ⁓ competitor product at price and basically optimize the efficiency of the sales team. So in that case, it's gonna be more like with menu as kind of a data model, ⁓ but across restaurant, it's gonna the same data model, ⁓ whether you're in Tokyo or or anywhere else. Omri: And when you started working with them, did they kind of have to change the way in which they've thought about their data model before? Because I assume they were trying to collect this information before, doing it in a very slow process, maybe one restaurant at a time, as you were saying. So did they kind of change the entire pipeline to this new kind of data coming in and the influx of data they need to manage? Amaury Desrosiers: Not really, like the implementation was actually fairly straightforward because What changed was the scale. The fact that, yes, they were capturing a couple of data points, like either manually or at a very low scale. Now we can basically feed that for the entire Japanese market on a daily basis. So the table were already there. They're already capturing the data. What changed is the scale of the volume and the frequency of the data that they can leverage and eventually the sample size that they have to make business decisions. Omri: Yeah, 100 % and does that then reflect on the way in which they have to change, I don't know, partitioning and how they're running things or do you usually see that? Amaury Desrosiers: 100%. Yeah, obviously there was a change process happening on top of it. But yes, they developed a dashboard on top of it with color coding specific region or specific neighborhood of each city. it's like, this is basically a high value area. If you need to spend some time anywhere, go here because this is where you have the lowest hanging fruits. Omri: Amazing. And do you see them ever struggling because being able to scale up your data infrastructure so quickly to handle the amount of raw data and the external data you're bringing is not an easy task at all. Do you see that being a challenge for companies being able to handle the scale? Amaury Desrosiers: So it could, but I think that's why we're happy to have a great partner like Azure Databricks and Snowflake where it makes it pretty much transparent for the customer. And that's been very, very efficient. Omri: Amazing. Now, if we look ahead and we'll say three years from now, what do you think the data stack is going to look like for most of the companies? How much of it do you think is going to be internal data versus external? How much of it is going to be ad hoc analysis batch processes that are coming in every day? And I think lastly, maybe more interesting, how much of it do you think is going to be consumed by humans versus agents actually consuming them? Also understanding what kind of questions you want to be asking. Amaury Desrosiers: Yeah, it's a great question. I'll try, but given how quickly AI changed, I think even predicting three months from now is going be tough, but I think for three years, I think the line between internal and external data is going to start to disappear for the end consumer. whether the row comes from UCRM or for a website that was rendered like 10 seconds ago, the analysts, more often like the agent, they won't care and they shouldn't have to. Omri: Yeah, fair enough. Amaury Desrosiers: I think the second point is like what you just mentioned, like most of the data will be consumed by machine and not humans. Today, I would say dashboard are still the primary interface, but in three years, agents will be the dominant consumers of data. that changed everything about how you design the stack, mean, from a token efficiency, structure output, latencies, schema clarity. it becomes much more important than like, you know, dashboard aesthetics. And I think the third one is ⁓ things are going to stop being a category. I think it's just going to be the default property. Just like, you know, think seeing real-time data platform ⁓ in is going to be saying like color TV. Yes, like by default, like yes, we know when you buy your TV is going be in color and something I think people are going to be expecting real-time to be to be the default mode as well. Omri: I think the last point you said is spot on. People are expecting things to be real time and no way we'll keep saying, we have real time data ⁓ a category. And lastly, and for any data who's listening to us and asking them, what do I do tomorrow to start working with data? What do you tell them? What do you think they do? Amaury Desrosiers: pick one decision that your business is making blind right now because you don't have an external data. Pricing, competitive intelligence, supplier risk, candidate research, something specific, So, not a strategy, just like a decision that you're making blind. And again, don't boil the ocean. web data flowing into your existing environment, join two internal data in front of one team, ⁓ even a two-week project, not just like two-quarter project. And three-merger what changed. Are you able make faster decision, better outcome, more automation? If you ⁓ point what changed, ⁓ you didn't pick the right case, and go back to step one. But by the way, go to the Nimbleway.com website ⁓ you can create an account, start using it. We have a trial that you can start using it ⁓ you know doing directly a API or plug it into a know codex, Claude code, Cursor, ⁓ I know, codex code, you name it and start getting data extremely quickly. Omri: Amazing. I'll just a few of my takeaways here. I think the last one you mentioned is, first of all, the fact about real-time data no longer being a category, but just an expectation of anybody building. The second one is that external data is becoming a first-class citizen in the data stack. ⁓ we're soon also not going to be talking about where the data came from, but just expecting it to be part of that ⁓ we use and making sure that we have both internal and external data to drive decisions. And lastly, being able to know what kind of questions we want to answer and then bringing in the right data for that. We need to stop thinking from a perspective of what data can I pull in and then let's see how I use that to, what do I want to answer? Cause all of the data is going to be available. So I think that's a wrap for today. Thank you so much for joining us and for sharing ⁓ your insights. And to anyone out there, if you enjoyed this, follow the Data Splash podcast. We have a lot more coming. Thank you. Amaury Desrosiers: Thanks, everybody.