Latest / Agentic Conversations (formally mlops.community) / The Current State of Agentic Retrieval - Qdrant Roundtable

Transcript

The Current State of Agentic Retrieval - Qdrant Roundtable · Agentic Conversations (formally mlops.community)

• 0:09
  What is up everyone? Hope you were jamming to that
• 0:14
  music as much as I was. I can see the chat is already
• 0:18
  getting off to a good start. I'm excited because we've got a
• 0:22
  packed house today to talk all about good old state of agentic
• 0:28
  retrieval. There's so much that we're going
• 0:31
  to get into and we've got so many incredible folks here
• 0:35
  today. But I want to set the scene
• 0:38
  before we bring out the guests of honor.
• 0:41
  And I want to talk to you about why this session feels so
• 0:45
  important. And it is probably why you
• 0:49
  actually came. You know, there's so many
• 0:52
  different challenges and changes that we've had with retrieval
• 0:57
  over the past two years. I have seen such an evolution.
• 1:04
  And so I thought, why don't we get together some people who
• 1:08
  have been knee deep in this whole retrieval game to talk to
• 1:12
  us about what it was like, what did we have back in the day and
• 1:16
  what do we have now and what are ways that we can optimize it so
• 1:19
  that we are finding success right now.
• 1:21
  So without further ado, I'm going to bring on the team from
• 1:26
  Quadrant. We've got the Devrel folks
• 1:30
  coming out. First up is my man Dylan.
• 1:36
  Where are you at it? Hey Dylan, how you doing?
• 1:40
  We've got Jenny, we've got Eva, Neil and Andre.
• 1:47
  I think this is the most people that we've ever had on a round
• 1:51
  table. So let's just kick it off, start
• 1:56
  strong with some hard hitting questions.
• 2:00
  I'm going to ask what has changed in the last two years?
• 2:06
  I I've seen so much. What do we need to focus on?
• 2:11
  Yeah, sure. Yeah, I can take that.
• 2:14
  Hi everyone, Neil here. Yeah, thanks for joining the
• 2:17
  session and thanks for the intro.
• 2:19
  Demetrios. There's so much happening in
• 2:23
  Retrieval and Surge. I, before I joined Quadrant, I
• 2:28
  didn't fully appreciate the, the field of search.
• 2:30
  And we kind of just expect search to work.
• 2:34
  And we definitely notice when search doesn't work well and we
• 2:38
  notice it through just getting like terrible results and, or it
• 2:44
  takes a long time to get results.
• 2:45
  We can't find what we're looking for.
• 2:48
  And the more data that we capture and from the real world,
• 2:52
  the more we need to get that information, get, get the data,
• 2:56
  the right data we need, get the information out of our, the data
• 2:59
  we're storing. So what the back to the
• 3:01
  question, what's changed? One of the biggest things has
• 3:04
  changed is agents and the topic of today's round table agents, a
• 3:10
  lot of them don't really search well, I would say, in my
• 3:16
  opinion, there's a lot of they, they kind of search like a, like
• 3:22
  a, a beginner would just search by just like kind of throwing a
• 3:26
  lot of things into the, into the hat and just like trying to like
• 3:31
  pull, pull, pull results. And they go through lots of
• 3:34
  turns and they burn a lot of tokens and they try to find this
• 3:39
  context, but they're not using all the tools available for
• 3:42
  search and this. And if you don't have the right
• 3:44
  context in your AI, that's going to really make your AI less
• 3:48
  efficient, less effective. It's going to prevent you from
• 3:52
  reaching your goals. So agents and, and one of the
• 3:56
  things that agents are doing that's also interesting is like
• 3:59
  a human might search like once or twice a minute or so.
• 4:02
  Agents are searching thousands of times per minute.
• 4:05
  So they're just going boom, boom, boom, boom, boom hitting
• 4:07
  it. And they're not able to evaluate
• 4:10
  those search results. So same way human can, because
• 4:14
  humans have a lot of more judgement that they can apply.
• 4:17
  So yeah, I'll stop there. But those are those are some
• 4:20
  things that have really changed. Dude, you're preaching to the
• 4:23
  choir on the different ways that agents search and then the ways
• 4:27
  that they interpret that data. And so I, I have some questions
• 4:32
  here about like, hey, now that we've got agents and these
• 4:36
  agents are figuring things out, they're kind of, we've all seen
• 4:40
  it. They will stumble through
• 4:42
  different problem statements where they'll try and figure it
• 4:46
  out and they don't necessarily go about it the most efficient
• 4:51
  way possible. I wonder if you all have found
• 4:56
  tricks on nudging the agents to be like, Oh, yeah, by the way,
• 5:02
  like there's obviously the go look in that file trick, but
• 5:07
  there potentially could be plenty more that you've seen.
• 5:10
  Because this at the end of the day, is a search and retrieval
• 5:14
  problem. And the faster that you can get
• 5:17
  those agents, that knowledge, the faster they can hopefully do
• 5:21
  what you've asked them to do. Yeah, I'm I'm actually going to
• 5:26
  take jump in here again, but I'm going to pass the Dylan on this.
• 5:28
  So Dylan, get ready. So just to set like a little
• 5:34
  more context there is, you know, it's not just the efficiency,
• 5:39
  but it's the effectiveness of search like finding the most
• 5:42
  relevant response, relevant results.
• 5:45
  And I think that it goes into evaluation.
• 5:49
  How do you evaluate that the results are good enough to use?
• 5:54
  Should you try again? Should you change your query?
• 5:57
  And if you should try again, when do you stop?
• 6:01
  When like how how long does the agent go before it stops?
• 6:05
  What tools can it use available to it?
• 6:08
  And Dylan's actually done self correcting agent loops that he
• 6:14
  can talk a little bit about. So I'm going to hand to you to
• 6:16
  talk a little bit about what you found.
• 6:19
  Yeah, absolutely. Thank you.
• 6:21
  So my first points that I'm going to talk about is that
• 6:25
  Neil, you just mentioned evaluations and I feel like a
• 6:29
  lot of people are familiar with like the general, you know,
• 6:33
  evaluation frameworks and evals that you can run on an agent.
• 6:38
  But also in the information retrieval space, we have a lot
• 6:42
  of our own like evals and metrics like NDCJMR that are
• 6:46
  like really relevant for, you know, search pipelines and that
• 6:51
  that we use extensively. But going into your what, what
• 6:56
  you mentioned about like self evaluating agents.
• 6:59
  So, you know, there's a lot of techniques for for that.
• 7:03
  You know, the most obvious one is to spin up like a small agent
• 7:07
  and say, hey, was that result relevance?
• 7:10
  Did it contain, you know, all the information that the user
• 7:14
  was looking for? But the problem here is that,
• 7:17
  you know, you add another like LLM roundtrip so that it is
• 7:21
  extra latency, extra cost, Ian, and it makes your pipeline
• 7:25
  slower, more complex. And so something that we've been
• 7:29
  working on was computing statistical signals that, you
• 7:36
  know, very cheap signals based on the, the results.
• 7:41
  And so, so for example, you know, when when you retrieve,
• 7:45
  you know, using like dense or sparse embeddings, basic
• 7:48
  basically you get like a score as a result.
• 7:53
  So for like dense embeddings, it is between, you know, like 0 and
• 7:57
  one which one being like the most confidence.
• 7:59
  And you know, you return like the top K results, so the top
• 8:03
  five or the top ten. And basically you can do some
• 8:08
  statistic based on like the spread of those scores or how
• 8:14
  the difference between the the top one and the top second
• 8:17
  results. And basically from here, try to
• 8:21
  create a programmatic, you know, signals that will that that will
• 8:28
  be able to tell you if the quality of your retrieval was
• 8:32
  good or not without virtually any adding any compute to your
• 8:37
  pipeline. And then once you you have that
• 8:40
  that signals that signal that tells you, you know, the the
• 8:43
  retrieval was good or not, then you can basically route your
• 8:48
  retrieval agents to like different paths.
• 8:51
  And you can choose, for example, to do like more expensive
• 8:55
  retrieval method, like a cross encoder or late interaction
• 8:58
  model. Or if the result was not good
• 9:01
  enough or not like not good at all at all.
• 9:05
  It's like, you know, there was like a huge divergent in in
• 9:08
  between like all the scores that you retrieved.
• 9:10
  This is when you can invoke an additional LLM.
• 9:14
  So the in order testings, this was like really a good way to
• 9:19
  tell apart like good from bad retrieval in a way that is
• 9:22
  really cheap and allows you to spend more money only only when
• 9:27
  it was necessary. And then we also have some
• 9:34
  something on that topic on skills.
• 9:36
  So I'll pass it over to Jenny. I think it's super funny.
• 9:40
  We're working like, you know, actually a multi agent chain out
• 9:44
  there because we're passing tasks to each other and
• 9:48
  Demetrius, you're the evaluator of how the pipeline is going,
• 9:51
  right? So we're just emulating search
• 9:55
  and informational retrieval from our heads in the going.
• 9:58
  And yeah, like the little addition as I think everybody
• 10:01
  knows about the genetic skills also, right?
• 10:05
  So they're also applicable in search.
• 10:07
  The problem is agents need to know how to use search as humans
• 10:13
  actually sometimes also do it bad because they also don't know
• 10:16
  how to search properly for something, but they do it
• 10:18
  differently. With agents.
• 10:21
  There is a good solution of teach them how to approach it is
• 10:24
  to show them skills. For example, we designed some
• 10:29
  bucket of search engineer skills which we're trying to maintain
• 10:34
  in quadrant which explained our agent which approaches our
• 10:39
  vector search engine on what to use and how to combine it to get
• 10:44
  different outcomes. For example, if you have a
• 10:48
  problem that your search doesn't show for you as an agent,
• 10:52
  relevant results at the top and you feel like there is something
• 10:55
  more to it, then there is a specific markdown part of the
• 10:59
  skill which tells, Hey, you should do this, this, this with
• 11:02
  our, with our APIs, I think we can share the link later.
• 11:08
  Or I could try to share the screen, but I'm so afraid to
• 11:11
  share. You know, it's the hardest
• 11:13
  thing. That's why like I'm senior
• 11:15
  developer relations, but I never managed to share a screen
• 11:18
  without sharing my private chats.
• 11:20
  So let's see, worst case we're just dumping the link and
• 11:23
  showing how the skill evolvement for the search pipelines is also
• 11:28
  helping our agents to use our search infrastructure properly.
• 11:34
  Classic. OK, I've got some follow up
• 11:36
  questions and there are some awesome questions that are
• 11:40
  coming through in the chat, but ever I wanted to give you some
• 11:44
  time in case there's anything you wanted to tag on here.
• 11:50
  Sure. So Jenny mentioned a little bit
• 11:52
  of how we can teach agents how to know search.
• 11:57
  So essentially when we added the agents, we moved from a very
• 12:03
  static system of RAG to something much more dynamic.
• 12:06
  But actually the search primitives that are underlying
• 12:10
  remain the same. So one research paper that came
• 12:13
  quite recently last month that I found really interesting on the
• 12:17
  topic is called Sera, which is the Super intelligence retrieval
• 12:22
  agent. And the whole idea there is to
• 12:27
  achieving super intelligence, which is when the multi round
• 12:30
  search is compressed into a single corpus retrieval.
• 12:34
  So we've essentially avoided the latency that Dylan was talking
• 12:40
  about the the latency lags. And what they've done that was
• 12:45
  quite relevant to to our today's topic is they figured out how to
• 12:49
  use the LLM to enrich the documents.
• 12:52
  So have this iterative loop whenever the search vocabulary
• 12:58
  was missing. So they looked at the queries
• 13:00
  that were generated, evaluated what queries that were missing,
• 13:04
  and essentially predicted omitted vocabulary and added to
• 13:09
  it before they passed it on to more traditional search
• 13:13
  paradigms like BM25. OK.
• 13:17
  There's a lot to unpack with all three of you saying this stuff
• 13:22
  because basically, let me just say what I caught real fast and
• 13:28
  then you can correct me if I'm wrong there.
• 13:32
  Dylan, You're talking about different ways that you can use
• 13:36
  statistics to leverage and get better results with the
• 13:41
  retrieval and with the agents recognizing what to grab and
• 13:45
  what to retrieve and what is used.
• 13:53
  Tell me what? Yeah, tell me what?
• 13:55
  Sorry, yeah. So basically it was like a cheap
• 13:57
  way to get early signs into bad or good retrieval without, you
• 14:03
  know, adding any latency or cost to your pipeline and then routes
• 14:08
  based on those signals. All right.
• 14:12
  Then from there, Jenny, you mentioned using skills and how
• 14:18
  you all at Quadrant have created a bunch of skills.
• 14:21
  Are those open source? I didn't catch that one.
• 14:22
  That was the quick. Everything that we do,
• 14:26
  everything ish that we do is open source.
• 14:29
  So, yeah, obviously. Drop that into the chat here and
• 14:34
  I can relay it to the chat or put it in wherever, whatever
• 14:40
  chat you want. But we definitely I want to see
• 14:42
  all of those skills. And lastly, Eva, you were saying
• 14:46
  that there is a paper on the Super intelligent retrieval
• 14:51
  agents and the way that they're able to leverage and I didn't
• 14:58
  catch now. Now my memory is starting to
• 14:59
  fade trying to hold all these things in my hand head.
• 15:03
  The Super intelligent retrieval agents were doing what now?
• 15:08
  So it's just a framework of how do we know that we've build an
• 15:11
  agent that essentially does successful retrieval?
• 15:17
  Incredible. OK, cool.
• 15:18
  So before I hit some of these questions from the chat and and
• 15:25
  I'm just like looking at the chat about big question that's
• 15:29
  coming up is like, hey, what about ground truth?
• 15:32
  How do we figure that out? What is cause to know if the
• 15:36
  retrieval is correct, right? We also need to know what the
• 15:40
  the actual thing is, the ground truth that we're looking for.
• 15:44
  Does anyone have opinions there? I see some things taken I.
• 15:54
  Would say, I would say a proper bunch and each of us has like I
• 16:01
  would say different sides of it. And also about what ground truth
• 16:05
  is, I think our problem is that we got the agentic systems and
• 16:12
  we got super excited and we're like, are they solving search
• 16:15
  fully? But the problem is that
• 16:17
  evaluations weren't fully solved before with the classical
• 16:20
  search. And now with agentic search, it
• 16:22
  gets even more tricky because you're like, is the ground truth
• 16:26
  that the agent successfully fulfilled the whole task and
• 16:29
  arrived to the right point? Or is there is a ground truth on
• 16:33
  the age point of the iteration of what it does in the process
• 16:38
  and there are several approaches to it.
• 16:40
  There is ground truth in retrieval as probably people
• 16:44
  know can be created from some golden data sets per SE.
• 16:50
  So some from the data that you already have and you know that
• 16:55
  this question should get this answer from your data set.
• 17:00
  Usually in production it's not so easy to get it, but LLMS got
• 17:05
  pretty good at helping to generate the synthetical data
• 17:09
  sets with golden truth. The one tip and opinion that I
• 17:13
  have as the person who previously worked in
• 17:15
  crowdsourcing as a devil of crowdsourcing approach, so where
• 17:19
  the humans were gathering these data sets is that don't
• 17:23
  immediately expect dumping your protectional data to LLM that it
• 17:28
  will create your ideal golden set evaluation.
• 17:31
  And creating the golden truth still is the work where you need
• 17:35
  to be in the dialogue like you're in the dialogue
• 17:38
  developing a cold project. So the relevance notion, the
• 17:42
  ground truth notion still comes from you as a domain expert and
• 17:46
  that can be like LLM could be used as a tool to develop the
• 17:51
  ground truth data set that you can then inject in your
• 17:55
  evaluations. But when it comes, I know I have
• 17:59
  too much to say 1 little thing, one little 11 little thing.
• 18:03
  But when it comes to the whole thing as the whole pipeline to
• 18:08
  be successful for you like solving the task, there are very
• 18:14
  different approaches on how to teach model to do the correct
• 18:18
  search trajectories which goes a little bit in more into
• 18:22
  reinforcement learning domain. And I feel like there is a lot
• 18:25
  of emerging there teaching agents to do the right decisions
• 18:29
  in search. And there are the ground truth
• 18:32
  is the result being correct given the task, given the search
• 18:38
  task, the search input. Maybe your audience knows Ralph
• 18:42
  root loops. So they're also to some extent
• 18:44
  applicable to the search. But Dylan here is coming from a
• 18:50
  rise and he was doing evaluations as as the bread and
• 18:54
  butter so. So, Dylan, this dovetails nicely
• 19:00
  into one of the questions that's in the chat.
• 19:05
  Someone was asking about how teams are versioning and
• 19:10
  evaluating retrieval pipelines in production.
• 19:14
  Yeah, absolutely. So you know, something that I
• 19:19
  want to preface for preface with first is that we are a vector
• 19:24
  search engine. So you know, we only provide
• 19:27
  only one of the cogs in the machine.
• 19:31
  And you know, this is really by purpose.
• 19:34
  We don't want to be selling like or providing like a retrieval
• 19:38
  agent. We really want to stay very
• 19:41
  close to the metal and provide like the, the best retrieval
• 19:45
  engine possible. But when it comes to those kinds
• 19:48
  of like evaluations and versioning, I will usually
• 19:53
  suggest to use like an evaluation framework or
• 19:56
  platform. Of course, you know, I come from
• 19:59
  our IAIS, so I'm, I'm, I might, my opinion might be biased
• 20:03
  there, but there's a bunch of free evaluation tooling that can
• 20:09
  help you do that that versioning there.
• 20:15
  Awesome. And I think Andre, you had some
• 20:19
  other thoughts about this too. Where are you?
• 20:23
  I got to change the view to see. There he is.
• 20:26
  There we go. Yeah, Andre's been doing.
• 20:29
  Space on the screen. So I, I would like to say that
• 20:34
  from my experience, what can also help if we jump back for
• 20:38
  the, for the golden set is when you kinda have some kind of
• 20:43
  assumptions based on the human, on some human manual labor work.
• 20:49
  What can be good for your golden set so that the LLM has some
• 20:54
  kind of criterias that it can evaluate about.
• 20:57
  And I would like to say that it's like an iterative process
• 21:00
  in which you try to improve your LLM as a judge.
• 21:05
  Yeah, time from time so that the golden set that it is creating
• 21:08
  and the evaluation in general becomes better.
• 21:11
  So it's a long process. It's iterative process that just
• 21:16
  becomes better as you train it more, yeah.
• 21:21
  Awesome. Well, we've got more questions
• 21:25
  in the chat. I see the let me just let me
• 21:29
  just grab this. So Eva missing search terms are
• 21:35
  Grafton dynamically. How does that handle prompt
• 21:38
  injection shenanigans, if at all?
• 21:44
  This is a good question so in the paper and feel free to share
• 21:49
  the link in the chat. Hopefully you have it.
• 21:52
  I I haven't found the cybersecurity aspect of it, but
• 21:56
  I think this would be like a really nice segue to agentic
• 22:02
  harnesses. And as you design your system,
• 22:07
  retrieval will not solve everything.
• 22:10
  You still need to add tools onto your agent that would make it
• 22:14
  more secure. But I think this is a really
• 22:17
  important design principle to to consider because prompt
• 22:21
  injection is definitely a very serious serious threats.
• 22:26
  And I think especially recently we had open claw and or miss an
• 22:32
  agent that came out. And with open claw there was a
• 22:35
  lot of hype, but also we saw when we did not properly sandbox
• 22:40
  it that it was able to just go completely wild and access all
• 22:44
  of our data. So yeah, I think that is not
• 22:48
  specifically mentioned in that paper, but consideration that
• 22:52
  every single good agentic retrieval system should have,
• 22:56
  yes. So let me continue with a few
• 23:01
  more questions from the chat and then we will go on to the next
• 23:05
  topic that I had in my notes. There is my biggest pain point
• 23:10
  from John is he's saying biggest pain point I've had with rag
• 23:15
  stacks is defining edge and entities dynamically with LLM
• 23:21
  analysts. Can you speak to different
• 23:24
  approaches to that problem? And I'll just throw that to the
• 23:28
  crowd. Anybody have strong thoughts?
• 23:35
  I agree because I've heard it a lot on the conferences, but I
• 23:38
  believe it's more of the graph ontology construction problem.
• 23:45
  And we as the vector search engine don't do the ontology in
• 23:50
  construction. We usually combine because in
• 23:54
  many, many domains it's a very good complementary thing.
• 23:57
  For example, in memory of agents, it makes sense to mix
• 24:01
  both graph and vector search. That's how many agentic memory
• 24:06
  providers do, for example, Cogni.
• 24:10
  But I've heard that list from practitioners on the conferences
• 24:15
  and don't quote on me. I would ask Neo 4 J guys on
• 24:18
  that, that if you atomize the task of ontology instruction in
• 24:24
  the sense you don't try to build the whole ontology of the
• 24:29
  domain, but you do it bit by bit in the more atomized setting
• 24:35
  that actually performs much better, like the outcome is much
• 24:39
  better. But I also know that Andrey
• 24:43
  before worked with ontology. So maybe you have like a little
• 24:47
  point of view on that. Yeah.
• 24:52
  So can you repeat the question again so that I I have the
• 24:57
  latest? The biggest pain point I've had
• 25:00
  with Rag stacks is defining edge and entities dynamically with
• 25:05
  LLM and analysis. Do you have any approaches on
• 25:10
  that? Oh, I would say that, yeah, the
• 25:16
  problem is that these are kind as, as Jane, you already said,
• 25:21
  some kind of ontologies that you try to provide to the vector
• 25:25
  search and to the semantics. So you you you kinda try to put
• 25:30
  the explicit semantics into the vector semantics.
• 25:33
  And I think these are laying down two different layers.
• 25:39
  Yeah, 1 is application layer and another one is, is more of
• 25:45
  implementation right of the technology.
• 25:48
  So I would say right now what you can do is basically just
• 25:56
  like the the current state of art we can say is that there are
• 26:00
  lots of GitHub profiles and GitHub projects that are trying
• 26:08
  to combine new 4G approach here with explicit entities and also
• 26:15
  nodes with something that is vector based.
• 26:19
  And there are some successes there because or what does the
• 26:26
  graph tell you? It can tell you the
• 26:28
  relationship, can tell you the dependencies and in some sense
• 26:32
  it can give you additional context that semantics not
• 26:35
  always can get, you know, because like you can have
• 26:38
  something that is 2000 from 2001 and something from 2014 and you
• 26:45
  want something specifically that is from 2014 to be more aligned
• 26:51
  on top. You can you can fix it of
• 26:54
  course, with some custom, some custom scoring, some custom
• 27:01
  relevance metrics that you provide by yourself.
• 27:04
  Maybe again, it depends on the domain, but I would say there
• 27:09
  are these approaches that they basically try to combine vector
• 27:13
  and graph in one stack. Yeah.
• 27:17
  Or you can try to encode in some way information into the red.
• 27:23
  But yeah, it's quite hard, yeah. Awesome.
• 27:27
  Well, let's keep it cruising because I want to get into
• 27:30
  memory and it feels like that is a perfect segue into memory and
• 27:36
  how folks are approaching this, how you all have seen the best
• 27:41
  in the business do it. And especially like I know there
• 27:44
  is a lot of talk about long term memory and the architectures
• 27:50
  that you have for those. And also if they are that
• 27:53
  valuable because sometimes I will not necessarily want
• 27:59
  something to be remembered, but it gets remembered.
• 28:01
  And then for every single question that I'm asking my
• 28:06
  agent, and I'll give you a concrete example of this, I
• 28:09
  told, I told the chatbot that I was vegetarian.
• 28:15
  And now for things that have nothing to do with food, it
• 28:18
  says, well, given your vegan lifestyle.
• 28:22
  And I'm like, I asked you a question about my taxes.
• 28:25
  Get the fuck out of here. Why would you need to reference
• 28:29
  my vegan lifestyle? So anyway, that's just a little
• 28:32
  bit of a tangent. Maybe we can talk long term
• 28:34
  memory or we can also. I'm going to throw different
• 28:37
  things out there like episodic memory.
• 28:39
  Do you want to go event based verse?
• 28:41
  Semantic memory? How do you do the factual verse
• 28:44
  conceptual? Who wants to take this one?
• 28:47
  I'll throw it. Throw it up there and let
• 28:49
  anybody rise to the challenge. Let me try to jump in because I
• 28:59
  relate a lot to this part about the vegan taxes.
• 29:03
  I think it's the classical, you know, LLMS, like please don't
• 29:07
  think about elephants. I think Dylan taught me that.
• 29:10
  And then immediately whatever you do, it's going to recall
• 29:13
  this elephants. That's why it's also remember
• 29:15
  when you do anything around memory is also to think about
• 29:19
  forgetting as a conception. But it's a very hard one, an
• 29:22
  interesting one. And there are like several tool
• 29:24
  links on how to do that. So basically whatever I wanted
• 29:29
  to say is very quick is that I think people out there probably
• 29:34
  tried Hermes or open claw agents because who didn't?
• 29:40
  It was my first Wow effect, to be honest.
• 29:43
  And we recently recently been like day before yesterday,
• 29:48
  actually, it's, you know, hype driven development a little bit
• 29:52
  released a plug in which allows to store episodic and semantic
• 29:55
  memory of a Hermes agent in quadrant.
• 30:00
  And I don't have a super sick demo or anything, but I chat
• 30:05
  with my RMS agent a little bit and some of the memories of me
• 30:09
  trying to prepare for this envelopes meetups got actually
• 30:12
  started in my quadrant classer. I can show it but before also
• 30:16
  like a quick note in case somebody just this not very well
• 30:20
  versed with the concept of different memories.
• 30:23
  There is a working one which is the context window.
• 30:25
  The chat you're in there is semantic or factual memory.
• 30:30
  It's the one that something is true about you.
• 30:32
  For example, that the Metris is vegan and likes to do taxes.
• 30:36
  Do you? No, OK, doesn't like to do taxes
• 30:39
  then happens to all of us. Then there is a procedure,
• 30:44
  procedural one. It's actually more like skills.
• 30:47
  So what to do in order to achieve something.
• 30:50
  And the fourth one would be episodic one.
• 30:53
  This perfect thing for the vector search.
• 30:56
  Actually, it's all the recalls of what you have discussed in
• 30:59
  the past and that it could help your agent to understand how to
• 31:05
  deal with the stuff better considering all its private
• 31:08
  previous knowledge. So I wired my Army's agent to my
• 31:15
  cordon cluster specifically for that to show you.
• 31:20
  Let me try to demonstrate it. It's going to be a very lame
• 31:23
  demo, but we have a cooler 1 so don't get don't get discouraged.
• 31:29
  This is nice because people were asking for demos in the chat,
• 31:33
  and so I think I just asked in our background chat if anybody
• 31:38
  else has one. Dylan, you have one too, huh?
• 31:41
  That we can throw later on. All right, cool.
• 31:44
  Yeah, I'll follow up with them as well.
• 31:47
  OK, I'm I'm going to tell you, you're going to love Dylan's one
• 31:50
  and you're going to maybe like mine.
• 31:53
  What am I currently showing? Is it my Slack or is it like UI
• 31:58
  of collections in quadrant? Hold on, I got to check it out.
• 32:03
  I see this no us so yeah, quadrant you're.
• 32:09
  Good. I'm good.
• 32:11
  OK. So as you see, this is a very
• 32:14
  impressive memory of 37 points approximately in quadrant.
• 32:18
  But basically what happens, all of the facts that I have in the
• 32:23
  conversation with my Hermes agent, which I'm not going to
• 32:26
  obviously share because I ask embarrassing questions all the
• 32:29
  time, that's why I wipe the memory.
• 32:32
  So all the facts and all the turns in the conversations are
• 32:38
  saved with the different metadata of the sessions that
• 32:42
  they happened in. And that will help agent recall
• 32:46
  some similar information. For example.
• 32:48
  I'm a big fan of this GE Chepa approach which Jan Dikun is
• 32:53
  recently saying that it's the next big thing for embeddings.
• 32:56
  So agent can find similar memories.
• 32:59
  You can see that with dense vector search for example about
• 33:02
  this method you will be able to find some other memories which
• 33:06
  are kinda about the same fact. And this plug in you can also
• 33:12
  visualize kinda the memories. There is not so much to see yet
• 33:18
  because there is not too much memories.
• 33:21
  But we can see that for example with the semantical embeddings,
• 33:25
  there are similar memories about me preparing some smart facts
• 33:29
  about the Transformers attention window will being very important
• 33:33
  for context windows being very important for vector search.
• 33:36
  And that's why vector search now won't be dead as much as RAC can
• 33:41
  become dead at some point. And if I would start chatting
• 33:47
  now with my Hermes agent, for example, that I will add the
• 33:53
  fact that well that Demetrius is vegan.
• 33:58
  So now all of my inputs are also going to be polluted about.
• 34:02
  This. No, it was the other one that
• 34:04
  was true. I like doing taxes.
• 34:07
  And Demetrius likes to do. I'm not showing you my telegram
• 34:13
  where I'm chatting with Ernies, but that's what's happening.
• 34:16
  So technically, if the double goals are nice to me, we should
• 34:21
  see that the points are going to get updated at some point.
• 34:25
  But if they are not, the demo gods are just not nice to me and
• 34:28
  you will see the nicer demo. We can come back.
• 34:31
  We can come back because I think my RMS agent obviously decided
• 34:35
  to sleep in the moment I decided to demonstrate something.
• 34:38
  But the general conception is that basically you have your
• 34:43
  memory bank because information about you that you do now will
• 34:48
  grow and grow and grow. And some point MD files won't be
• 34:52
  enough to recall all of the information.
• 34:54
  And then you need some, well index which will organize this
• 34:58
  memories, let them forget and let them be surfaced in the
• 35:03
  right moments. And I think this is very
• 35:05
  important. Where are we going?
• 35:06
  Because the information, as Neil said at the very beginning,
• 35:09
  grows and grows and grows. Demo gods absolutely obliterated
• 35:13
  me. But I know that Dylan has a very
• 35:16
  cool 1 so everybody can forget this.
• 35:20
  And oh, go ahead, sorry, we finished it.
• 35:24
  I. Think we've got, well, I wanted
• 35:28
  to talk for a minute about forgetting and that whole thing
• 35:34
  because I know that can be very difficult.
• 35:35
  And there's also a question in the chat that I want to bring up
• 35:39
  about like vector databases still being the preferred choice
• 35:43
  for long term memory. So maybe Eva, can you talk
• 35:47
  forgetting real fast? And then if you have any
• 35:49
  thoughts about vector databases, I can imagine I know which way
• 35:53
  you're going to follow on that one.
• 35:55
  But let's let's go. Hopefully you won't forget that
• 35:59
  one. So yeah, I mean, Jenny showed
• 36:04
  what's like the benefit of of memory and dealing of all of
• 36:09
  that. But I think memory is also
• 36:12
  polluting a lot our context window.
• 36:14
  So forgetting is actually a massive and really interesting
• 36:18
  topic of how do we even decide. Being vegetarian might not be
• 36:23
  relevant when you fill your taxes, but on your next session
• 36:27
  you might be trying to optimize your diet and that information
• 36:31
  will be very important. So this is the world of decay
• 36:37
  functions and also relevance feedback.
• 36:39
  So agents have this nice thing that you can iteratively tell
• 36:46
  them what exactly in that particular search query is
• 36:51
  relevant and you can boost that. So this is the ability where I
• 36:55
  think vector search actually really shines because you can
• 36:59
  figure out for that particular session, what information should
• 37:03
  I include in my decay function and just forget.
• 37:06
  And that can be based on temporary time stamp or that
• 37:09
  could be filtered by keywords, etc.
• 37:14
  And then there's another thing, which is some memories actually
• 37:18
  overtime that you're putting into the sections into our
• 37:22
  sessions, my duplicates. So that's another really
• 37:26
  important thing is in order not to completely jam packed your
• 37:30
  context window and actually get out of your engine agent what
• 37:34
  you want, you can use vector search to de duplicate those
• 37:38
  memories. So we have very clean context
• 37:41
  window and next time you're filing your taxes, it's just
• 37:44
  that and you can boost it and decay based on certain filters
• 37:49
  and scores the information that's not relevant for the
• 37:52
  session. That's awesome, Neil.
• 37:56
  I feel like you had some things to say.
• 37:59
  Yeah, I haven't. Talked to you in a while.
• 38:01
  Yeah, I don't have anything insightful.
• 38:04
  I just wanted to hand over to Dylan with a little bit of
• 38:09
  context. We're talking about memory and
• 38:12
  forgetting and search. And we, we weren't planning on
• 38:17
  showing this demo on, but since there were requests for demos,
• 38:21
  we'll go ahead and show it. This is a demo demo.
• 38:24
  Dylan's gonna show on actually on device search.
• 38:28
  And so when we think about memory and forgetting an AI,
• 38:30
  like physical AI is becoming a really interesting area and how
• 38:35
  agents and AI on physical devices need to be able to
• 38:39
  remember certain contexts, be able to use that context, and
• 38:44
  then how humans can interact with that.
• 38:46
  There's also something really cool that Dylan can can show.
• 38:50
  So yeah, Dylan handing over to you.
• 38:53
  Absolutely. Thank you.
• 38:54
  And yeah, so you know, we're kind of like steering away from
• 38:58
  like the agency topic. But this is the good thing about
• 39:01
  vector search is that it's not only limited to to rag and
• 39:05
  Agentic. And so we're today we're so I'm
• 39:08
  going to show showcase your project that I built for a
• 39:11
  robotic use case. So can you guys see my screen?
• 39:16
  OK. Not yet Hold.
• 39:20
  On Let me do my Job, there we go.
• 39:23
  All right, perfect. So this is like just like the
• 39:27
  the GitHub project. It is completely public and free
• 39:31
  and open source. So if you guys want you to run
• 39:33
  this yourself, so today we're going to run it on like a
• 39:36
  prerecorded video, but it works with like any, any kind of
• 39:40
  input. You can connect your your camera
• 39:42
  and it will stop working live. Can you make it a little bigger?
• 39:47
  Yes, absolutely. There we go.
• 39:51
  Now we're getting there. All right, so basically we're
• 39:55
  starting with zero that agents knows nothing about the world.
• 39:59
  It doesn't have like a database of like labels of what items
• 40:03
  looks like, of what is a chair, what is the floor lamp, what is
• 40:06
  a coffee table. And so we have like three kinds
• 40:09
  of models that run in parallel. So it's all all running on
• 40:12
  device, it's all running locally.
• 40:16
  You know, if I turn off my network right now, you would
• 40:17
  lose me, but I would not lose that product.
• 40:21
  And so basically what what's happening here is that we have
• 40:24
  first a an image recognition, an object recognition model that is
• 40:29
  called yellow, and then a second model that is an image to text
• 40:34
  model. And this model basically creates
• 40:36
  a label and also a description for each item that it sees.
• 40:42
  And then what we can do is that basically we can do semantic
• 40:46
  search on every item that the robot has seen before.
• 40:52
  And so, you know, it started with like a completely clean
• 40:56
  memory when I, you know, when I opened up that that app, that
• 40:59
  robot has never seen anything before.
• 41:02
  And now it's able to recognize every single objects and
• 41:06
  memorize them and then recall when and where is so those
• 41:10
  objects. So, you know, the applications
• 41:12
  for robotics are pretty pretty much endless.
• 41:17
  And here basically you can so you know, this graph is a 2D
• 41:22
  representation of the actual embedding space.
• 41:25
  And so you can see the, the robot, the robot's brain being
• 41:30
  built in real time and you can see all the memory of basically
• 41:34
  all the concepts that are close together.
• 41:37
  So you see like all the hallways are grouped together here.
• 41:40
  We can see like there's like the dining table and chairs that is
• 41:44
  grouped together here. You know, we can see that we, we
• 41:47
  went into the bathroom so that those memories are pretty far
• 41:51
  away in the embedding space. And then you can really recall,
• 41:56
  you know, everything that you has that you have seen before
• 42:00
  and the the robot will be basically to able to recall
• 42:05
  every single mirror that he has seen.
• 42:08
  And yeah. And so you know, this is just
• 42:10
  to, so you know, we're, we're not selling embedded models,
• 42:15
  we're not selling vision models. We're really just that memory
• 42:20
  layer. We're just.
• 42:21
  We have a model that creates embeddings based on those image
• 42:25
  and text, and we allow for very quick search and recall on those
• 42:31
  memories. That's so awesome.
• 42:36
  Whose house is that? Is that your house?
• 42:39
  Oh, I wish. I was going to say that you
• 42:44
  negotiated a pretty good salary if that is your house.
• 42:48
  No, that that was just, you know, like on online tour from
• 42:51
  like a leasing agent or a realtor.
• 42:54
  Oh that's awesome. That is so cool.
• 42:56
  The fact that it's on device too with these the small models is
• 43:01
  really impressive. Yep.
• 43:03
  So, so you know, this was like a way to demo like Quadrant Edge,
• 43:08
  which is the embedded version of quadrants and this version can
• 43:13
  run on a Raspberry Pi. I'm not even like the top of the
• 43:15
  line US1. I have one form like five years
• 43:18
  ago that runs it just fine. Awesome.
• 43:23
  Now there was a question that I said in the chat I was going to
• 43:28
  ask and I totally skipped over it.
• 43:30
  I told folks so we're I want to get to it.
• 43:34
  It's asking about vector databases still being the
• 43:38
  preferred choice for long term memory or our knowledge graphs
• 43:42
  and structured memory stores gaining traction.
• 43:50
  I, I would say from what I'm saying, it's not an either orbit
• 43:57
  of both. We're, we're they, they both
• 44:01
  have their uses in different contexts.
• 44:05
  But you know, like dense, dense vectors, dense embeddings are
• 44:10
  going to give you a lot of like semantic meaning, but like
• 44:13
  vector stores are not really the best for like relationships
• 44:17
  between those different entities.
• 44:21
  And so using like we, we have a great partnership with Neo 4J
• 44:27
  and there's a video I put in the private chat.
• 44:30
  Maybe you can share Demetrios, but like graph rag and using
• 44:33
  knowledge graphs with vector stores together can make a
• 44:37
  really good substrate for memory overall.
• 44:40
  And just to kind of like shout out a couple other companies
• 44:44
  doing this is like Cogney and Men 0.
• 44:49
  Cogney has they use both like transactional data or well, they
• 44:56
  use like transactional data, they use vector data stores,
• 45:00
  then they use knowledge graphs all together.
• 45:02
  So you can check those out too. Nice.
• 45:08
  A question came through about the demo and it was mainly about
• 45:15
  how detailed all of these captions are.
• 45:17
  Like are you getting things down to the level of types of
• 45:21
  material and style or is it just coffee table?
• 45:29
  For Dylan. Sorry, can you repeat that
• 45:32
  question? You were looking at the chat
• 45:34
  huh? You were you were busy or you
• 45:36
  were re watching the demo thinking damn this is a really
• 45:39
  good product that I made How? How?
• 45:44
  How do you know me so well? That's well played.
• 45:48
  So, so basically how much detail do you get from those different
• 45:55
  captions or whatever it's being labeled?
• 45:59
  Is it going down to the like hey this is a postmodern coffee
• 46:04
  table made out of glass or is it just like coffee table?
• 46:08
  Yeah. So it really depends on like you
• 46:12
  know, the image to text model that you you want to use.
• 46:17
  So I use SIGLIP 2, if I remember correctly, which is a very like,
• 46:23
  you know, basic like open source model that can that can run on
• 46:27
  your on your device. And so that model will usual.
• 46:30
  And I did not try to ask for much in depth definition.
• 46:36
  You know, I was just looking for coffee table table here.
• 46:39
  So maybe that model has the capability to have a more in
• 46:42
  depth definition. But if not this one, I'm sure
• 46:46
  that, you know, image to text models are very good these days.
• 46:50
  And if you want a really like detailed explain the issue of or
• 46:55
  like description of the object, I'm sure this is something that
• 46:58
  can be done. Nice.
• 47:02
  So we've got, I want to bring us back a little bit to the topic
• 47:09
  du jour, which is a genetic retrieval.
• 47:13
  We were going hard on that and the chat was loving it.
• 47:17
  And then we also, the chat was asking for some demos and we got
• 47:20
  some cool stuff. But there's a lot that we can
• 47:25
  talk about still with agentic retrieval.
• 47:28
  And so maybe we can veer in that direction.
• 47:34
  Neil, I feel like you have some things you wanted to say.
• 47:39
  Yeah, sure. I can.
• 47:41
  You know, we got like roughly 10 minutes left.
• 47:44
  And so as we center back on agentic retrieval, some general
• 47:52
  things that I think the audience should, you know, some ground
• 47:56
  setting, ground levelling things that I think the audience should
• 47:58
  all be aware of with vector search is that even whether it's
• 48:05
  pertaining to retrieve a genetic retrieval or just any retrieval,
• 48:09
  it's not only semantic search. And this goes back to the Neo 4
• 48:12
  J and graph Rag kind of question too.
• 48:15
  But vector search is really vectors and embeddings are going
• 48:21
  to be the highest density format in which you can store
• 48:23
  information. You're taking in dense
• 48:27
  embeddings, you're taking tons of unstructured data.
• 48:29
  And yes, you're storing that into a fixed dimensional vector,
• 48:33
  but there's also sparse embeddings.
• 48:35
  There's also, there's different embedding models.
• 48:37
  All vectors are is a format and you as you use different
• 48:42
  embedding models for images and videos and memory conversations
• 48:47
  and all this like embeddings are just a way, they're a vehicle
• 48:50
  for this. And we can do keyword search and
• 48:55
  lexical search and we can do all these different exotic things or
• 48:59
  maybe exotic strong words, just advanced skilled ways to find
• 49:03
  the needles in the haystack. And that becomes really
• 49:08
  important for agentic retrieval because just centering back on
• 49:12
  agentic retrieval, you want to have access to information in
• 49:18
  the most efficient way. And we, we have looked at file
• 49:22
  search and other approaches and there are places for file
• 49:24
  search. There is not necessarily you
• 49:26
  must use vector search for everything.
• 49:30
  You must set up an embedding model and you must set up your
• 49:32
  chunky. You must do that for everything.
• 49:34
  It may not make sense in certain scenarios, but when you as we
• 49:39
  start collecting more data in our, our agents become more
• 49:43
  sophisticated. Are their search tools become
• 49:48
  better? I think that vector search has a
• 49:51
  really strong place there. And I also like intentionally
• 49:54
  use that term vector search instead of vector database
• 49:57
  because you want a system that optimizes for not just storing
• 50:04
  and not just storing and having like a blind retrieval method to
• 50:09
  get your vectors out. You want to actually have
• 50:12
  something that is focused on the retrieval side of the equation,
• 50:17
  the search side of the equation. So like that's why I quadrant we
• 50:20
  consider ourselves a vector search engine.
• 50:22
  But yeah, I'll stop kind of rambling there.
• 50:24
  I don't know if anyone else wants to add thoughts to where
• 50:28
  they see vector search fitting into agentic retrieval overall.
• 50:35
  I can jump in on the scale. So Neil mentioned how like we're
• 50:41
  operating at millions of billions of of vectors scale and
• 50:45
  search will certainly get much, much bigger.
• 50:48
  So early search primitives like the BM20 fives of of the world
• 50:52
  are already 5 orders of magnitude smaller in terms of
• 50:57
  memory that they consume on the machine with the latest one.
• 51:01
  And another research that I wanted to share with you comes
• 51:05
  from the group research group called said one.
• 51:10
  They have a really interesting approach and they think that
• 51:13
  reinforcement learning for identic retrieval is the way to
• 51:18
  get to that next 5 order of magnitude efficient identic
• 51:25
  retrieval. When the context window gets
• 51:27
  bigger, when the number of vectors that we're have to parse
• 51:30
  through, it gets much, much bigger.
• 51:32
  As I'll share the link so you can read into their approach and
• 51:37
  why reinforce reinforcement learning exactly.
• 51:41
  Yeah, I know, Jenny, you had mentioned that earlier too
• 51:43
  maybe, but can you talk a little bit more to that because I like
• 51:50
  this idea. We were talking about RL 2 for
• 51:52
  evals and being able to create these environments, which are
• 51:59
  super trendy these days, I guess you could say.
• 52:04
  And so you have these environments and then you can
• 52:07
  eval them. But the RL for retrieval, what
• 52:10
  is that even like? Break that down for me a little
• 52:13
  more. Sure.
• 52:14
  So what I got out of that paper is that the TLDR is basically
• 52:20
  they're using a multi turn RL and it's a mixture of synthetic
• 52:24
  and real questions. So Jenny mentioned before the
• 52:27
  golden data set and the reinforcement learning approach
• 52:32
  would be not that we have like human generated golden data set,
• 52:37
  but it is the environment where you learn, you interact with
• 52:42
  over time with some real questions that resemble that
• 52:47
  golden data set. And some of them are completely
• 52:50
  AI generated. And there needs to be a reward
• 52:54
  system designed in place that rewards every single time that
• 52:59
  our agentic system hits the correct answer and punishes it
• 53:03
  whenever the learning step was not made correctly.
• 53:06
  So the paper outlines how you can design those, but I, I won't
• 53:11
  be diving too much in the detail there.
• 53:14
  So that's the, that's the high level overview of what exactly
• 53:20
  they're doing there. Yeah, that is so cool to see and
• 53:25
  I feel like there's so much potential there.
• 53:28
  The thing that I always wonder, and I also see that folks in the
• 53:32
  chat are asking this, it's more along the lines of like, hey,
• 53:37
  I've got this agent. How do I either set up a
• 53:41
  framework or encourage it to do different kinds of searches or
• 53:50
  search techniques at different times?
• 53:52
  So I want to do some kind of semantic search when there's
• 53:56
  XYZ, like is there a framework? Is there a way that is it a
• 54:00
  skill that we were talking about earlier that you can say like,
• 54:03
  hey, agent, here's whenever you need to retrieve something,
• 54:07
  here's what you should go through.
• 54:09
  And that feels like probably the easiest.
• 54:12
  When I see Jenny, you're shaking your head.
• 54:14
  You might have some thoughts. No, because I think it's
• 54:18
  exactly, you answer the question, there are two ways,
• 54:20
  right. There is like the way zero shot
• 54:23
  way you teach it procedurally, which is like the set of skills,
• 54:28
  the set of search skills and it has its pros because it's easy
• 54:31
  enough and you can adopt it easily enough because it's just
• 54:35
  writing. I mean it's not just, but it's
• 54:38
  composing the set of instructions and we kind of know
• 54:41
  how to approach that. And the second branch that we
• 54:44
  are seeing now emerging is to actively teaching agents to
• 54:49
  search through this reinforcement learning
• 54:51
  environments. I don't see now easy ways of any
• 54:56
  practitioner to do that, but I feel like that might be the next
• 55:01
  thing in the future because everything becomes more
• 55:03
  accessible now, right. So maybe we will get our Aral
• 55:07
  gyms. Do you see my biceps?
• 55:10
  Yeah, I'm saying the Aral Gym, but we will get them soon where
• 55:14
  you can actually give the set of what you want to achieve and the
• 55:19
  set of the search tools and your problem and it will converge to
• 55:23
  the like the small search agent will be able to converge to a
• 55:27
  path which actually is, well what you want to see in your
• 55:32
  specific pipelines. We're going to stay and watch
• 55:36
  that because we represent kind of the default layer that humans
• 55:42
  or agents build upon. I wouldn't discard humans steal
• 55:45
  from all of this picture, by the way, because I'm human.
• 55:49
  So I still, I still sometimes want to search stuff myself.
• 55:52
  So I think it's just important to have the tooling which will
• 55:57
  be usable by all the categories of the search users.
• 56:02
  I think it was a tautology, but you get me I guess.
• 56:05
  And I think what we were also trying to say that vector search
• 56:10
  is also so much more than just retrieving you a semantically
• 56:13
  similar facts. I feel like we kind of fell into
• 56:17
  the strap of the rag cage where you just think about it as the
• 56:22
  machine that spits out you the text chunk based on your
• 56:26
  question. But I think there are other
• 56:29
  emerging interesting parts where it could be used for anomaly
• 56:34
  detection, data analysis at scale, image to text search,
• 56:38
  videos to whatever audios, so all of that stuff.
• 56:43
  And at some point agents will be also able to use it 3D.
• 56:46
  And I'm really looking forward to see how they're going to
• 56:49
  approach this part of vector search.
• 56:52
  Well said. I think that is a perfect way to
• 56:58
  wrap it up. I know that there are still
• 57:01
  folks in the chat that are stoked and asking questions.
• 57:05
  I will just mentioned that everyone who is in the chat and
• 57:10
  watching this right now, on your left hand sidebar there is a
• 57:16
  button that you can press that says match.
• 57:19
  If you go there and more than two people do it, you will
• 57:24
  randomly get put together with somebody else that is watching
• 57:28
  this. So you can meet someone that is
• 57:31
  just about as crazy on retrieval as we are, and you can talk to
• 57:38
  somebody that is watching this too.
• 57:40
  So it's a great way to bond with the rest of the community if you
• 57:43
  want to stick around for more time.
• 57:46
  But for this session, I think we are going to wrap.
• 57:50
  I know that there are some great places the Quadrant team hangs
• 57:54
  out. You all have an awesome Discord,
• 57:57
  so I'll drop a link for that in the chat.
• 58:00
  And then of course, if you weren't already bought into
• 58:07
  everything that quadrants do in and following these good folks
• 58:11
  that are here with us today, you should definitely do that.
• 58:15
  Go and give Quadrant a star if you haven't already on GitHub
• 58:21
  and stay up to date with what all of their amazing Devrel is
• 58:27
  doing. Follow them on LinkedIn and X
• 58:29
  and all the places. So thanks everyone for this
• 58:33
  excellent session. I will bid you farewell.
• 58:37
  And for those that want to hang around, click the match button
• 58:41
  on the left hand sidebar and I'll see you all later.
• 58:46
  Thank you. Thanks everyone.
• 58:47
  Thank you. Thanks to meet you.
• 58:49
  Bye.