Latest / Agentic Conversations (formally mlops.community) / AI Is Fast. AI Projects Are Slow. Let's Fix That.

Transcript

AI Is Fast. AI Projects Are Slow. Let's Fix That. · Agentic Conversations (formally mlops.community)

• 0:00
  We care a lot about saving people money.
• 0:02
  Over the year that this pipeline is going to be up, if I can, you
• 0:05
  know, cut my Lol costs in half the.
• 0:07
  Cost savings thing is huge man. Oh, that was cool.
• 0:15
  You missed that that. Was a good one, huh?
• 0:17
  That was a good one. Very.
• 0:18
  Cool. So I think it's really important
• 0:23
  to get the overall context of what we're talking about here.
• 0:27
  So, you know, you could say software engineering is dead,
• 0:30
  right? And while it's not really dead,
• 0:32
  certainly the old definition of software engineering is quickly
• 0:36
  evolving away. And what we see is the need for
• 0:41
  developers to transition to this new way of working.
• 0:44
  And I like to just give a very simple example.
• 0:47
  You know, Company A adopts efficiency AAI which is doing
• 0:52
  the same with less. So what happens there?
• 0:53
  They lay off 50% of their developers to maintain their
• 0:56
  current revenue, current growth path.
• 0:59
  Company B adopts opportunity AAI, which is doing more with
• 1:03
  the same. So what do they need to do?
• 1:04
  They need to keep their developers quickly retrain them
• 1:07
  in this new way of working so that they can produce 2X4 X 10X
• 1:12
  and that 10X advantage in the top line revenue for that
• 1:15
  Company B ends up compounding over time.
• 1:18
  And you can see where this goes between those two companies.
• 1:21
  So while, well, right now it's a little more short sighted with a
• 1:25
  lot of efficiency AI going on in the industry and a lot of these
• 1:28
  either real AI layoffs or AI washing, it doesn't really
• 1:32
  matter. It's imperative that developers
• 1:35
  take those steps to build their AI expertise.
• 1:38
  And this is where really Rocket Ride sees an opportunity to help
• 1:42
  those developers get up to speed, get productive building
• 1:46
  AI solutions, AI workflows, AI automation, you know, fast.
• 1:52
  So yeah, tell me about how you feel the world of a software
• 1:56
  developer is changing. I imagine there's just now we're
• 1:59
  using a lot of coding agents and a lot of the code isn't being
• 2:02
  written by actual humans, it's being written by agents.
• 2:05
  And we're kicking off sub agents or we're kicking off parallel
• 2:09
  agents. There's long running tasks, all
• 2:11
  of that. What does that entail now?
• 2:14
  In what have you been seeing out there?
• 2:17
  Well, I think in the context of Rocket Ride it really, we really
• 2:20
  acknowledge that coding is no longer the bottleneck.
• 2:23
  Code generation is no longer the bottleneck, which it used to be.
• 2:27
  So the bottleneck just move somewhere else, right?
• 2:30
  Whether that's intentionality, whether that's tool discovery
• 2:34
  and selection, whether that's quality and post production, The
• 2:39
  the whole idea here is that there's new roles for to be
• 2:43
  filled by the developer and those are painful steps that can
• 2:48
  be painful certainly for the inexperienced.
• 2:52
  I like this intentionality piece.
• 2:54
  Can you talk to me more about that?
• 2:56
  What I've noticed about, you know, the development process,
• 3:00
  my development process in the last, you know, a couple of
• 3:04
  years using coding agents and all that kind of stuff, it's
• 3:07
  really quick to actually develop, you know, code, but
• 3:11
  it's also really quick to develop crappy code, right?
• 3:15
  And you know, a lot of times even I love cloud AI.
• 3:19
  I mean, it's, it's a really, really good coding agent from my
• 3:23
  point of view. Sometimes it gets stuck in these
• 3:26
  loops and sometimes it doesn't take into consideration good
• 3:29
  engineering practices. It writes not the best
• 3:33
  algorithms in the world. And then you spend, you know, an
• 3:36
  hour trying to figure out, oh, you made a mistake on step A,
• 3:40
  you're in on step N, right? And then it forgets everything
• 3:43
  that you've done. And guess what?
• 3:45
  You end up with spaghetti code that's not worth actually
• 3:48
  shipping. And to try and get it into a
• 3:50
  shipping state is very, very difficult at that point in time.
• 3:54
  Yeah. We're just talking about how
• 3:56
  compact is not the best when you get the context and you need to
• 4:02
  compact it. And so there's a lot of hacks
• 4:05
  right now that folks are doing one being just put it into a
• 4:08
  document so that you have your whole planning session or
• 4:15
  whatever it is that you're trying to do.
• 4:16
  The To Do List etcetera etcetera is on a document and then you
• 4:19
  don't have to lose the context when you compact it.
• 4:24
  And if you start a new session, you can reference the document
• 4:27
  always. That does help a lot.
• 4:29
  You know, one of the the keys to actually making Quad successful
• 4:32
  is a good plan, you know, giving it access to the full plan
• 4:36
  upfront. And when you do that, it
• 4:39
  actually comes out with better code.
• 4:41
  But that implies that it's not great at iterative, you know,
• 4:43
  engineering because it doesn't necessarily remember what it did
• 4:47
  three days ago. And this module over here where
• 4:50
  this is reusable, you know, I I told it to go out and build a
• 4:53
  UI. I ended U with two or three
• 4:55
  different C style sheets, which were all common.
• 4:59
  And it's like, you know, good engineering ractice create a
• 5:01
  common C, you know, and move forward and use that, you know,
• 5:05
  for theming and all that kind of stuff.
• 5:07
  Claude's not really good at that.
• 5:09
  Claude is looking, you know, really good at, you know, taking
• 5:12
  a particular task you know, or plan and implementing that plan
• 5:16
  regardless of whether you know you have 500 files in there.
• 5:20
  It's not looking for all the CSS's and all the common styles
• 5:23
  on what the background paper looks like.
• 5:26
  Yeah, you find that cloud is a little bit lazy.
• 5:30
  Yes, well, I fight with Claude quite a bit actually.
• 5:34
  You know, it's just like, no, I don't want it done that way, you
• 5:36
  know? OK, look at common dot CSS and
• 5:39
  figure out, you know, what are the common styles here?
• 5:42
  How do you apply themes? Why are you doing it this way?
• 5:44
  Not this way when you did this way yesterday, I mean.
• 5:46
  It's simulating a developer, so of course it's lazy.
• 5:50
  That is true. You want the more lazy the
• 5:53
  better if you're a developer. But it is relentless, so a lot
• 5:57
  of times it can get there. It just means that you've got to
• 6:01
  burn a bunch of tokens. Oh yeah, I'm in trouble for
• 6:07
  that, Yeah. Yeah, I can believe.
• 6:08
  It, yeah, it's I So you know, from a Rogrammer's standpoint,
• 6:12
  you know, I started in, you know, assembly language Z80,
• 6:16
  believe it or not, Z80 assembly code if anybody reckons
• 6:18
  remembers AZ 80. This was like an 8 bit
• 6:21
  microprocessor way back when. And then I went to BASIC and
• 6:24
  then I went to, I did some dappling in COBOL and Fortran
• 6:27
  and then all the way up, finally ended up in JavaScript and
• 6:31
  TypeScript about 3-4 five years ago.
• 6:33
  And this AI evolution with Claude, you know, and Claude
• 6:37
  code, I just feel there's another, you know, language for
• 6:40
  me to actually interact with. I don't care about, you know,
• 6:43
  what kind of code it produces underneath what language it is.
• 6:47
  I want to interact at that top layer, but I also wanted to
• 6:50
  write quality code. If you're going to pick
• 6:52
  TypeScript, fine with me. No problem.
• 6:54
  Pick TypeScript, but write good TypeScript.
• 6:58
  So what you've been seeing out there as far as the
• 7:01
  difficulties? From my point of view is the
• 7:04
  ambiguity, if you know, in the bridge between the English
• 7:07
  language, I program in English obviously, and the technical
• 7:10
  jargon about, you know, structure and creating good
• 7:14
  engineering principles underneath, you know, it's like
• 7:16
  a good architecture. How do I actually express that
• 7:19
  to Claude? What is a good architecture?
• 7:22
  Because that's a very subjective, you know, point.
• 7:25
  What is a good architecture? You know what is good?
• 7:28
  Structure definition and reusability and component
• 7:31
  reusability, all those kinds of stuff.
• 7:33
  So if you look at my memory dot MD, it's probably like that long
• 7:38
  because I specifically go on and say I like this, I like this, I
• 7:41
  like this and that. Does help primarily letting it
• 7:45
  live to help you create that solid architecture?
• 7:48
  Consistency. Yeah, yeah.
• 7:50
  So I'll, I'll say, you know, in my memory, I'll say go out and
• 7:53
  look at common CS, you know, CSS, look for styles, you know,
• 7:57
  look for this particular architecture.
• 7:59
  I like, you know, a snake case and TypeScript versus, I'm
• 8:03
  sorry, camel case and TypeScript snake case and Python, you know,
• 8:07
  and and so on and so forth. So I have all these rules to
• 8:11
  assist clotting in some kind of, you know, semblance of
• 8:14
  organization. And it's like, you know, I, I,
• 8:18
  we have two, we have in Rocket ride, we have two SDKS.
• 8:21
  We have one in client of TypeScript, one in Python,
• 8:25
  right. And so I'll say, OK, make this
• 8:27
  change in the, the TypeScript SDK, OK.
• 8:31
  And it doesn't, even though in my memory, I've told it, you
• 8:34
  know, when you update one of the SDKS, we try and keep them In
• 8:37
  Sync. So update the, you know, Python
• 8:40
  And then it will go into the Python And, you know, do a camel
• 8:43
  case because that did it with, you know, TypeScript.
• 8:45
  But that's not standard Python. So it's like you always have to
• 8:49
  check every line of code, you know, to make sure that it's
• 8:53
  actually, you know, based on good engineering principle.
• 8:56
  Yeah, I guess the question that I am constantly asking myself is
• 9:01
  if it runs, even if it's not pretty, does it matter?
• 9:08
  That's a great question. I triggered you both there a
• 9:11
  little bit. Yeah.
• 9:13
  You know, if you're a software developer and you care about
• 9:17
  your craft, then you want to know that it's architected for
• 9:23
  maintainability, readability, reusability.
• 9:27
  If you're a vibe coder, then your judgement is, does it do
• 9:31
  what I want it to do? Great.
• 9:33
  So it really depends on are you a citizen developer or are you a
• 9:37
  software developer, right? But at the same time, think
• 9:42
  about all the infrastructure code that you have to go through
• 9:47
  and build each time you're building an AI application.
• 9:49
  Do you really want to reinvent that wheel over and over and
• 9:53
  over again? Rely on the particular version
• 9:56
  of the particular coding agent that you're using.
• 9:58
  And you know, God forbid if you change coding agents and they do
• 10:02
  something different, you know, these coding agents are trained
• 10:04
  off of publicly available source code.
• 10:07
  Not all of that code is the best architecture code in the world.
• 10:10
  So what you're doing is you're basically risking that you're
• 10:13
  gonna have different implementations of that
• 10:15
  infrastructure code at each time you build an AI application.
• 10:18
  That's really one of the foundations for why we decided
• 10:21
  to build Rocket Ride is let's go ahead and design A framework
• 10:26
  that standardizes that infrastructure tier and that
• 10:29
  glue codes, which is kind of almost a waste of time anyway,
• 10:32
  because it's not really tied to the value of what you're trying
• 10:35
  to build the the solution. It just is necessary.
• 10:38
  So we've abstracted that away and, and done it in such a way
• 10:42
  that it's robust, scalable and standardized.
• 10:46
  And that's one of the aspects of, you know, what motivated us
• 10:49
  to build this thing in the first place.
• 10:51
  Yeah, actually we had a Lunch and Learn last week and it was
• 10:54
  all about skills. And one of the key learnings
• 10:57
  from a paper that one of the members of the Lunch and Learn
• 11:03
  read was you have to build the skill for the model that you're
• 11:07
  using, because different models prefer skills in different ways.
• 11:11
  And that was new to me. I didn't realize that you want
• 11:14
  to have it. That kind of specificity.
• 11:16
  I probably should have known for the exact reasons that you're
• 11:20
  saying, like each model is trained in its own way and it
• 11:23
  has its own particularities, but just doing that, like it gives
• 11:29
  you so much lift. If you think about, OK, I'm
• 11:32
  using codecs or I'm using Claude, and this is how Claude
• 11:37
  skills work, this is how codec skills work, whatever it may be,
• 11:41
  then you can get a little more lift.
• 11:43
  Are you thinking in those terms? The way I see it is we've taken
• 11:48
  away some of that open judgment that the particular agent will
• 11:54
  will use to kind of follow a certain pattern.
• 11:57
  And I'll give you a practical example of why that's relevant.
• 12:01
  You've built you as a software developer, have built a solution
• 12:04
  for the finance department that does XYZ.
• 12:07
  And now the legal department wants a very similar solution,
• 12:11
  but it's a slightly different use case and it's a different
• 12:13
  data set. And now you go and you tell your
• 12:16
  coding agent without our framework and you say, OK, now
• 12:18
  modify that. You know, take that application
• 12:21
  that I just built as a base and now modify it for this, this,
• 12:25
  and this. It ends up going and changing
• 12:27
  things that you didn't want it to change.
• 12:29
  And by keeping it following these certain patterns for the
• 12:33
  stuff that isn't relevant to the differences between the two, it
• 12:37
  enables you to have more reliability.
• 12:40
  Hey, I've already got that thing running in production.
• 12:42
  It's already working. Why do I need to take risk in
• 12:44
  areas I don't need to take risk in for things that don't matter
• 12:48
  for that solution? So that this is an example of
• 12:50
  why that's valuable from my perspective.
• 12:52
  So I don't think it's so much that it's agent specific as it
• 12:57
  is whichever agent is coming in. It's not really a skill so much
• 13:00
  as these are the rules. These are the patterns that you
• 13:03
  need to follow to build solutions in our tool.
• 13:07
  So you're defining in the framework.
• 13:09
  These are the important things. These are the rules and this is
• 13:12
  how I wanted to act and that can't be changed as you are
• 13:18
  changing the agent. Well, there's a lot of
• 13:20
  flexibility to customize, to build new nodes, to compose
• 13:26
  solutions and pipelines the way that you need to.
• 13:29
  But there's like a some fundamental primitives that are
• 13:35
  there for a reason. As we all know, agents work
• 13:37
  differently, right? A career AI agent works much
• 13:40
  differently than deep agent, which works much differently
• 13:43
  than lane chain agents, right? I mean, it's, and how they
• 13:46
  interpret, you know, the, the context that you give it and how
• 13:51
  they interpret the question and the ultimate goal is very
• 13:53
  different. Not only that, you can take a
• 13:56
  deep agent, for example, and give it two different LLMS. You
• 13:59
  know, cloud AI and you know, and, and you know, Chad GBT, you
• 14:04
  know, it works differently. You know, and not only does it
• 14:08
  not work the same way, but a lot of times you get different
• 14:12
  results every time you call it. Because you know, the, the LLMS
• 14:18
  are free to go out and do whatever they want to do when
• 14:20
  you have to actually work with what they give you.
• 14:24
  So The thing is, is, is what we did is, is we really wanted to
• 14:28
  be able to compare agents and their performance.
• 14:31
  And then each agent and all of its tools can use different
• 14:34
  LLMS. So bring the correct LLM to the, you know, to the job.
• 14:38
  So let's say a tool requires an LLM and that tool is very, very
• 14:42
  good at reading tables out of PDFs, for example, right?
• 14:46
  Bring the right tool to the job where the main orchestrator may
• 14:49
  actually require more intelligence.
• 14:51
  So it goes with quad or, you know, chat DBT where somebody
• 14:57
  else may want to go with, you know, Gemini is something,
• 15:00
  right? It's so bring the right tool to
• 15:02
  the job. And then one of the things that
• 15:04
  I really struggled with as a programmer when I first started
• 15:07
  doing this is OK, do I use lane chain or do I use Haystack or do
• 15:11
  I use Llama? Would it Llama index?
• 15:15
  It's like, OK, which one do I use?
• 15:16
  Well, which one is better? I don't know.
• 15:18
  So it's that trial and error that is so time consuming,
• 15:23
  right? So with what I wanted to bring
• 15:27
  the tool to is plug in anything you want.
• 15:31
  They're both all given the same inputs.
• 15:34
  Plug in what you want, compare the performance and you can
• 15:37
  literally do that in 2 minutes and see how deep agent actually
• 15:40
  performs on something versus crew AI.
• 15:44
  Now, once you do that, you pick your best agent.
• 15:48
  Well, does this work OK with sonnet or does it work OK with,
• 15:52
  you know, GPT? Does it work with GPT 3.5 Turbo?
• 15:55
  Because that's cheaper for me to run.
• 15:57
  So not only can you turn the agents, but you can turn the
• 16:00
  LLMS and do direct comparisons in your pipelines to see exactly
• 16:04
  which is the best for your solution.
• 16:06
  And are you running these pipelines various times to be
• 16:09
  able to say, OK, this one? They're all run simultaneously,
• 16:13
  so you can give it an instruction and it can send it
• 16:16
  over to, you know, crew AI and let crew AI do its work.
• 16:19
  And then you can get the results from crew AI and then it will do
• 16:22
  it to the line chain and then it will do it to deep agent, give
• 16:26
  you all the results and say, oh, this one sucked at it, so we're
• 16:29
  not going to use that one. Let's replace that one.
• 16:31
  Try different LLLM in it, run it again.
• 16:34
  Boom, you've got your results all on the screen.
• 16:37
  Wow, that gives you a lot more insight on to is it me or is it
• 16:41
  the pipeline, is it the agent, is it the LLM?
• 16:45
  What is it exactly that is not working here when?
• 16:48
  Somebody then if you really want to dig down into it, there's a
• 16:51
  full trace on exactly what agent did what, what the agent was
• 16:55
  asked, the results from the agent through you know what the
• 16:58
  the tool was passed what the tool results came back.
• 17:01
  So you can trace through each different logical step that it
• 17:04
  took through, went through to, you know, end up at your final
• 17:08
  answer. It's that full observability to
• 17:10
  recognize where things are falling over.
• 17:13
  What is the best agent for the job?
• 17:16
  Yeah, OK. Talk to me more about the
• 17:19
  primitives and how you break this down.
• 17:22
  I mean, we have many, many different, what we call nodes,
• 17:25
  OK? Nodes are like, you know, taking
• 17:28
  a piece of text and chunking into documents, you know, and
• 17:31
  you can do it many different ways.
• 17:33
  There's a code, a code chunker that actually recognizes C and
• 17:37
  JavaScript functions and all that kind of stuff.
• 17:40
  There's a regular text parser that, you know, chunks text
• 17:43
  based on paragraph or sentence for however you want to do it.
• 17:46
  Then there's things like, you know, vector stores.
• 17:49
  We have I think support for eight different vector stores
• 17:52
  right now, The most popular ones, we have an
• 17:55
  Http://requester that you can go out and you know, get stuff from
• 18:01
  the from the the web. We have a support fire caller,
• 18:06
  what, 12 different LMS at this point in time.
• 18:09
  So I mean each one you build up these chains exactly what you
• 18:12
  want it to do. I have data I want it sent to
• 18:14
  here, I want it sent over here to do parsing on it.
• 18:17
  I want the text, I want to take that text and run it through a
• 18:19
  summarization engine. But before I do that, I want to
• 18:23
  do an anonymization on it so it doesn't have any PII in it.
• 18:26
  Then I want to chunk it up and then I want to put it into a
• 18:29
  vector store. So you get to build up these
• 18:31
  pipelines. You know, we basically look at
• 18:34
  it as, you know, a box of Legos. You know, a little tiny red Lego
• 18:38
  is not interesting, right? A white Lego, you know, 4x4 Lego
• 18:42
  is not interesting. But when you start building
• 18:45
  stuff out of these Legos, that's when it becomes interesting.
• 18:48
  And that's what we're really focused on.
• 18:50
  You know, I have been constantly thinking about how much of the
• 18:56
  pipelines do I need to define versus letting one of these
• 19:04
  agents just figure out what the best way of going about it is if
• 19:08
  I give it the boundary conditions, right?
• 19:11
  I don't know if you've played around with that at all.
• 19:13
  As to I want to clearly say this is the workflow because I know
• 19:18
  what the workflow is and I plug in the different parsers.
• 19:22
  I give it that LLM brain on certain steps, but at the end of
• 19:26
  the day it's a bit of a DAG. That's that's exactly what we do
• 19:31
  OK. We standardize something called
• 19:33
  on what we call lanes and lanes. We have a right text lane, a
• 19:37
  right table lane, a right question lane, right video lane,
• 19:41
  audio and image. And based on the kind of data
• 19:44
  that's going through it, it flows through these nodes and
• 19:47
  you can do different things. You can do OCR, which takes an
• 19:50
  image and outputs to the text lane.
• 19:53
  Then from the text lane, you can take that and plug it into an
• 19:55
  anonymization text lane, which produces more text, but it's
• 19:59
  anonymized text. Then you can then chunk into
• 20:02
  documents if you want. So it's basically that plumbing
• 20:06
  between the two things is the is the boring stuff, right?
• 20:11
  And it's the stuff that LLMS, you know, Claude typically get
• 20:15
  wrong. It's the plumbing.
• 20:16
  It's not the actual, you know, AI calls, it's the plumbing and
• 20:20
  getting everything to work together that, you know, these
• 20:23
  coding agents usually screw. Up.
• 20:25
  Yeah, Talk to me more about that because I want to just say give
• 20:30
  it to Claude, let Claude figure it out on its own.
• 20:34
  Yes. So with we've created this
• 20:38
  complete set of documentation that you can do two things with.
• 20:41
  Claude can actually build custom nodes for you to accept the text
• 20:46
  input and do whatever you want and send it out.
• 20:49
  That's one little Lego, right? But the more interesting case is
• 20:53
  I want to parse a PDF and anonymize it.
• 20:56
  And, you know, I'll put it to my text lane.
• 20:59
  OK, return the answer. Claude will build that for you.
• 21:02
  It'll put all the Legos together because it completely
• 21:04
  understands exactly what each one of these nodes do.
• 21:08
  So you don't even have to go out and think about I need to
• 21:10
  connect this and this and this. Claude knows how to do it
• 21:14
  because we've instructed it all about these different nodes and
• 21:18
  how they work and interoperate together, and it puts it
• 21:21
  together. OK, so the different nodes are
• 21:26
  very understandable by Claude code or by whatever coding agent
• 21:31
  you're using and you're giving it simple framework to leverage
• 21:37
  when it wants to create these workflows.
• 21:39
  Right. And the reason why it's easy is
• 21:41
  because we've standardized on these lanes, right?
• 21:43
  It's very easy for Claude to understand.
• 21:45
  This is the text lane. When you have text, run it
• 21:48
  through here. This is the image lane.
• 21:50
  You want to do OCR on it, run it through this, right?
• 21:54
  So, Claude. Those are in simple terms that
• 21:56
  Claude usually gets pretty darn accurate on.
• 21:59
  Yeah. And we keep saying Claude, but
• 22:01
  any coding? Agent, right.
• 22:02
  OK. So I use Claude.
• 22:04
  I'm sorry. Yeah.
• 22:05
  So these these three elements, right, the agents MD, which kind
• 22:08
  of defines the rules and the instructions, the documentation
• 22:12
  and then the source code itself, right.
• 22:14
  And with that, like you said, you know, we've actually given
• 22:17
  natural language instruction for the agent and watched it compose
• 22:22
  these relatively complex, I think 45 node pipelines to
• 22:27
  accomplish the the objective, right?
• 22:29
  So, I mean, that's where we're going anyway, right?
• 22:32
  And how how detailed are you in explaining in English each step
• 22:39
  versus just explaining what you want at the end?
• 22:43
  And then Claude figures out I need to create this 45 nodes to
• 22:46
  get to that end state. Claude is good enough and our
• 22:50
  instructions are good enough. I have APDF give me the text
• 22:53
  back. OK, I have I have an invoice
• 22:57
  that I want to stick into a my sequel table.
• 22:59
  Here's the table name and it'll figure out exactly what what it
• 23:02
  does. Because we have a My Sequel
• 23:04
  connector, we have a table recognizer extractor and puts
• 23:08
  this whole pipeline together to actually take a printed invoice
• 23:12
  that you get on a piece of paper, send it through the OCR,
• 23:15
  and then actually put it out into a database.
• 23:18
  And am I plugging in I can bring my own OCR model?
• 23:21
  Yep. And then you're giving me that
• 23:23
  observability on top. So if the OCR model I can test
• 23:26
  them in different ones. I can see this one actually
• 23:30
  gives me better results. Yes.
• 23:32
  And then how are the evals being done?
• 23:34
  Well, one of the things that I did which was really interesting
• 23:38
  pipeline that I wrote is that I used for actually testing OCR is
• 23:43
  I had an image, right? And then I on the, we call it
• 23:47
  the dropper and you basically just drop it on an image file
• 23:50
  and it sends it through the pipe.
• 23:52
  I had all my different OCR models, right?
• 23:55
  And with those OCR models, they all have text output.
• 23:58
  I took those text output, put them into an LOM and say which
• 24:02
  is the best 1. So it ranked all the different
• 24:06
  models based or the the different OCR models that I had
• 24:09
  based on my inputs so that I could get an impartial
• 24:12
  judgement. OTROCR is better on this, you
• 24:15
  know, on this particular data or easy OCR is better on this or
• 24:19
  you know, a handwritten is a completely different thing,
• 24:22
  right? But that's but that's what you
• 24:24
  can actually inherently test in like 2 minutes.
• 24:28
  And you can do it for every use case, which I hadn't thought
• 24:30
  about because there's certain use cases that are going to be,
• 24:33
  they're going to skew towards 1 model versus the other.
• 24:36
  Yes, especially on OCR. OCR.
• 24:38
  It has broad variances on how good it is depending on the
• 24:41
  model. But then you can extrapolate
• 24:43
  that out for any type of use case.
• 24:46
  So if it's the PII anonymizer model.
• 24:49
  You sentence transformer and better chunker parser, you name
• 24:54
  it. And at the end of the day, if I
• 24:56
  want to bring something, let's say that I'm using a paid tool
• 25:00
  like extend, I know they have a great OCR model.
• 25:04
  I can just bring that plug it in as one of those nodes, because
• 25:08
  the nodes, if I'm understanding correctly, you're basically not
• 25:11
  opinionated about. No, no, we don't care what node
• 25:15
  it is. As a matter of fact, we have
• 25:16
  nodes that we send data to. We send data to Google to do
• 25:20
  object recognition or you can run object recognition on site.
• 25:23
  We're not opinionated though. The whole concept of of rocket
• 25:27
  right is really pick the best tool for the job, right?
• 25:31
  And how do you actually prove that it's the best tool for the
• 25:34
  job is run it? Yep.
• 25:36
  Yeah. We don't care if it goes up to a
• 25:38
  website or whether you're running it locally or on Prem or
• 25:41
  wherever. Just, you know, be able to prove
• 25:44
  what's the best solution is. Yeah.
• 25:46
  OK. Well, this makes sense on what
• 25:49
  you're doing and how you're doing it.
• 25:50
  And then are there any other primitives that we should know
• 25:53
  about? Oh boy.
• 25:56
  One of my personal favorites is what we call the frame grabber.
• 26:00
  And a frame grabber is really cool because what you do is you
• 26:02
  feed video into it and we can detect scene transitions within
• 26:07
  a video. OK.
• 26:09
  And then we can tell you exactly the time that it, it actually
• 26:12
  did that. We can pull out, you know, every
• 26:14
  3rd frame or a frame every 15 seconds or five seconds or
• 26:18
  whatever. And then we can send those
• 26:20
  through object recognition and from object recognition, we can
• 26:24
  say yeah or no, whether this matches whether there's a blue
• 26:27
  car in there or not. And then we can actually pull
• 26:30
  highlight reels out of out of, you know, videos based on what
• 26:34
  the content is, send those frames to our assembler at the
• 26:39
  at the back end and get a full video with highlight reels, you
• 26:42
  know, for, you know, out of the big video.
• 26:45
  So I like the frame grabber. That's my favorite one that in
• 26:48
  transcription actually. Oh nice.
• 26:51
  Yeah, transcription is a huge one especially.
• 26:55
  I, like many other people, am addicted to the speak to text on
• 27:02
  your computer. I know a bunch of friends who
• 27:04
  are using local models to make that happen.
• 27:07
  I just pay for a service out there, but this is.
• 27:11
  So one of the nodes that we have right now is is speech to text,
• 27:15
  All right, That's that's, you know, one of the nodes and
• 27:18
  that's transcribed. So you put in audio or video.
• 27:22
  Audio is faster, a lot more compressed.
• 27:26
  You can put audio in and then it converts it to speech going on
• 27:30
  the text line. From the text line you can go
• 27:32
  and convert that into a question, issue that question to
• 27:35
  an LLM, and LLM gives an answer. Then we can send back the text
• 27:39
  to the answer and we're about ready to release right now is
• 27:42
  our text to speech. OK?
• 27:45
  So then we can convert it back into speech and then and then
• 27:49
  answer you so you can actually talk to your LLM and it'll talk
• 27:53
  back. It's not just a one way St.
• 27:55
  Yeah, why use Rocket Ride versus just using the native coding
• 27:58
  agent? I think we may be asking too
• 28:00
  much of, of our, you know, code Gen.
• 28:03
  AI to actually do because not only are we having to ask them
• 28:07
  to write our application for us or, you know, the user interface
• 28:10
  and then the engine, we're also asking them to figure out how to
• 28:14
  call and put all these different models together, right?
• 28:18
  And then what if I have to change a model 6 months from
• 28:21
  now? Do I have to go back in the
• 28:23
  clot? And we've already talked about
• 28:25
  how bad clot is it actually incremental stuff, right?
• 28:29
  So when you start making changes and maintaining that, it becomes
• 28:32
  much, much more difficult. Opens your your, you know,
• 28:35
  horizon for huge, you know, issues going forward.
• 28:40
  Not only that when you when you ask Claude, you have to be very
• 28:43
  specific if you taking care of all the async problems, the
• 28:46
  multi, you know, multi threaded problems, the multi processing
• 28:49
  problems, all these other things that you know are just nitty
• 28:53
  gritty details. Yeah, it works fine as a proof
• 28:56
  of concept on my laptop. Deploy that and guess what?
• 28:59
  It falls all over itself. Talk to me a bit more about
• 29:03
  those problems that you've seen happening and and then how you
• 29:06
  solve them. The solution to the problem is
• 29:08
  really a good architecture upfront, OK?
• 29:11
  It's like deciding these are how the lanes are going to fit
• 29:14
  together. This is how the components are
• 29:16
  going to fit together. This is their little piece.
• 29:19
  And then we control the outside of the piece.
• 29:21
  We give them everything that they need inside in order to do
• 29:24
  the task, but then we control all the outside so that we can
• 29:28
  run 256 threads concurrently without all the models stepping
• 29:33
  all over themselves, right? O you can send 60 chats, 60
• 29:37
  people can actually be chatting with this thing at the same time
• 29:40
  without it falling over those kinds of issues.
• 29:44
  A good architecture and actually disciplined engineering, you
• 29:48
  know, like I mentioned earlier, is absolutely critical for, you
• 29:52
  know, scalability, durability and maintainability upfront
• 29:56
  because of unless you design that upfront, you've got
• 29:59
  problems. You know when you start going
• 30:01
  into production, you're going to have multi thread issues.
• 30:05
  Then you have with Python. Python.
• 30:07
  Oh my goodness, sometimes I love Python, but I hate Python
• 30:10
  because they have the synchronous model and the async
• 30:13
  model and they too don't get along very well, right?
• 30:16
  So, you know, trying to explain to Claude, OK, this model like
• 30:20
  Crew AI Agent requires asynchronous, but yet this model
• 30:25
  over here requires synchronous. So now you actually have to say
• 30:29
  async dot run or async dot, you know, 2 thread in order to get
• 30:33
  it to actually, you don't interoperate with each other.
• 30:36
  Claude misses that a lot, I'll tell you that.
• 30:39
  And then six months from now you're going to go in and
• 30:41
  change, you know, your OCR to an asynchronous model.
• 30:44
  Good luck. Sounds like you've.
• 30:48
  I've been there. I can feel it in your voice.
• 30:51
  It's true. Let's talk a little bit about
• 30:53
  runtime observability and how that plays into it.
• 30:56
  There's different aspects to runtime observability.
• 30:59
  So for example, you've got a running pipeline, it's working,
• 31:02
  but you're not satisfied with the quality of the answers from
• 31:05
  the LLM. Why?
• 31:06
  What's going on? We, we enable that introspection
• 31:11
  at the input output of each node in that pipeline, what the LLM
• 31:16
  is doing, how they're talking to each other, what kind of answers
• 31:19
  it's getting. You might be looking at why is
• 31:22
  the latency on this particular agent so bad, right.
• 31:25
  You can dig in and understand where, where the latency is
• 31:27
  coming from. You can understand errors.
• 31:31
  So there's different ways in which we provide that level of
• 31:35
  introspection to enable you to quickly get to the root cause of
• 31:39
  what your problem is if there is a problem.
• 31:41
  When you talk about, you know, 10 or 15 components in a
• 31:45
  pipeline, the only thing that has to fail is 1 and your
• 31:49
  results sucks, right? That's that's the bottom line.
• 31:53
  A lot of times trying to determine which one of those
• 31:56
  components actually failed or didn't produce the results.
• 32:00
  Is it, you know, step in two or is it step 9 and being able to
• 32:05
  end it not only during development, going through and
• 32:08
  looking at everything that was done within the pipeline, but
• 32:11
  also in production too. You can keep all this, this
• 32:14
  logging information in production and somebody has you
• 32:17
  know, a problem and you know in production and says to me, well,
• 32:21
  this really screwed up. Well, let me go take a look at
• 32:23
  the log and see what the lol did here and you know what the
• 32:27
  transcriber said here? Oh, you gave it a bad image or
• 32:30
  something like that. So it's not only in development,
• 32:34
  it's actually more key on observability and production
• 32:38
  because that's when your failures are actually going to
• 32:40
  happen. And another aspect is cost
• 32:42
  observability, right? So as you run these AI enabled
• 32:46
  applications at scale, how do you know whether you know what's
• 32:52
  what's, what's costing you the most tokens, right?
• 32:55
  You can, you can dig into that. You can then fine tune and say,
• 32:58
  hey, you know, can we find a cheaper model?
• 33:02
  Am I running out of quota and why am I getting this error now?
• 33:05
  It's because you know, I've got my auto top off, you know,
• 33:08
  capped. That I, I've thought about a
• 33:11
  bunch because you can quickly get out of your depth,
• 33:16
  especially when you're with a team and you're thinking like,
• 33:19
  why did all of a sudden we are just like, our spend went up 20
• 33:24
  grand last week and you're thinking, I hope it was for
• 33:27
  something good. I, I hope it means we're doing
• 33:31
  way more productivity or something like that.
• 33:33
  But if you don't have that visibility into it, you can kind
• 33:36
  of be, you know, flying blind. Absolutely.
• 33:40
  You know, the, the ability and being able to look and see all
• 33:44
  the transactions, not only what the node did, but between those,
• 33:48
  why did this node go over here instead of over here?
• 33:51
  All that is really, really important, not just in in, you
• 33:55
  know, development, but in production.
• 33:57
  And how many production systems actually have VS Code debugger
• 34:00
  connected to it to find out exactly what happened.
• 34:03
  I can imagine a world where some folks are using rocket ride.
• 34:08
  They're getting that observability and from the logs
• 34:11
  they're just going back and they're updating autonomously.
• 34:15
  Well, the interesting part about that is you can actually take
• 34:18
  those logs, feed them back and see if you can go with a cheaper
• 34:22
  LLM. That's what I was thinking.
• 34:23
  Take a look at the inputs that the LLM's are getting, gather
• 34:27
  those all up and say issue these and check the results against
• 34:31
  what my new proposed LLM is doing.
• 34:33
  With what I got before on the really hugely expensive LLM,
• 34:37
  maybe I can save a few bucks. Yeah.
• 34:39
  I was thinking, I wonder if anybody has run auto research in
• 34:43
  that regard, because it feels like something that potentially
• 34:46
  you could optimize with just testing different LLMS. Maybe
• 34:51
  it's testing prompts than trying to still get the same output, or
• 34:55
  at least the same. You're passing certain amount of
• 34:58
  tests and. That'd be really interesting to
• 35:02
  actually automatic automate that.
• 35:04
  That's what I was thinking because you know what the end
• 35:07
  state is, right? So it's easy to try and figure
• 35:11
  out and with auto research, I think that's the key.
• 35:13
  As long as you know what that end goal is, it will.
• 35:16
  Now you may spend some money going for it.
• 35:20
  And so it totally defeats the purpose of the whole like, oh,
• 35:23
  we're going to save money by running out of research for the
• 35:26
  next 24 hours and burning a bunch of cash.
• 35:29
  But. But you know what, if it saves
• 35:31
  you in the end over the year that this pipeline is going to
• 35:34
  be up, if I can, you know, cut my LLM costs in half, it's
• 35:37
  actually worth it, right? One, and things are changing on
• 35:40
  a weekly basis, right? And some of these newer model
• 35:43
  versions that are coming out are actually more token efficient.
• 35:46
  So what better way to find that out?
• 35:49
  Then as soon as these new models come out, actually run that
• 35:53
  through your pipeline and see if you're actually saving and
• 35:55
  you're, you know the results are going to be there, right?
• 35:58
  They're not going to regress in terms of capability, but they're
• 36:00
  doing it at a cheaper token cost.
• 36:02
  Yeah, it's almost like something that you would want to just kick
• 36:06
  off a long running agent that does that every couple weeks or
• 36:11
  every month. It's constantly trying to
• 36:13
  optimize as far as, all right, let's scan, maybe scanning
• 36:17
  hugging faces a little bit wild, Wild West because you never
• 36:21
  quite know what those models are doing.
• 36:24
  But yeah, it's probably worth looking into if you really are
• 36:29
  trying to optimize that. I think majority of the folks
• 36:33
  are just trying to make something work and they're not
• 36:35
  at the let's make it cheap phase yet.
• 36:39
  That might come in six months or a year, or it might come in five
• 36:43
  years, who knows? Well, it depends on who you talk
• 36:46
  to. The engineers just want to get
• 36:47
  it to work. The CFO says I'm not going to,
• 36:49
  I'm not going to pay the bills. So, you know, the key is really
• 36:55
  designing it upfront, making sure what you have works.
• 36:58
  And you can do that not just, you know, from guessing, you can
• 37:02
  actually prove that this model actually works for you because
• 37:05
  it's cheaper. And we'll go with that up front.
• 37:09
  I think there was something interesting that you were
• 37:11
  talking about earlier on the inspiration behind building
• 37:15
  Rocket Ride. Can we go into that and what the
• 37:18
  pains were that you were seeing out in the market and why you
• 37:22
  felt like we need something that can give these LLM's a little
• 37:28
  bit more determinism when we're kicking off all these agents?
• 37:32
  I can go through my AI journey. Yeah, give us the whole
• 37:36
  breakdown. On OK, so, so we started in a in
• 37:40
  a company called Operavi, OK and Operavi I've been in the backup
• 37:44
  in, you know, data, you know, synchronization world forever,
• 37:49
  right. And so when we, when we first
• 37:53
  started, you know, Operavi, we were doing data intelligence and
• 37:58
  automation and we actually started reading files and not
• 38:02
  just blobs and sending them over, you know, from to AWS or,
• 38:06
  you know, S3. And that we actually started
• 38:08
  trying to be intelligent on it, like reading the text out of it
• 38:11
  and, you know, trying to figure out, you know, the
• 38:14
  classifications of data and all that kind of stuff.
• 38:16
  So we started, I started my journey on AI.
• 38:21
  This was probably about 3-2 years ago.
• 38:24
  Three years ago and it was actually very simple.
• 38:28
  It was actually taking a file, parsing it, chunking it up,
• 38:34
  embedding it, and putting it in a rag system so that a user
• 38:37
  could very easily chat with their data over, I don't know,
• 38:42
  1.5 billion files or something like that.
• 38:45
  The Classic 2020. Four, that didn't work very that
• 38:47
  didn't work very well. It was so expensive to actually
• 38:51
  embed all that data that it was just not.
• 38:54
  But you know, it was a learning experience for me.
• 38:57
  And so we have this engine that we, you know, was doing this,
• 39:02
  this data processing and actually pushing through the
• 39:04
  pipelines. And then we got the idea.
• 39:06
  Oh, well, you know what, we could actually add this over
• 39:09
  here and this over here. And what if we had this kind of
• 39:11
  node and, you know, all this kind of stuff.
• 39:13
  And then it just really started gelling into, you know, a more
• 39:18
  general product then, you know, just pushing files through it
• 39:22
  and, you know, doing a rag on it.
• 39:25
  So that's where Opera Rocket Ride was actually born is from,
• 39:30
  you know, technology that we've been working on for the last six
• 39:33
  or seven years with, with operavi.
• 39:36
  And, and the reason why we decided to split this off and
• 39:39
  start a new venture is when you think about that core business
• 39:43
  for Operavi, it was really around enterprise selling to
• 39:47
  business units and IT personnel in order to help them discover
• 39:53
  and clean up and leverage their prepare their data to be
• 39:57
  leveraged. So it's really around that data
• 40:00
  discovery and preparation, whereas this new tool was really
• 40:03
  around taking that data, those curated data sets and actually.
• 40:09
  Doing something with them and that that market is really
• 40:12
  developers and so it's completely different market,
• 40:15
  different sales motion. And so that's what the reason
• 40:19
  why we split it off. But as Rod says, this engines
• 40:22
  already been proven at petabyte scale running in an existing
• 40:26
  company, you know at enterprise. And it's probably worth noting
• 40:30
  here too that Rocket rides fully open source, right?
• 40:33
  Correct. So that's awesome.
• 40:36
  Yeah, we donated it. Oh nice.
• 40:38
  Yeah, we donated it all to a foundation.
• 40:40
  So we're part of the Linux Foundation.
• 40:43
  And what is it? The AAIF, Right, So we're part
• 40:47
  of the AIF foundation. It's completely open source.
• 40:50
  Every bit of it. The C++, the T, the Tika code
• 40:53
  and Java and all the the Python is all open source.
• 40:57
  Do you want to go down the C++ route or do you want to give me
• 41:00
  a recent? Bug.
• 41:03
  I'll go with the concrete blood. Blood This is.
• 41:06
  This is this is. Crazy.
• 41:07
  This just happened to me Thursday.
• 41:10
  Oh my goodness, it was like a nightmare.
• 41:12
  I spent like, I don't know, 12 hours on Thursday and then I got
• 41:18
  up at 6:00 in the morning and spent another 9 hours on it.
• 41:23
  The problem was with crew AI. OK, crew AI agents, it really is
• 41:28
  a good technology. I love Crew AI actually, but
• 41:32
  their API OK is synchronous and. What were you trying to do?
• 41:37
  What was the? I was trying to get 2 agents
• 41:39
  running at the same time. OK, two chats.
• 41:42
  OK Actually I had three chats. I said write me a 500 word story
• 41:47
  about a dog and in another word I another one I said write me a
• 41:50
  500 word story about a cat. And then the third one was write
• 41:54
  a 500 word story on an elephant. OK, very simple chap, right.
• 41:59
  But it but it runs through these things.
• 42:02
  And since we were using the synchronous call, it's not multi
• 42:06
  threaded. The problem was as soon as you
• 42:08
  issued the second, you know, right near 500 word story about
• 42:11
  a cat, all of a sudden the dog was thinking the dog story was
• 42:15
  talking about the cat too, with the elephant story just hung.
• 42:19
  So it was all stepping all over itself.
• 42:22
  And it turned out that it was, you know, crew AI on the
• 42:25
  synchronous mode. So then we actually had to go
• 42:28
  back and figure out, OK, how do we do this?
• 42:31
  Because it has three different API calls, you know, to actually
• 42:34
  do it. It's it's, I don't remember what
• 42:37
  the there's a synchronous 1, then there's a synchronous one
• 42:41
  that is an asynchronous one that is actually using the
• 42:44
  synchronous loop, and then there's a truly asynchronous one
• 42:47
  that they actually introduced about 60 days ago on their
• 42:50
  latest stuff. And you saw this just by looking
• 42:52
  at the logs in Rocket Ride? Yeah.
• 42:54
  And you recognized. Wait a minute.
• 42:56
  Wait, why is this over here and why am I getting dog talk in the
• 43:00
  cat, you know, chat and it's like OK, something's wrong here,
• 43:03
  right? Nothing.
• 43:05
  It didn't work out well. So figuring out exactly what to
• 43:09
  change on the crew AI node and the crew, you know, sub agent
• 43:13
  and the orchestrator node that we have to use the async
• 43:16
  version. The new async version allows us
• 43:19
  to run 30 different tasks concurrently without them
• 43:22
  stepping all over each other. That's what Rocket Riot actually
• 43:25
  does because you know what, we'll find a problem, we'll fix
• 43:29
  it. Now, you know, you can just drop
• 43:31
  in these crew AI chats and you know, they just work.
• 43:34
  You don't have to worry about the three different API's and
• 43:37
  how the three different API's work differently and break and
• 43:40
  have global variables. That's not the only one that
• 43:43
  happens to either. It's like OCR is Oh my goodness,
• 43:46
  OCR is it's all over the place whispers great, but you know,
• 43:52
  it's it's all these different little details about all these
• 43:55
  things that you know your your coding agents are going to miss.
• 43:59
  It seems to work just fine because, you know all the
• 44:01
  developers have been working, you know, 1 crew AI agent.
• 44:05
  We haven't released it yet, but all of the developer, yeah, it
• 44:08
  works. And you know, I've got a chat up
• 44:10
  in it, you know, does this and this.
• 44:12
  Have you tried a second chat? Have you tried 10 chats,
• 44:15
  throwing it at it? So those are the problems that
• 44:18
  you run into when actually developing an AI solution.
• 44:22
  And this is actually an example of why a tool and a framework
• 44:27
  like Rocket right is valuable because it's also community
• 44:30
  driven. So as the community discovers
• 44:32
  these kinds of issues that are general issues and they get
• 44:37
  fixed within the framework of the tool, everybody else
• 44:40
  benefits and you no longer see that problem.
• 44:42
  Man, I'm just thinking about all the different APIs, the
• 44:46
  different nuances that the coding agents aren't really
• 44:50
  catching. I've also gone through this a
• 44:52
  few times where I'm just yelling at the coding agent like read
• 44:56
  the docs, just read the docs and look, it says it right there.
• 45:00
  And then I have to copy and paste like do this.
• 45:03
  No, you're looking at it wrong. You're thinking about it wrong.
• 45:06
  Do this, implement it this way. But by that time, it's like, why
• 45:10
  didn't I just write it myself? Why am I having to go and copy
• 45:14
  paste the docs? And sometimes you get, I think
• 45:16
  nowadays it's a lot better. Most documentation sites have
• 45:22
  like LMS dot text. But Even so, we had a a talk
• 45:27
  recently at one of the coding agent conferences where they
• 45:31
  were, they did a bunch of research on how coding agents
• 45:35
  use different products and how well they're able to understand
• 45:41
  how to use a product versus others.
• 45:43
  And some the coding agents get like that because it's very well
• 45:47
  written documentation. It's like machine understandable
• 45:50
  or code coding agent understandable documentation
• 45:54
  versus, oh, this documentation is written for a human and so
• 45:58
  it's not quite the same. A human understands and they're
• 46:01
  probably looking at it and thinking like, yeah, of course
• 46:03
  it's fine. But then the agent has trouble
• 46:06
  when it tries to iterate over that.
• 46:08
  And then you get into those situations where the agent
• 46:11
  doesn't quite understand what it's doing wrong.
• 46:13
  It's just doing something wrong and you want to be able to
• 46:16
  debug. It I may be odd, but one of the
• 46:19
  things that I do is I, I, I go through the documentation, I
• 46:22
  then ask quad and GPT 5, OK, write this prompt on a scale of
• 46:29
  1 to 10 and you know, GPT usually comes back with 9 or 10
• 46:35
  or something like that. Claw will say it's a six, and
• 46:38
  it's like, OK, how can I improve that?
• 46:40
  What can I do? Rewrite the prompt for me so
• 46:42
  that you better understand it the next time.
• 46:44
  And that's an iterative process that we go through not only on
• 46:47
  prompts but on documentation and stuff like that.
• 46:50
  Because, you know, at the end of the day, we want Claude and GPT
• 46:53
  to actually understand what we're trying to talk about.
• 46:55
  So the best thing you can do is ask them.
• 46:59
  Yeah, I've heard folks doing this by way of skills, but and
• 47:03
  it's a little bit like a quick and dirty fine tune of a model
• 47:08
  on a certain product or a certain tool.
• 47:11
  So you can say, OK, now you're going to be using Crew AI.
• 47:15
  We've created a specific Crew AI tool.
• 47:17
  So you have all of these different understandings if the
• 47:20
  documentation isn't good enough. It's an art, it really is.
• 47:25
  And there's so many moving pieces with it that, you know,
• 47:27
  and, and the more that you can actually base things and, you
• 47:31
  know, make the pieces so that they stick together correctly,
• 47:35
  the easier your job becomes because then there are not so
• 47:38
  many moving pieces anymore, right?
• 47:41
  You have your own cloud. Can you talk to me a bit more
• 47:43
  about what that is and how it benefits folks?
• 47:48
  So Rocket Ride launched on March 4th with this open source tool
• 47:51
  that we've been talking about. Later this week, we launch our
• 47:55
  preview of our cloud service because what we've basically
• 47:59
  made available for free through open source is the way to build
• 48:03
  AI applications. Once you actually have an AI
• 48:05
  application and you want a runtime for it.
• 48:08
  Assuming this is not a, you know, personal project, you want
• 48:11
  to have many users using it, you need somewhere to host this
• 48:14
  thing and you want to be able to obviously have it scale and
• 48:18
  scale at a cost efficient way. So the Rocket Ride Cloud is
• 48:21
  going to be basically the the preferred option for us at least
• 48:26
  to to run those AI applications at a reasonable cost, right?
• 48:31
  That's that's the the premise. So one of the some of the
• 48:34
  options that you have to actually deploy these pipelines,
• 48:37
  you know, once you draw all the pretty boxes and connect
• 48:39
  everything up, you know it's working on your laptop.
• 48:42
  What are the three options you have to deploy?
• 48:44
  You can deploy as a service on an on Prem server somewhere.
• 48:48
  So you can supply your own GPU's, you can supply all your
• 48:51
  own stuff, your own hardware and run it yourself.
• 48:54
  Completely managed. We're not involved at all,
• 48:56
  right? You get the source code, you can
• 48:58
  put it wherever you want, the same kind of deal with it.
• 49:01
  You know, most AI companies like Novus and Quadrant, they have
• 49:04
  the open software and then they have this hosting software,
• 49:06
  right? If you don't want to worry
• 49:08
  about, you know, your GPU utilization or getting the
• 49:12
  appropriate GPU, that's where our cloud really comes in.
• 49:15
  You can also deploy via Docker too.
• 49:18
  So if you want a Docker, then you know, you can put it in your
• 49:21
  own on Prem. The advantage of actually going
• 49:23
  with the the Rocket Rye cloud is #1 economy of scales, OK.
• 49:28
  Because if you're, if you're spending X number on open AI and
• 49:33
  then you, you know, X number of requests and then you have Y
• 49:36
  number of requests with this, you know, Llama parse and then
• 49:39
  you know, all this kind of stuff.
• 49:41
  We aggregate everything. So we get economies of scale.
• 49:44
  Everybody who's using Open AI, you're bundled in with them as
• 49:50
  far as that pricing goes. So we can actually run it a lot
• 49:53
  cheaper for your external resources.
• 49:56
  Or is it everybody that's using open AI in Rocket Ride Cloud?
• 50:02
  It's my own company that's using it, or.
• 50:06
  It's, it's your own company, but it's, it's our own company, but
• 50:09
  we're aggregating the inference request to open AI.
• 50:13
  So we get cheaper pricing. We pass that on to you, much
• 50:16
  better pricing that you can buy it yourself.
• 50:18
  Open AI through. You, yes.
• 50:20
  OK, through your cloud. Yes, all right.
• 50:23
  So when you when you drag over the open AILOM, OK, that whole
• 50:28
  thing is being run in our, you know in our cloud.
• 50:31
  So therefore you're with everybody else's open AI
• 50:35
  connections and we aggregate all those the open AI, you know,
• 50:39
  usage into a much better price negotiated price.
• 50:43
  So we can deliver open AI services at a much, you know,
• 50:48
  reduced rate. And same with with Llama Parse
• 50:52
  and all the other ones that are charged services.
• 50:54
  Not only that, you only have one API key to to manage OK, because
• 50:58
  now we charge you for the open AI usage.
• 51:02
  So all you need is the rocket ride key and it gets you all
• 51:05
  these different services based on your usage.
• 51:08
  So. You're in a way you're also
• 51:11
  being a gateway. Yeah, yes.
• 51:14
  Yeah, OK, tell me more. One of the challenges in this
• 51:19
  new era of AI is you see a lot of custom tool stacks for each
• 51:25
  developer. And how does that work in a
• 51:27
  collaborative team setting? So by standardizing on something
• 51:30
  like the Rocket Ride framework now you upgrade to using the
• 51:34
  cloud service and to run those pipelines, we can provide
• 51:39
  collaborative features that enable people to work together
• 51:42
  on bigger projects. We're also look going to be
• 51:46
  offering governance and and auditing and compliance as well
• 51:50
  as SoC 2. And as we probably mentioned,
• 51:54
  we've come up with ways to maximize the compute resource
• 51:59
  utilization to lower our COGS, which we can then pass those
• 52:02
  savings on to users as well so that we can offer this cloud
• 52:06
  service at a cheaper rate than say, you'd find on the open
• 52:09
  market. You know, let's let's go into
• 52:11
  that for just a second. What's really interesting is is
• 52:14
  the the Rocket Ride Cloud and how we deal with GPUs.
• 52:17
  We have something called the model server which allows
• 52:20
  aggregation of inference. So let's say we have 100
• 52:24
  different customers running easy OCR, right?
• 52:27
  Normally what would happen is you have 100 different easy OCR
• 52:31
  is loaded all over these GPUs all over the place, right.
• 52:35
  So that means that you've got a lot of GPU resources that you're
• 52:40
  essentially, you know, wasting. So with our model server that we
• 52:44
  run within within rocket ride and within the rocket ride
• 52:48
  cloud, we only have one OCR easy OCR and we do inference on
• 52:52
  really powerful GPU on you know, so all those 100 customers are
• 52:58
  sharing that one GPU with the easy OCR.
• 53:00
  Now that sounds terrible, you know, some kind of bottleneck or
• 53:04
  something like that. But we have the gas GP is first
• 53:06
  of all. Second of all, we scale.
• 53:09
  If the queue starts getting too full and latency is actually,
• 53:13
  you know, going up, we'll actually throw in another easy
• 53:16
  OCR on it, you know, so we do 2 inferences concurrently.
• 53:19
  If it's continues to grow, then we'll add a third easy OCR and
• 53:23
  it's so it scales dynamically, but most importantly, it scales
• 53:27
  down as well. Now what would you do with, you
• 53:31
  know, inference on on Prem right, using, you know, easy OCR
• 53:35
  or whisper or something like that.
• 53:37
  You're actually paying 24/7 for a GPU that may be only use 5
• 53:43
  minutes a day on a whisper model or 8 hours a day.
• 53:46
  You're still spending 16 hours hosting this whisper model that
• 53:50
  you're not utilizing. So in order to get the
• 53:52
  performance, you have a choice either by really expensive GPU
• 53:56
  that's only going to be used 8:00 to 8:00 to 10 hours a day,
• 53:59
  or use Rocket Ride Cloud, which you only get charged for usage.
• 54:04
  So if it's only used for 10 hours a day, we'll kick that
• 54:07
  model out when you're not using it and bring in another model
• 54:10
  and reduce costs. And one of the things that we're
• 54:13
  able to do is because we understand what's in each of
• 54:18
  these pipelines using our framework, we can judiciously
• 54:21
  load these models in a smart way to again maximize the
• 54:27
  availability of GPU for other usage.
• 54:31
  I'm sure there's somebody out there thinking like, OK, that
• 54:34
  sounds great, but it also sounds dangerous because if I'm sharing
• 54:39
  GPU's with other people's companies, like my data is
• 54:44
  private and sensitive. And how do you make sure that
• 54:47
  it's isolated for just me? So when we do the inference and
• 54:52
  we put when we put models onto onto server, we typically batch
• 54:57
  things into 64 or 128 batches. They go into the GPU's and
• 55:02
  unless NVIDIA has some really serious bugs and then does
• 55:05
  crosstalk, you know amongst the different channels, they're each
• 55:09
  individual channels. So when we feed in 128 channels
• 55:14
  into the GPU, tell it OK, go do the inference, do the operation
• 55:18
  now and then on the back end, we pull everybody's stuff off of
• 55:22
  it. So we split it back up.
• 55:23
  We're only sharing it while it's on the GPU doing the inference.
• 55:27
  So unless there's a real serious problem with the GPU, there's
• 55:31
  absolutely no crosstalk between the between different different
• 55:34
  inference loads. So the lanes are isolated.
• 55:37
  The lanes are isolated. The cost savings thing is huge,
• 55:42
  man. And I think that more and more
• 55:45
  as time goes on, folks are going to just start asking for that.
• 55:49
  You're probably already seeing it.
• 55:51
  And it's like, yeah, all right, we're doing the math here.
• 55:54
  And if we want to scale or if we really roll out this product to
• 55:59
  our whole user base, it's going to start being expensive.
• 56:03
  We've seen this pattern before, right, with cloud, OK, and, and
• 56:08
  how everybody wanted to go to the cloud and then the CFO said,
• 56:11
  what the Hell's going on, right. It's like, and I'm even my CEO,
• 56:16
  we says we spent $25,000 a month on AWS.
• 56:19
  And it's like, why? Where did that?
• 56:22
  Yeah, where did it go? It's like, I don't know.
• 56:26
  So it's like this is, this is just the beginning.
• 56:29
  We're just trying to get to work right now.
• 56:30
  But in the next year or two years, you're going to see a lot
• 56:34
  of pressure, you know, coming on, on the operational cost.