Latest / Agentic Conversations (formally mlops.community) / We Can All Be AI Engineers and We Can Do It with Open Source Models // Luke Marsden // #273

Transcript

We Can All Be AI Engineers and We Can Do It with Open Source Models // Luke Marsden // #273 · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Luke Marsden, my title is CEO and founder, and I
• 0:05
  take my coffee with an Arrow press at home, which I drink a
• 0:08
  lot of, which is just like Arrow Press, plus like nice coffee
• 0:12
  from a local brewery, plus a splash of milk.
• 0:15
  But I really like a flat white if I'm in a fancy cafe.
• 0:18
  Folks, welcome back to another ML OPS Community podcast.
• 0:21
  I am your host Dimitrios. Today we're talking to a long
• 0:26
  time friend, Luke Marsden, and I appreciate whenever he comes on
• 0:32
  this podcast, he's been on a few times and that is because he is
• 0:36
  one of the reasons that this podcast and this community
• 0:41
  exists. He's the guy that told me we
• 0:44
  should start a community. When I was working for him, he
• 0:47
  was CEO and I was just the lowest man on the totem pole and
• 0:52
  that is why we are here today. I am with you in your ear.
• 0:57
  Luke talks about two things that I want to highlight before I get
• 1:02
  into my musical recommendation of the episode.
• 1:05
  First thing is AI specs, which is all about creating standards
• 1:15
  as YAML files for how to have CICD for AI apps.
• 1:24
  And the second thing that I absolutely enjoyed him diving
• 1:29
  into and getting really technical with us on is how he
• 1:32
  set up the evals, specifically the tests for AI specs and his
• 1:39
  product, Helix. All right, let's dive right into
• 1:44
  it. As always, if you enjoy this
• 1:46
  episode, share it with just one friend.
• 1:48
  And Speaking of friends, I got to catch up with an old friend
• 1:52
  and my old friend, old musical group that I have not listened
• 1:57
  to in ages. Let's just bring out rat a tat.
• 2:03
  Here's your musical recommendation for the day,
• 2:10
  bringing you a little bit of variety in that for You page.
• 3:11
  Luke Marsden. I will always have a very
• 3:14
  special spot in my heart for you because I remember those times
• 3:22
  where I had no idea about technology at all.
• 3:26
  I specifically remember a time where we were walking down the
• 3:29
  street in London after a conference and I was asking you
• 3:34
  what Kubernetes was. And you were telling me.
• 3:36
  And then I was like, so it's like Hadoop and you're like, no
• 3:40
  different things, very different.
• 3:42
  I'm like, how is Hadoop different than Kubernetes?
• 3:45
  And so you went and you have always been so patient with me
• 3:49
  in explaining things. And that is why I'm very excited
• 3:52
  to talk with you again so you can be patient and show
• 3:55
  everybody else how patient you are when you explain things to
• 3:58
  me. Bless.
• 3:59
  Thank you Demitrios. And likewise, you have a very
• 4:02
  special place in my heart because like when dot science
• 4:05
  shut down and the envelopes community got born from it, it
• 4:10
  was your drive and passion that made it a thing.
• 4:14
  There we go. And when I was speaking at the
• 4:18
  at an ML OPS community meet up in San Francisco a few weeks
• 4:20
  ago, I said something at the beginning which was like, it's
• 4:24
  so incredible to be here in San Francisco with this thriving
• 4:28
  community around me that you and I started back in those dark
• 4:31
  days. Thank you.
• 4:33
  For people that do not know, Luke and I were working at Dot
• 4:37
  Science. You were the CEO at Dot Science
• 4:40
  and I was just some random sales guy and who was trying to learn
• 4:43
  about tech. And right before the company
• 4:47
  went out of business, Luke told me, hey, maybe we should do
• 4:50
  something with a community or something.
• 4:52
  And I said, OK, let's try it. And that's how the MO OPS
• 4:57
  community was born. Let's get into CICD, Fourgen AI
• 5:03
  and also you've come at AI and traditional ML and data from a
• 5:10
  DevOps background. And so I think you've always
• 5:13
  been very anchored in software engineering practises and
• 5:19
  software engineering. And so that's part of why I
• 5:23
  think you had that inspiration in the beginning to start dot
• 5:26
  science and say we need lineage, we need tracking for all of this
• 5:30
  stuff because if you make changes, who's going to know and
• 5:33
  how are they going to know? And what are you going to be
• 5:36
  able to do about it? And as my friend Chad likes to
• 5:40
  say, it's data change management and we have hub to help with
• 5:44
  just regular code change management.
• 5:47
  Let's start there like CICD for AI.
• 5:49
  What does that even mean in your eyes?
• 5:52
  Yes. So thank you.
• 5:53
  And I think I'd like to start that by talking about how
• 6:00
  software engineering gets done. And we've had decades to figure
• 6:05
  out how to do software engineering well and to ship
• 6:07
  software well. And when we're shipping just
• 6:12
  regular software, there's basically 2 fundamental pieces
• 6:16
  that you need to be able to do production ready software, and
• 6:22
  that is tests and deployment. And so that means being able to
• 6:29
  test your software automatically before you merge a pull request
• 6:34
  to the main branch, for example. And it means being able to
• 6:38
  control which version of software is running in
• 6:41
  production. And often there's a term that
• 6:43
  used called get OPS, which is this idea that was coined by my
• 6:46
  old boss Alexis Richardson at Weaveworks back in the day.
• 6:50
  And the idea there was like, you should keep a record of what you
• 6:54
  want running in production in a Git repository and then how and
• 6:58
  then reconcile that against production.
• 7:00
  So there's these two key ideas of CI, which is continuous
• 7:04
  integration, which just means testing your software, and then
• 7:06
  CD, which overlaps with this idea of Git, Git OPS, which is
• 7:11
  continuous delivery. And it's like being able to know
• 7:14
  what version is running in production, roll it back easily
• 7:16
  and all that stuff. So then, OK, probably everyone
• 7:19
  I'm talking to already knows that because like CICD is a
• 7:22
  fairly common idea. But the question then is how
• 7:24
  does it apply to to generative AI apps like LLM apps?
• 7:31
  And the thing I'm trying to really push on at the moment is
• 7:35
  to argue that it's basically exactly the same as software.
• 7:40
  It's just that the definitions of what we mean by testing and
• 7:44
  deployment are slightly different.
• 7:46
  And so if you hear people all the time talking about evals, I
• 7:49
  know, I think I saw you had Gideon on recently talking about
• 7:53
  evals. And that's great because how you
• 7:56
  test Gen AI apps is just evals. It's like evaluations and, and
• 8:01
  yes, so my argument is basically to, to do production ready Gen
• 8:06
  AI with you need CICD for Gen AI.
• 8:10
  And then we can break that apart, which is the CI piece is
• 8:14
  you do testing, you test your app before you merge a change
• 8:18
  into the main branch. And the way you do that is with
• 8:21
  evals and then you deploy the app.
• 8:24
  And then the thing that we're trying to make a reality is this
• 8:28
  idea that the spec for an AI app should just be version
• 8:33
  controlled YAML, and that you could almost deploy it to a
• 8:36
  Kubernetes cluster in the same way that you deploy like an
• 8:39
  ingress record or something. I can go into more detail and
• 8:43
  explain what I mean. Yeah.
• 8:43
  Tell me that I haven't heard that before.
• 8:45
  Version controlled YAML. Yeah.
• 8:47
  So I mean, if you're a sort of Kubernetes person or a dev OPS
• 8:51
  person in general, you'll be familiar with this idea of
• 8:54
  having a Git repository full of different YAML files for
• 8:58
  different kind of parts of the application.
• 9:00
  So you might have an irregular software application, you might
• 9:04
  have five different micro services, a database and so on.
• 9:09
  And you would have all the different parts of that
• 9:15
  application are described by Kubernetes manifests.
• 9:20
  So they're like dot yaml files in git that say this is a
• 9:24
  deployment object. For example, this deployment
• 9:27
  object says I'm going to deploy the front end, for example, like
• 9:32
  a web front end or an API server.
• 9:35
  And I'm going to, I want that to be at least five of them.
• 9:39
  And you can scale up to 10 or something.
• 9:41
  And they want to have this Docker image.
• 9:43
  And then when you apply that to your Kubernetes cluster,
• 9:48
  Kubernetes will reconcile that and be like, OK, I'm going to
• 9:51
  make sure that there's always five of these running.
• 9:53
  And if one of the servers dies, I'll spin up some more.
• 9:56
  And it will, it will keep things in equilibrium like that.
• 10:00
  And so the, the thing that we're trying to make real that we have
• 10:04
  this project called AI spec.org is to try and define a similar
• 10:09
  spec for how you can define a generative AI application.
• 10:14
  Because if you can do that, then the exciting thing is that you
• 10:17
  can, you can manage that generative AI application in
• 10:21
  exactly the same way that you manage all your software.
• 10:23
  And why are we doing this? It's in order to make generative
• 10:27
  AI much more accessible to people who are familiar with
• 10:31
  DevOps and and software engineering because I think
• 10:35
  there's no read like the tooling has got good enough now that you
• 10:38
  can do that. And and there's no reason that
• 10:40
  we can't all be AI engineers, whether we're a business person
• 10:43
  just prototyping an application or a DevOps or software person
• 10:48
  trying to productionize an application by adding tests to
• 10:51
  it and then doing get OPS with with the manifest.
• 10:54
  Yeah, so. How do you how so?
• 10:57
  I love AI spec by the way, and if anybody wants to or feels
• 11:02
  inspired to join that movement, there is a channel in the
• 11:05
  envelopes community all about that.
• 11:07
  And how do you foresee a Kubernetes manifest giving
• 11:18
  detail about the system? The AI system?
• 11:22
  Yeah, what does that even look like?
• 11:24
  So I'll walk you through that. The AI spec is basically, it's
• 11:28
  basically, if you're not familiar, I guess everyone's
• 11:30
  familiar with YAML, right? But a YAML is basically just
• 11:32
  like a configuration file written in a text format.
• 11:35
  And I'll just go through the building blocks of of an AI
• 11:40
  application, if that's OK, and we can talk about how each of
• 11:44
  those can be represented by one of those fields in this YAML
• 11:47
  file. And this is just so I'm super
• 11:50
  clear on it. It would be a micro service,
• 11:53
  which is a model. At the end of the day, you're
• 11:57
  getting a model out of it or you're like hitting a model or
• 12:01
  is it a rag pipeline like what are we looking at?
• 12:05
  Yes, you can think of it as a micro service.
• 12:08
  How it's implemented can vary, but I but in particular, you can
• 12:14
  think of it as like part of your application, a micro service.
• 12:18
  And the AI spec argues that we should try and come up with a
• 12:22
  standard format of describing not just the model, but also a
• 12:26
  system prompt for the model. That's the baseline, but then
• 12:29
  also a definition for what we mean by knowledge for a kind of
• 12:34
  a basic form of rag that everyone can use that that we,
• 12:37
  and then you can have all the different parameters that you
• 12:40
  can twiddle to, to make it better chunk sizes and things.
• 12:43
  But then also API integrations. So this is something I'm really
• 12:47
  big on at the moment because it's what we're finding most of
• 12:51
  our customers want, which is the ability to integrate these LLMS
• 12:55
  into business systems. Because it's, it's all very well
• 12:57
  to be able to just have a chat with a chat bot.
• 13:00
  But yeah, it's much more interesting if the chat bot can
• 13:03
  talk to your Jira instance and you can talk about the current
• 13:07
  Sprint, right? Or you can say, write some code
• 13:10
  for me for this issue that I've been assigned, and then it will
• 13:13
  bring in all that context. So yes, we're trying to make
• 13:17
  this AI Spec B. Let's see is which model you're
• 13:20
  using. Any system prompts knowledge, IE
• 13:24
  RAG? I think knowledge is a nicer,
• 13:26
  more accessible word than RAG, because RAG sounds all fancy.
• 13:29
  Retrieval, augmented generation. What are you talking about?
• 13:32
  It's just knowledge. It's giving the model knowledge.
• 13:34
  There's integrations, which is this like integrating with
• 13:36
  business systems, and then there's tests, right?
• 13:39
  Remember, we were talking about testing.
• 13:41
  So you should put the tests for your AI app in the app spec
• 13:46
  itself so that they all get versioned together, right?
• 13:48
  That's how you would do it with software.
• 13:50
  You'd have your tests, your test code right next to your real
• 13:52
  code, and then the deployment piece, which is like, how do you
• 13:55
  actually deploy that? But in terms of the actual AI
• 13:57
  spec, it's model prompts, knowledge integration and tests.
• 14:01
  And I think if we version all those things together, then
• 14:03
  we're going to have a better time.
• 14:05
  And does because for me it feels like there is a lot of, there's
• 14:12
  a lot more things that go into the, if we're talking RAG, they
• 14:19
  go into building a rag system that aren't on that manifest.
• 14:23
  Like how do you see that being reflected?
• 14:26
  So that's a really good question.
• 14:27
  And we've had some users come along and want to do more
• 14:31
  advanced rag systems than the one that you get out-of-the-box
• 14:34
  with AI spec. And the way that we've handled
• 14:37
  that is to have some other fields that are like optional
• 14:42
  fields that are not like required.
• 14:45
  Basically, you should be able to have a good time most of the
• 14:48
  time by just having a knowledge spec inside your AI spec that
• 14:54
  says scrape this website, strip off headers and footers, go 10
• 14:58
  links deep, go oh, and refresh every hour or refresh nightly.
• 15:03
  Because those are some of the examples for what we're doing in
• 15:05
  our like out-of-the-box RAG. But then we, we had these other
• 15:09
  users come along and say, we're used to building these much more
• 15:11
  sophisticated RAG systems, that they might have different
• 15:14
  databases and then a sort of agent in front that decides
• 15:18
  which database to query based on like the user's intent and
• 15:22
  things like this. And So what we did there was we
• 15:24
  said, if you want to get fancy like that, then let's use, let's
• 15:29
  have some other fields in that spec that allow you to break out
• 15:32
  of the simple version into a more sophisticated
• 15:36
  implementation. And that might either be that
• 15:39
  you, I think how we did it was that you can reference an HTTP
• 15:42
  server that you're running that has some specific RAG endpoints
• 15:47
  that are like customised ingestion and query endpoints.
• 15:51
  So the idea is, yeah, like we can make it easy for most people
• 15:54
  out-of-the-box, but then when you need to go in and do the
• 15:57
  more detailed RAG work, then you can break out of that box.
• 16:02
  And what tools would you use to implement those HP endpoints?
• 16:08
  You'd probably use Llama Index or whatever other sophisticated
• 16:13
  tooling you have for building advanced RAG systems.
• 16:15
  We're explicitly not trying to compete with the likes of Llama
• 16:18
  Index with this, with this proposal.
• 16:21
  We're trying to make AI spec. We're basically trying to make
• 16:24
  it so that if you step one, any business person can prototype an
• 16:28
  application in a web interface that's like actually a really
• 16:31
  nice property. Step 2.
• 16:34
  Any person with some technical skills like a sys admin or a
• 16:40
  DevOps person? Product programmer.
• 16:42
  Sorry. Or.
• 16:44
  A product manager exactly with some technical skills could
• 16:48
  build a this YAML spec and version controller and write
• 16:51
  some tests for it so they know that it keeps working.
• 16:53
  And if they change one prompt, does it regress other cases that
• 16:56
  you care about? You think of it as like test
• 16:58
  driven development for Gen AI. We need to get to that point as
• 17:02
  a community I think. And then yes, if you want to go
• 17:05
  and do more advanced drag stuff and you need to learn more about
• 17:08
  the rag to make it good enough and your tests will tell you
• 17:11
  hopefully if the quality is there by the way, because the
• 17:14
  tests is just evals, then then yeah, you can break out the box
• 17:17
  and and do more more sophisticated stuff.
• 17:21
  This is a bit of a tangent so I'm going to warn you right now.
• 17:23
  So I'm going to go down a different Rd, but I had the
• 17:29
  realisation or almost a thought experiment where I was talking
• 17:34
  to Fernando probably 3-4 weeks ago about how he is constantly
• 17:41
  trying to use AI to solve problems that he has within the
• 17:45
  business. And he's rapidly prototyping
• 17:48
  things and he's rapidly trying to stand up agents or whatever.
• 17:52
  And one thing that he said in passing, but it's stuck with me
• 17:57
  is how easy it is right now for him to create a agent to do
• 18:04
  something versus if he were to bring on a SAS solution to try
• 18:10
  and do that. Yeah.
• 18:12
  And I thought this was fascinating because it's a
• 18:16
  little bit of you can rapidly prototype something so quickly.
• 18:22
  And if you are very comfortable with building AI apps, for lack
• 18:29
  of a better term, then it may be faster for you to build the AI
• 18:33
  app then that uses AI with all its headaches then it would be
• 18:39
  for you to not use AI. And that for me is mind blowing
• 18:43
  because up until now I've always been in the camp of the first
• 18:50
  rule of AI is don't use AI. Unless you have to, yeah, don't
• 18:55
  just faster everywhere like. Exactly.
• 18:58
  But then if it's easier for you to create a prototype that will
• 19:03
  do what you want by using it versus hard coding it, that's
• 19:09
  where I my eyes opened U in it from something that he said in
• 19:13
  passing. And so I think what you're
• 19:15
  trying to do here is really cool because you're just making that
• 19:18
  rapid prototyping even easier for folks.
• 19:22
  And so talk to me about almost these 3 levels, right?
• 19:25
  You've got the non-technical person that can prototype
• 19:29
  something, you've got the semi technical or their technical,
• 19:32
  but maybe in a different domain than an AI engineering domain.
• 19:36
  And then you've got the full on production engineer for AI apps
• 19:42
  domain. What does it even look like?
• 19:45
  If I'm a non-technical person, how do I interact with it and
• 19:50
  how do I get value from this? Because I know that is something
• 19:53
  that it feels like is the IT, it opens up so many doors.
• 19:59
  Absolutely. And I think on the point of that
• 20:02
  comment that you said Fernando made, I, I do think it's super
• 20:06
  interesting time that anyone can now develop software quite
• 20:10
  quickly, especially with Cursor and Claude and I think.
• 20:16
  The age of rapid prototyping or just about anything you want to
• 20:19
  build is definitely upon us. I think there's the.
• 20:22
  Sorry, there's the, yeah, we got 90% of the way there and then
• 20:26
  the other 10% was really hard. Yeah, it takes longer than
• 20:29
  getting 90% of the way there. I'll come back to this sort of
• 20:32
  three layers thing if you like. I think the level 1 is like the
• 20:37
  Super non-technical person, like the only requirement is a web
• 20:39
  browser and Full disclosure, like we got massively inspired
• 20:45
  by the GPTS feature in ChatGPT when we started thinking about
• 20:50
  enabling these sort of very non-technical users to to do it.
• 20:54
  Of course, our angle is like open source models and make it
• 20:58
  run locally. So if you're, if you have like
• 21:01
  regulatory reasons, for example, not to be able to send your data
• 21:04
  somewhere else and you want to be able to do what I'm talking
• 21:07
  about, then come talk to me, the non-technical user being able to
• 21:10
  prototype something quickly. I think that is really
• 21:13
  exemplified in in the GPTS feature in ChatGPT.
• 21:18
  And when GPTS launched, I was initially a bit sceptical of it
• 21:21
  because they'd had a couple of iterations before that that
• 21:24
  didn't go so well with. I can't even remember what they
• 21:26
  were called, but they were like different plug insurance for
• 21:29
  ChatGPT that everyone was a bit meth.
• 21:32
  But then I started hearing something really interesting at
• 21:34
  conferences and meetups that I went to, which was I actually
• 21:39
  started hearing about serious business use cases where people
• 21:42
  had used GPTS internally inside a company to do really quite
• 21:47
  complicated things. So for example, there was a
• 21:49
  there's a company here in Bristol that is doing risk
• 21:53
  assessments. This might sound slightly
• 21:55
  terrifying, but they're doing risk assessments for for the
• 21:58
  film industry. I'm going to go and shoot a film
• 22:00
  like on a great iceberg somewhere.
• 22:03
  What do I need to think about? And they'd rigged up like they
• 22:06
  called it like a chain of GPTS. They'd literally written this,
• 22:09
  they'd done this all inside, inside ChatGPT, like by pointing
• 22:13
  and clicking to prototype an application that could generate
• 22:16
  these risk assessments automatically, or at least
• 22:18
  generate suggestions for them that humans can then check.
• 22:21
  And they were having success with this thing.
• 22:23
  And I was like, oh, maybe I shouldn't be ignoring this GPTS
• 22:25
  feature because actually it's like a silent winner potentially
• 22:29
  because it's actually getting adoption inside inside
• 22:32
  businesses. So the I'll recap what that is.
• 22:35
  I'm sure most people listening know already.
• 22:37
  But what you can do is you can go into the ChatGPT and you can
• 22:40
  create what's called a GPT like your own kind of customised
• 22:44
  thing. And you can give it like an
• 22:46
  avatar and a system prompt, which is like instructions like
• 22:50
  behave like this is what you should do.
• 22:52
  This is what you shouldn't do. But then what's really
• 22:54
  interesting is that you can also give that GPT, but your sort of
• 22:58
  customised chat bot, you can give it knowledge and you can
• 23:02
  give it integrations. And the knowledge piece is a
• 23:06
  basic rag pipeline like we were talking about earlier.
• 23:08
  Can we give everyone a basic rag pipeline?
• 23:11
  That works most of the time for a lot of use cases.
• 23:14
  And it's literally put a website into this thing or a drag and
• 23:17
  drop a Word document into it and, and that works and it works
• 23:21
  pretty well. And then the integrations piece
• 23:24
  is where it gets super interesting for me is that you
• 23:26
  can say, OK, if I've got like a, it's called a Swagger spec for
• 23:32
  an API for a business API, I can also just put that in there.
• 23:36
  And now this LLM, this chat bot can make API calls on my behalf.
• 23:42
  And so it can do things like integrating into business
• 23:44
  systems. And yes, so that's this sort of
• 23:47
  basic level of what we're trying to enable with open source
• 23:51
  models is the ability for anyone to come along who's
• 23:55
  non-technical and use a web interface to construct one of
• 23:58
  these AI apps. And when I say app, what I mean
• 24:02
  is it's like a GPT from ChatGPT. So does that make sense?
• 24:06
  Yep, Yep. So basically, throw some
• 24:09
  information at it, give it a system prompt and integrate it
• 24:13
  with whatever you're working with, and you're off to the
• 24:15
  races. Exactly, exactly.
• 24:17
  Now this is where it gets more interesting.
• 24:20
  I'm a software DevOps person, right?
• 24:23
  That's my background, like you said, and if I think about
• 24:26
  people building apps by pointing and clicking in web interfaces,
• 24:30
  I'd scream and cry a little bit on the inside.
• 24:34
  I don't know if you've ever heard of Jenkins.
• 24:37
  It's like the old CI system, but Jenkins was configured by people
• 24:41
  logging into it and pointing and clicking in a web interface.
• 24:44
  And it was widely hated in sort of DevOps, like more
• 24:48
  sophisticated DevOps teams, I guess maybe 10 years ago.
• 24:52
  Because it's that's the wrong way to do it.
• 24:54
  You shouldn't like, how are you going to make this system
• 24:57
  reproducible if we lose our Jenkins server?
• 24:59
  We can't be a snowflake. You had this idea.
• 25:02
  Of individual unique piece of software.
• 25:06
  Yeah, exactly. And there's no software should
• 25:08
  be declarative. We should be able to recover
• 25:10
  from losing a server by just spinning another one back up
• 25:13
  again. It should be cattle, not pets.
• 25:15
  Sorry, I know you're vegetarian, but I've.
• 25:16
  Actually heard it. It's funny enough somebody the
• 25:19
  other day said this in passing. Oh, what you're describing right
• 25:22
  now could be called gooey OPS because you have to work with a
• 25:27
  gooey to get what you want. Yeah, and it's bad and wrong.
• 25:32
  Don't forget that. Ever.
• 25:34
  So it's so the interesting idea here is OK, so we can make
• 25:38
  building AI apps more accessible to non-technical people by
• 25:42
  giving them a web interface that's like the GPTS editor in
• 25:45
  ChatGPT. And like you can see that today
• 25:48
  if you go and look at Helix dot ML and like you log into the
• 25:52
  SAS, you can see we've built this app editor thing in there,
• 25:55
  which is that same. It's basically a GPTS clone from
• 25:59
  ChatGPT. And that's you can see that on
• 26:01
  the AI spec website as well. There's a little video demo of
• 26:04
  it. But but then you get me like
• 26:10
  waving a stick and saying that's the wrong way to do it.
• 26:14
  So how do you bridge from, there's a, a GUI where you can
• 26:18
  do GUI OPS to build your prototype of your AI
• 26:23
  application? And then how do you go from that
• 26:26
  to, to something that you feel good about from a dev OPS
• 26:29
  perspective with? And the, the way that we're
• 26:32
  trying to push that is that with GPTS from ChatGPT, there is no
• 26:39
  output format of a GPT. You can't take it and and export
• 26:45
  it. It's just there in the web
• 26:47
  interface and it lives there. And if you go in and someone
• 26:50
  clicks buttons and changes it, then it changes.
• 26:51
  And do you know that it got better or not when you changed
• 26:54
  it? I don't know, Like you're not
• 26:56
  doing evals on it. The idea that we're pushing then
• 26:58
  is that this app editor for these AI apps should output
• 27:04
  version controllable YAML. Right.
• 27:07
  So there's the bridge. It's like you go from a
• 27:10
  non-technical person having prototyped something.
• 27:13
  And the thing that I'm really from an organisational
• 27:15
  perspective, I think it's quite interesting to ask the question,
• 27:19
  could a product person or ACEO or someone who's non-technical,
• 27:23
  right? Prototype an AI application in a
• 27:26
  GUI like this, but then hand the artefact, which is this YAML
• 27:30
  file off to a technical team inside the organisation who are
• 27:35
  maybe a bit closer to AI engineering.
• 27:37
  But they don't actually need to be AI engineers.
• 27:38
  They just need to be software DevOps people who are familiar
• 27:41
  with pull requests and YAML files and ID ES and then ask
• 27:45
  them to productionize it. So that's the kind of
• 27:48
  organisational question is can we enable people to prototype
• 27:52
  things easily and then bridge that gap into productionizing.
• 27:56
  So I can talk more about productionizing if you like,
• 27:58
  but. The main question that I have as
• 28:01
  you're saying this is for that GBT use case, which is almost in
• 28:07
  a sense productivity use cases. I guess you could say usually
• 28:15
  when I think about a GB TI am getting productivity from the
• 28:23
  LLM in some way shape or form. It's as you said, plugging into
• 28:27
  Jira and maybe creating code or summarising the last or the next
• 28:32
  Sprint. Yep, etcetera, etcetera.
• 28:34
  Or it's looking at Notion or Google Docs and helping me find
• 28:38
  and create a new doc with knowledge of past docs.
• 28:42
  Sure. What about when my app or my
• 28:50
  company product wants to incorporate AI into the product
• 28:56
  itself? Are you just purposefully not
• 29:00
  trying to go after that use case right now?
• 29:03
  That's actually a really good question.
• 29:05
  I shouldn't sound so surprised, but it's no, it's, it's a really
• 29:09
  interesting question. I, I, I think you can get to
• 29:14
  both of those use cases through this, but I think that if you
• 29:20
  were starting from the perspective of wanting to
• 29:22
  incorporate AI into an exit into your business application, you
• 29:27
  probably wouldn't start with a security OPS way to, to set up
• 29:32
  the endpoints. But what you can still do with
• 29:34
  the AI spec is create API endpoints that you can call from
• 29:40
  from your business application. And so actually what we have,
• 29:45
  what we are building here is probably a faster way to get
• 29:48
  started at incorporating AI into your into your application that
• 29:52
  your company's building by just spinning up some of these
• 29:55
  endpoints. And what we did there, which
• 29:57
  might be interesting to people is we went all in on the open
• 30:02
  AIAPI. So we said everything should be
• 30:06
  open AIAPI chat completions API on the way in and on the way
• 30:10
  out. And, and what we've So what
• 30:14
  these AI spec compatible things do is you can have this like
• 30:19
  chunk of YAML that defines knowledge API integrations and
• 30:23
  then you can surface that whole app as an open AI compatible
• 30:28
  API. It's so it's you're talking to
• 30:30
  an LLM. Well, you are, but it's like
• 30:33
  you're talking to an LLM, but it's an LLM that's have that's
• 30:36
  had that's been imbued with superpowers like that.
• 30:39
  LLM already knows about the contents of your RAG store
• 30:42
  because it's got RAG built into it and it already knows how to
• 30:45
  make API calls into into API endpoints.
• 30:48
  It's done like a bunch more of the work than you get, for
• 30:52
  example, just with function calling from the open AIAPI
• 30:55
  because it will actually make the API call for you and
• 30:57
  summarise the response. So yeah, I hadn't really thought
• 31:00
  about it like that before. So it's interesting.
• 31:03
  But I would say that you can satisfy both those use cases of
• 31:07
  your productivity use case, which is starting from playing
• 31:11
  around, prototyping and then maybe pushing that into wanting
• 31:14
  to roll it out to more internal users and productionizing that
• 31:17
  and making sure that you've got quality there.
• 31:20
  But you could also use this approach for adding AI
• 31:24
  capabilities to an existing app because it's all open AI
• 31:27
  compatible API all the way through.
• 31:29
  And when I was looking at that, I was looking at the open AISDK
• 31:33
  support in different languages and frameworks and it's
• 31:36
  everywhere. I wasn't surprised, but I was
• 31:38
  like there's if you want to plug open AI compatible API into your
• 31:41
  WordPress site, there's ten different WordPress plug
• 31:44
  insurance to do it. And the same is true in Java,
• 31:46
  Spring and like all these other different commute like language
• 31:49
  communities. So I think that was a no
• 31:51
  brainer. Yeah, because it's fascinating
• 31:54
  for me. I guess The thing is when you
• 31:58
  want to incorporate it in into your product, it's a little bit
• 32:02
  more complex because how are you going to present this in your
• 32:06
  product as this AI feature? Is it going to be just one of
• 32:09
  those little star buttons that people can click on and then
• 32:12
  they have super AI features? Or is it directly when someone
• 32:17
  is doing what they normally do first it hits open AI and it
• 32:25
  tries to give you an answer before it goes into the product.
• 32:28
  And for each use case, it's going to be different.
• 32:31
  And I think the way that you prototype those is a little bit
• 32:36
  more in depth. And it also, I'm not sure if
• 32:41
  it's something that is going to be done by the CEO or the
• 32:44
  product manager over the weekend, but maybe it is.
• 32:46
  I know that when we had Phillip on here from Honeycomb, he said
• 32:53
  something along the lines of I think I prototyped our features
• 32:58
  are AI features within the app over weekend and.
• 33:04
  And then it was. That whole thing of it took me a
• 33:07
  weekend to create the prototype, but then another six months to
• 33:10
  actually create the real production ready thing.
• 33:13
  But it goes back to the conversation.
• 33:15
  And really, I think the theme of what we've been saying is you
• 33:18
  prototype it to get an early signal on if it's valuable.
• 33:22
  And then if it's valuable, you can hand it off to the right
• 33:27
  folks who can productionize it. But it's a lot easier to see if
• 33:32
  it's valuable when there's a prototype.
• 33:35
  Yeah. And if you start playing with it
• 33:38
  and if you start to see it, is it able to do what I have in my
• 33:44
  mind? Yes.
• 33:45
  Can we at least get there a little bit, or am I totally
• 33:49
  crazy for thinking that? Yeah, absolutely.
• 33:52
  And then if it is able to do what's in your mind, you can
• 33:55
  write a test for it, which makes sure that it's going to continue
• 33:57
  to work. And the idea of being able to
• 34:01
  take this like AI spec from all the way from the prototype in
• 34:05
  the web interface through the YAML format and have the same
• 34:09
  thing, the same artefact throughout that then get version
• 34:11
  controlled and iterated on is you can add those tests early
• 34:16
  and you can make assertions about what business capabilities
• 34:19
  you want this thing to be able to perform early on.
• 34:21
  But yeah, go into the test. And like how do you see those
• 34:25
  working or what have you seen for the test that can be
• 34:31
  valuable ways of creating them, using them, doing them, all that
• 34:35
  stuff. Besides, I know we've talked
• 34:36
  about evals and you have your evals.
• 34:39
  And so I guess you have a whole bunch of use cases or evals that
• 34:45
  you're looking at and you want to continuously be adding to
• 34:49
  those evals as you find new use cases or new new ways that your
• 34:55
  users are interacting with the feature.
• 34:58
  But what else are you seeing? Yeah, So I can describe.
• 35:02
  A bit about kind of my experience of building this
• 35:06
  Helix test feature and, and some of the learnings from that.
• 35:10
  Now I'll start by saying we're, we're not trying to replace
• 35:17
  something like deep eval. There's lots of great evals
• 35:20
  tools out there that are really strong for doing general purpose
• 35:24
  evals and have all sorts of suites for things that they can
• 35:27
  handle. Is the model label or is the
• 35:29
  system able to remember things that were said at the start of
• 35:31
  the conversation later in the conversation?
• 35:33
  And and that's awesome, but what we're doing with the tests that
• 35:37
  we're putting into AI spec is making it is like having this
• 35:40
  hyper specialised evals piece specifically for testing the
• 35:46
  features that are enabled by the AI spec itself, which is the
• 35:50
  knowledge piece and the API calling.
• 35:53
  Because. And I'll just tell you about the
• 35:55
  actual experience I had of building this Jira integration.
• 35:59
  The whole reason that we started down this road of, of adding a
• 36:05
  limited evals feature to, to our product was because we were
• 36:10
  talking to a prospect, a large bank in the Middle East.
• 36:13
  And they, they said to us, we want, we want Jira integration.
• 36:18
  Can you build that and show it to us?
• 36:20
  And we're like, OK, we'll have a go.
• 36:22
  Give us a week. Yeah, I said it was.
• 36:24
  Thursday I said we'll get it to you by Tuesday, which was a
• 36:26
  stupid mistake, but but you know me, you've worked with me.
• 36:29
  Some things hasn't. Haven't changed at all exactly.
• 36:33
  So anyway I. We, we were like, OK, let's
• 36:36
  build this Jira integration. And the, this prospect also gave
• 36:39
  us this like list of things they wanted the Jira integration to
• 36:42
  do. And I was like, this is a
• 36:44
  perfect use case for for building this kind of basic
• 36:48
  evals piece that we wanted to add because the questions were
• 36:52
  things like what? Issues are there.
• 36:55
  And the most interesting one I thought was like write code for
• 36:57
  this issue. So suppose you have a front end
• 36:59
  developer and you want to give them a cheat code which is like
• 37:03
  they can start work on that issue by talking to the system
• 37:07
  that will you can say write code for issue like DS9 and it will
• 37:12
  go and retrieve the issue and then start writing code that
• 37:14
  will solve it. So get the user start or get the
• 37:17
  developer started on on being more productive.
• 37:20
  I was like OK so this is super interesting.
• 37:23
  I plugged in the JIRA like open API spec and started testing it
• 37:27
  and of course it doesn't bloody do what you want it to do, so
• 37:30
  why would it be easy? What's that?
• 37:32
  Why? Would it be easy?
• 37:33
  Why would it be easy exactly? Fortunately we've got all these
• 37:36
  fields in the AI spec that allow you to customise the prompting
• 37:39
  that goes into the API calling, but I'll tell you why it didn't
• 37:41
  work out the box. It didn't work out the box
• 37:43
  because a big part of the JIRA API is that you have to send it
• 37:47
  JQL which is the JIRA query language.
• 37:50
  So the JIRA has invented this whole like SQL ish syntax for
• 37:54
  how you search over your JIRA issues.
• 37:56
  And the models that I was using even though I was using like
• 37:59
  Llama 3.1 and I tried Quan and I tried a bunch of these up
• 38:03
  because we do have they were not versed.
• 38:05
  In JQL they didn't. They were not well versed.
• 38:08
  In JQL. Or they did.
• 38:09
  They weren't able to just figure out from the open API spec that
• 38:12
  they needed to write JQL and somehow recall how to write JQL
• 38:16
  and how to handle all these different cases.
• 38:19
  So at that point I was like, it's time to write this eval
• 38:22
  system I've been wanting to write so that I can build the
• 38:25
  JIRA integration. Because what you really want is
• 38:30
  you want to be able to iterate on the prompting that goes into
• 38:32
  the making these API calls and know that.
• 38:35
  So I had sites these 7, seven different like cases.
• 38:38
  I wanted to write 7 different tests.
• 38:40
  And if I'm changing the prompting to try and get show me
• 38:43
  overdue issues working, I want to know that hasn't broken what
• 38:46
  issues are assigned to me, which was one of the other cases.
• 38:49
  So it was a really good opportunity to to put this
• 38:52
  together. And then the other thing, the
• 38:54
  other signal that we had for might be useful to build out
• 38:57
  this like mini evals thing for these specific AI spec features
• 39:02
  of knowledge and API integrations is that we have
• 39:07
  this wonderful customer in Germany called AWA or Ava
• 39:11
  networks who are brilliant. They've been with us since the
• 39:14
  very beginning and they were doing exactly the same thing on
• 39:17
  the app that they were building with Helix.
• 39:19
  And their app is one for, it's a natural language interface to
• 39:23
  renting heavy machinery. So if you have, if you're like
• 39:26
  on a building site and you're driving from like Hamburg to
• 39:29
  Berlin and you're a building site manager at the moment, you
• 39:33
  phone someone up and you say, have you got a crane that can
• 39:35
  handle 3 tonnes in Berlin next Thursday?
• 39:38
  And So what this company is doing, they're they're doing a
• 39:41
  natural language interface for that so that this company can
• 39:44
  scale their sales team. So it's exciting news case and
• 39:48
  so just a little side. Note for the JIRA integration
• 39:52
  specifically, you are writing one big prompt that is trying to
• 39:57
  encapsulate the seven seven different use cases almost.
• 40:02
  So the way it actually works, and it's quite nice to, we've
• 40:05
  got the time probably to dive into how does the API
• 40:09
  integration work. It comes in three-part three
• 40:11
  parts. So there's a classifier, a
• 40:15
  request builder, and a response summarizer.
• 40:19
  Those are the three bits and I'll explain why.
• 40:21
  So the classifier, it takes the user's query and it just
• 40:25
  decides, do I need to run an API call based on the user's query?
• 40:29
  And that's, that's a lot like function calling in the open
• 40:32
  AIAPI. But actually we didn't use
• 40:34
  function calling in the implementation because we
• 40:36
  started before function calling was even available in the open
• 40:38
  source models. So actually we just do this all
• 40:41
  with prompting and asking the model nicely to out for Jason,
• 40:44
  which actually works quite well though.
• 40:46
  So the yes, there's these three pieces in the API integration,
• 40:49
  right? There's we call it is
• 40:50
  actionable, which is this classifier.
• 40:52
  If the user is just asking what's the capital of Paris,
• 40:54
  sorry, the capital of France, then then you don't need to call
• 40:58
  the product catalogue API to get the answer to that question.
• 41:01
  So the classifier just determines like basically which
• 41:03
  API to call, if at all. Are there any APIs that we want
• 41:07
  to call then the request builder?
• 41:09
  So yeah, go ahead. It's always going to go.
• 41:12
  To the LLM no matter what. And the classifier isn't
• 41:15
  classifying different LLM calls, it's just classifying which tool
• 41:21
  it's going to use, correct? So the way you can think about
• 41:24
  this is that there's one LLM call into let's just say Helix,
• 41:31
  like the the system that's processing the AI spec, that's
• 41:34
  providing the AI spec compatible job, AI spec compatible app.
• 41:38
  There's one API call that comes in from the user and that ends
• 41:41
  up spawning like sub API, sub LLM call.
• 41:44
  So there's one LLM call that goes into the system from the
• 41:47
  user and that ends up spawning like sub LLM calls that go and
• 41:51
  do these like intermediate jobs that I'm describing call comes
• 41:54
  in from the user. Like I said, everything is open
• 41:57
  AI compatible API. Big believers in that.
• 41:59
  We'll classify the request. Is this request that that
• 42:04
  requires an API call at all or is the user just saying, hi,
• 42:08
  how's it going? My name's Bob.
• 42:10
  Hey, great to meet you. I don't need to call the product
• 42:12
  catalogue API yet. And but then if it does require
• 42:16
  an API call, then the second prompt is construct the API call
• 42:20
  for me. And that's actually a bit more
• 42:22
  interesting because it's OK. The in the first classifier is,
• 42:26
  oh, I might have three different AP is I can call and I know
• 42:28
  their description. So based on the user's question,
• 42:31
  I can figure out which API to call.
• 42:33
  Then the second prompt is saying oh in the first prompt I figured
• 42:38
  out that I want to call the product catalogue API.
• 42:41
  But now I've got a more interesting job which is based
• 42:44
  on the Open API spec, IE the Swagger spec for the definition
• 42:49
  of that API. How do I construct the API call
• 42:52
  that will answer the user's question?
• 42:55
  And the output of that sort of sub LLM call, if that makes
• 42:58
  sense, is a Jason object that describes back to Helix how to
• 43:04
  construct that API call. Helix then goes and actually
• 43:07
  makes the API call. The API call comes back and then
• 43:10
  the third piece is summarising the response from the API.
• 43:14
  So the API gives you a chunk of Jason back.
• 43:16
  You don't want to just slap a chunk of Jason in the user's
• 43:18
  face. So then the job of that third
• 43:21
  piece is to turn that back into English or natural language and
• 43:25
  say, oh, here's a nice summary of the response.
• 43:27
  OK, so walk me. Through.
• 43:29
  Now you're doing that with JIRA and.
• 43:31
  I say. Give me a summary of all of my
• 43:35
  issues. And so that when I say that
• 43:39
  first it goes to Helix and Helix then says, OK, this needs to go
• 43:46
  to JIRA. It will hit another model that
• 43:50
  says, how do I construct an API call to JIRA?
• 43:54
  And the other model will say, funny enough, JQL is your
• 43:57
  ticket. Here's how you use JQL for a
• 44:02
  JIRA call. It sends that back to Helix and
• 44:04
  then Helix will send the API call with the JQL to the JIRA
• 44:10
  API. That's it.
• 44:11
  It gets what it needs and it sends back Jason to Helix.
• 44:16
  Helix then says OK, now LLM call again to explain to me what the
• 44:24
  hell is inside of this big ass Jason file.
• 44:26
  That's right. So and then yeah, it goes back
• 44:29
  to the user. Exactly.
• 44:30
  Exactly. And so that all happens like in
• 44:32
  a couple 100 milliseconds. I was going to say in like a few
• 44:36
  minutes. It's just that happens in three
• 44:38
  to four minutes. No, it's pretty quick.
• 44:40
  Like we use the AB models for this stuff, actually.
• 44:42
  They're nice and fast and yeah, but from the user's perspective,
• 44:47
  they just ask. Have you?
• 44:47
  We've got any cranes, We say, yeah, we've got 7 different
• 44:50
  cranes you can rent. Which one would you like?
• 44:52
  So it's like it's nice and natural from the user's
• 44:55
  perspective. But you're right, that's exactly
• 44:56
  what's going on under the hood. Wow.
• 44:58
  OK. And now the piece that I, I
• 45:02
  think spawned this whole little conversation that we're having
• 45:06
  is the prompt that you are doing to make sure that it's covering
• 45:15
  each one of your use cases or your test cases.
• 45:19
  You're basically testing it on each one of these LLM calls.
• 45:25
  So I can now so. I can now answer your previous
• 45:27
  question, which is like what are the prompts that go into the
• 45:31
  system? And the answer is that there's
• 45:34
  three of them, basically the the prompt for the classifier.
• 45:38
  And all of these can be customised in the areas in the
• 45:41
  AI spec and you find yourself having to customise them when
• 45:44
  you do any non trivial integration like the JIRA
• 45:46
  integration. So the is actionable prompt,
• 45:48
  like the classifier prompt that can be changed because you might
• 45:51
  want to give the LLM more information about when to choose
• 45:55
  a certain API or not or give it like synonyms for things that
• 45:59
  the users often say that it might not figure out on its own
• 46:02
  and things like that. Like for which API call to make.
• 46:05
  Then there's like how do you construct the API request?
• 46:09
  So then we have the prompt for like how do I construct the API
• 46:13
  call and of. And of course, if the model
• 46:15
  doesn't know how to write JQL, then that's the prompt where you
• 46:18
  put where you teach it, you teach about JQL.
• 46:21
  And then there's the prompt that's used for summarising the
• 46:24
  response back to the user. And that's super useful for
• 46:26
  customising the format of the response.
• 46:28
  And for example, for this customer in Germany that their
• 46:31
  app, all the prices are always in euros.
• 46:34
  And if something comes back from the API, then you have to say
• 46:36
  that it's available. I remember that off the top of
• 46:38
  my head because I had to customise those prompts for that
• 46:41
  app because the API doesn't say the prices are in euros, but you
• 46:44
  want to quote them to the user in euros.
• 46:46
  So you can put all this like useful tweaking and finessing in
• 46:50
  in the way that the interaction works in there.
• 46:53
  So yeah, you've got these three different prompts you can edit.
• 46:56
  And now the question is, how do you stay sane while you're
• 47:00
  trying to make all of these different cases work?
• 47:02
  And the answer is you do test driven development just like you
• 47:05
  do with software, but you're testing the output, the final
• 47:08
  output, correct? And that's the other.
• 47:09
  I can now answer your other question, which is are we
• 47:11
  testing the individual prompts or the whole thing?
• 47:13
  And it's an integration style test.
• 47:15
  So we're testing the whole thing.
• 47:17
  And so the, the tests I'm looking at them here on, on my
• 47:21
  screen, they say something like test Jira issue search.
• 47:25
  The steps are that the user asks what issues are there and the
• 47:30
  expected output is a list of nine issues with brief summaries
• 47:33
  and or details. And that's it.
• 47:35
  We're just writing these test cases in natural language as
• 47:37
  well. And then we're using the LLM as
• 47:40
  a judge in order to to take the response back from the system
• 47:44
  and then tell you whether it's good or not.
• 47:47
  And this is how you do CICD for Gen AI open case models.
• 47:51
  Yeah. You're because basically that LM
• 47:54
  is a judge is coming in when it is summarising the Jason that
• 47:59
  it's getting out of the API on that last call, correct and so.
• 48:05
  I'll give you a concrete example.
• 48:07
  So I'll give you a couple of concrete examples because they
• 48:09
  might, this might be interesting for like how we actually built
• 48:11
  this, you know how I built this, this JIRA integration.
• 48:14
  So let's begin with, I plugged in the open API spec and I
• 48:17
  started asking you questions like what issues are there?
• 48:21
  What? Issues are there that are
• 48:21
  assigned. To me, what overdue issues are
• 48:24
  there? I was just giving me garbage and
• 48:26
  I looked into it and I looked, I clicked the debug button inside
• 48:29
  our test thing and like it shows me all the internal LLM calls
• 48:33
  like the three steps that I described and I could see it's
• 48:36
  generating garbage JQL. Like this thing doesn't know how
• 48:39
  to write JQL. And so I started Googling.
• 48:43
  I think I might have asked Claude to be honest, but I
• 48:45
  started looking up like, oh, how do you write JQL?
• 48:48
  And then I started adding to to the second prompt, which is that
• 48:53
  one where you help the model construct the API, the API car.
• 48:57
  Here are some example and it literally says in the prompt I'm
• 49:00
  reading out examples of how to specify JQL and I say what
• 49:04
  issues are there? Goes on to the empty string.
• 49:07
  If you just send no JQL at all it will send you all the will
• 49:10
  return all the issues. Get issues assigned to me maps
• 49:14
  onto assignee equals current user, open bracket, close
• 49:16
  bracket. Write code for issue DS9 you
• 49:20
  start. You need to look up DS9.
• 49:21
  So you do key equals DS9 and then show me overdue issues.
• 49:26
  The JQL for that is due date is less than start of day.
• 49:29
  Open bracket, close bracket. So I was like, OK, I can
• 49:33
  actually. And the way I wrote those was I
• 49:35
  wrote a failing test. I ran the test suite and the LLM
• 49:40
  as a judge like right and cheques all of the answers.
• 49:42
  And it says this this failing test for show me overdue issues
• 49:45
  isn't giving you the right answer.
• 49:47
  And then I can go in and inspect the JQL check that it's not
• 49:49
  giving you the right thing. And I can just add to the list
• 49:52
  of I've taught you how I'm teaching you how to write JQL.
• 49:55
  So I'm just going to add another items that list.
• 49:57
  And then the really nice thing about the satisfying thing about
• 50:00
  that is that you then run the tests again and you see they
• 50:03
  pass. And now you know that they're
• 50:05
  always going to get checked and that that use case is always
• 50:09
  going to carry on working in the future, even if someone else
• 50:12
  goes in and changes the prompting for something else to
• 50:14
  enable some other use case. Because, and this is how we do
• 50:18
  CICD, you don't merge a failing pull request to Maine, right?
• 50:24
  If you've got a pull request and there's tests failing, then in
• 50:27
  general the policy and engineering teams is unless it's
• 50:30
  a super emergency, you don't merge the pull request to Maine.
• 50:33
  Yeah, I wouldn't know anything. About that I just forced push
• 50:36
  fine so. Do I this is and then I.
• 50:42
  Throw in slack that this is fine dog.
• 50:47
  I don't even do that. No, I don't force push, but I do
• 50:50
  commit to Maine sometimes. But I'm a very bad person and I
• 50:52
  should feel a shame.