Latest / The Ruby AI Podcast / Evaluating LLMs with Leva

Transcript

Evaluating LLMs with Leva · The Ruby AI Podcast

• 0:00
  Hey everybody, welcome to another episode of the Ruby AI podcast. I'm your host today, Valentino Stoll, and we have a very special guest today, Kieran. And, you know, we had you on, Kieran, to talk about that, but you're also very prominent in the Ruby AI space as well.
• 0:31
  I was not aware of this until I saw this talk.
• 0:34
  You are outspoken about a number of AI development best practices, even.
• 0:39
  I've even taken on your Git workflow, Worktree workflow, and it works really well.
• 0:46
  But yeah, why don't you introduce yourself?
• 0:48
  Tell everybody where you started with this and what you're up to.
• 0:53
  Thank you for the invite.
• 0:54
  Yeah, very glad to be on there.
• 0:56
  I'm very passionate about Rails.
• 0:58
  I'm very passionate about AI.
• 1:00
  And I've been building AI products using Rails for the last four years now.
• 1:09
  And yeah, I've seen the ugly and the hard parts and would love to share a little bit about that, what I've learned over the years.
• 1:17
  And Leva is one of those tools that came out of it.
• 1:20
  I'm building Quora Computer, which is a tool to hand off your inbox, your Gmail inbox, and Quora will manage the emails that are not super important to respond immediately, brief them twice a day and summarize them.
• 1:37
  And everything that is important, which is mostly 20, 10% of your emails that you receive will stay in your inbox and you will feel great because you look in your inbox and it's emptier.
• 1:49
  So I'm building this tool and obviously you need to run evals to see whatever you do is working.
• 1:56
  And in this case, the first thing was categorization and you start to look around and yeah, there is nothing for reals.
• 2:06
  Like you just do a gem search and it's like, okay, there has to be something.
• 2:10
  Andrew Cain probably has to have made something or someone has had to made something and it didn't really exist.
• 2:17
  And I did Python in the past as well.
• 2:20
  There are great tools for Python.
• 2:22
  Like I used them and I was like, well, it's not too hard so why don't i just build this and many of the tools i've built for myself are just like needs like i have a few gems that i just wanted and no one else made them and most of the time it's finding like a needs to solve for, and I just do it.
• 2:46
  And the beauty with AI is also like, it enables me to do this actually in a week instead of like a month.
• 2:53
  Because before I've done gems and they're like really small, but it takes a lot of time.
• 2:56
  And if you have an idea and a problem, now you can either build it in your own app, or you say, well, maybe I spend one day extra, make the nicer abstractions and just create a gem.
• 3:07
  So yeah, that's kind of how it came to be.
• 3:10
  And I would love to talk about any obstacles or things I solve building with AI because it's really fun.
• 3:18
  I love seeing this.
• 3:20
  Honestly, I've built almost something exactly the same thing as this internally before. It's great to see this out in the public and honestly, really well designed.
• 3:30
  So I'm curious what your opinion is of the gem creation process with a lot of this AI stuff.
• 3:38
  Because personally, I've had a good experience. I'm curious where you find it works really well and how it's freeing up time for you like in that way it's a good question because you can build internal things and normally like this is how i used to do it before ai i always did it internally and maybe i had a lib file or some abstraction already internally in the app and then oh i will move this to a gem later and then you never move it to a gem later because it works there's no reason to move it and it's like there's just a little bit of friction to do it because suddenly yeah like you work with two say two places and you need to test and like in a test app and a dummy app you you have all of that. So what I've learned is like, if you decide to make a gem, just make a gem.
• 4:28
  Because if you start out with already a dummy app and everything, it's actually easier to go than like abstract something out and having to do all of this thing.
• 4:39
  Because when you start out with an idea, you have energy energy you're passionate and motivated to do it and after that like why are you motivated to create a gem like for me creating the gem is really like i want to reuse this into multiple projects i do multiple other side experiments because I just want to see how to build and what works and having a gem is nice because I can just add it and it works so yeah like how it worked the Leva gem particularly is heavily inspired by other libraries so first of all I dream of what is my ideal readme. So I start with readmes. I always write my readme first and think like, how would it look like? And then I just take a walk and think about, did I miss anything? What is the API? Did I miss use cases? And I refine this readme until I'm like okay this is complete enough and it's about finding abstractions and finding an API that is like I like and this is just taste I build it for myself so like whatever I like a certain way which for me needs to be modular but it needs to bring actual solutions and I wanted to use it with whatever I want.
• 6:06
  I don't want lock-in on a library or lock-in on like open router or I just want to have more of an adapter like where I can just run whatever I want to run.
• 6:18
  So it's like having this freedom, but it should solve real problems for me, which is I want a UI because I'm visual.
• 6:27
  I want to have a UI on production that can work with encrypted data, similar to like I was inspired by the workbench from Anthropic there, where it's like you have the prompt and the variables and you can kind of debug. And in the end end all these things come together and i just sketch out things write a readme and just start building and the process of building back in the day was sonnet 3.5 and composer in cursor which was amazing if you look at it now you're like it's so old-fashioned but back in the day was like oh it can create multiple files wow all right so another huge leap in advance exactly but in the end evaluations are not very complicated the complicated part is like how do you design your models so that you have the data and store the results?
• 7:27
  Like, it's not very complicated.
• 7:29
  You just run something.
• 7:30
  Is it the same?
• 7:31
  Yes or no.
• 7:31
  And that's basically it.
• 7:34
  So that's where it went.
• 7:36
  And yeah, I just shared it on my GitHub.
• 7:40
  I never thought anyone really would look at it.
• 7:42
  But like at some point I got a text from chris oliver he said hey you're in you're in the talk and some kind of talk he was presenting in the talk in in brazil i think and irina was talking about my jam you're like oh she's talking about your jam i had no idea so she kind of figured she found it somewhere and she was talking about ai and ruby and how we should promote like a community work and yeah so that's when people started to like it and use it and it went from there so like the goal is really just to build something really useful for myself but i'm very happy if people yeah that's how it should start. And that's really the philosophy. So, and then I'm very happy if other people can use it as well.
• 8:28
  Yeah, I've been using it. We have this little app that I like to build with all of the things that we invite guests on with that just manages the podcast. And so I've been using it just like as a way to like analyze the prompts and agents that we have for doing that intake and yeah it works really well evaluations that i've done you know it's langsmith that's like the staple everybody starts with it's connected everybody yeah it's python it has an api you could submit your requests there yeah and also with privacy and encrypted information like it's hard to do that. But yeah, like absolutely.
• 9:05
  It's inspired by that.
• 9:06
  Yeah, totally.
• 9:08
  And so looking at this, it's very similar in nature, right?
• 9:11
  Like where you just have your experiments and then you can run your individual evaluations against different experimental data sets.
• 9:19
  And, you know, it has a very similar feel.
• 9:22
  That's the problem with a lot of Rails apps. Like you mentioned, the, you know, privacy and security, where you have all of your data in the Rails app, and how do you get it out and make use of it, right?
• 9:34
  To make sure that you're getting your snapshots and making sure you're consistently executing things, right?
• 9:41
  With all of this disparate systems.
• 9:44
  All of the changing results that you get with llms right so i'm curious like i imagine that you've used langsmith before or some similar evaluation tool how has your personal like transition been since keeping it in your stack tech stack now is it a lot better are you finding things that you know you're missing out on that you now have extra work because you have to make it in leva right so langsmith i love it's great but for me like i work with encrypted data in a database and like i just cannot use it i'm not allowed to use it because of privacy policies and security audits like it's just not possible to do so is it better yeah because i can use it instead of not use it and is it like the most amazing vc funded evaluation framework that's super slick no that's not what it is. But think of it like Blazor from Andrew Cain, for example. Is it the best, most amazing business intelligence?
• 10:53
  No, but it just solves the problem. And it does it in a way that's not super complicated. It's pretty simple, so it's easy to learn learn and you can always move to another service or get another pipeline set up to do evaluation and like i'm sure you can do that but the whole point is like i was the only developer for seven months and i just needed something so it's very practical and that's where it came from. Like, does it solve my problem?
• 11:25
  Yes.
• 11:26
  And the fun part is that it lives really close to the production data.
• 11:31
  So you can do really cool things with it.
• 11:33
  You can just go into your app, see something, and you're like, oh, that's wrong.
• 11:37
  And then I have a button that says debug and leave.
• 11:39
  And then it just preloads the data into a data set, loads the correct prompts and the runners and everything.
• 11:48
  And I can just start jamming on the prompt.
• 11:51
  And I think reducing friction is very important because if I had to first figure out the whole context, which is very hard because it all lives on production.
• 12:01
  And then like copy paste variables from here to there and go into another tool if it was like that i wouldn't have done it so anything to reduce friction i would say is very nice so that's the experimentation side in addition to the evaluation side but yeah i i've many other ideas i want to also add to leva that i think are very useful that I'm experimenting with now.
• 12:25
  It's hard to show this because there is, like you mentioned, a UI to this, which is kind of critical to the whole process is being able to visualize these, like what is happening, right? Before we dive into that, though, I loved your talk in San Francisco, where you basically made an evaluation to observe like business metrics on like what was it AWS cost management or something like that and that to me is like maybe something that isn't talked about enough is what to evaluate from LLMs in general like where do you start measuring things what do you measure what are your like critical components when you're like building something with a new prompt or a new agent or whatever it is that you're like wrapping?
• 13:11
  Yeah, that's a good question.
• 13:12
  So you can do this with any framework, but how I evaluate is first of all, you just create a prompt and you just jam on it and you see it doesn't work.
• 13:24
  Doesn't not work?
• 13:26
  Like, let's take an example of categorization.
• 13:28
  This is a simple example, but also what I do with emails.
• 13:32
  Is it important enough to stay in your inbox or should we brief it?
• 13:37
  Like, it's a very important decision.
• 13:39
  And you can start with a prompt.
• 13:41
  Hey, if it's urgent, then put it in my inbox and otherwise just breathe it.
• 13:45
  Like that is the prompt.
• 13:46
  But how do you know that works?
• 13:49
  Well, you go insert an example email and click run.
• 13:53
  And then it says yes or no.
• 13:56
  And you're like, oh, it works.
• 13:58
  But then you start collecting real world emails.
• 14:03
  This is local.
• 14:03
  But if you deploy, suddenly you have emails that say promotion urgent.
• 14:08
  You have two days left.
• 14:09
  And suddenly the promotion is also urgent.
• 14:12
  And you're like, dang, okay, this prompt does not work.
• 14:15
  And the whole point is then adding that email to a data set and say, oh yeah, but if it looks like a promotion that says urgent, maybe then it's a promotion. So that's refining the prompt and then you run it and then suddenly your two emails say, yep, it did it. It's really important to see your prompt work. And how I think about it is like for real developers, test-driven development or like writing tests for your codes just ensures your code works. And you need to do the same thing for prompts. Actually, you need to do it even more because prompts are not deterministic. So they can do whatever they want. And it's more important to add tests to them. And evils are just tests for prompts. So if there is logic embedded in the prompt, which most of the time there is some logic, sometimes it's summarization only, but lots of the time there is some logic or decision-making.
• 15:16
  And you want to see if it works.
• 15:18
  And it's very important to do that at a larger scale where you have more examples of actual real data.
• 15:26
  And you could even do that on like development where you already ask an LLM, hey, create synthetic data for like all these things.
• 15:35
  What could go wrong?
• 15:37
  And really push for that.
• 15:38
  And you can already start with a little bit of a better prompt before you even go live.
• 15:44
  So that's kind of the framework and that's one example and another one that's very important is you have a prompt that works that is on production but google launches a new model or anthropic launches the new model and the new model is supposed to be way better and how do you know? How do you know if it's better? And if you have evals in place, you can just run it.
• 16:09
  You just say, okay, run the new model.
• 16:12
  How much better is it?
• 16:13
  8%, 2%, 6%?
• 16:15
  And that's very useful.
• 16:17
  And the flip side is someone launched a way cheaper model, but it's also worse, maybe. How do you know?
• 16:25
  Well, same, you just run the model.
• 16:28
  And my example was, so we use Haiku.
• 16:31
  I think Haiku 3.5 for categorization for a long time, but then Flash came out, Flash 2.0, and I just wanted to see, because Flash was like 10 times cheaper.
• 16:43
  Way cheaper, like 10 times cheaper, way cheaper, like 10 times cheaper is a good amount of cheaper for millions of emails.
• 16:49
  So I just ran the test, the evaluation for my categorization, like how much worse is Gemini?
• 16:56
  And Gemini was 6% worse.
• 16:59
  I think it was, and I was like 6% worse for 10 times cheaper is great.
• 17:10
  That was like 150K per year saved by just making that switch.
• 17:15
  But it's very scary to make that decision and just YOLO it. You should not just switch the model and hope for the best.
• 17:19
  You need to know what you're doing.
• 17:21
  So it will allow you to move faster with more confidence, similar to tests.
• 17:27
  If you have a passing CI, CD pipeline, you can ship faster.
• 17:32
  So it's a very similar story for LLAMs.
• 17:35
  It's interesting you tested model changes because in my mind, I feel like prompts are almost coupled to the models that they use in a lot of ways. Like the models are trained for specific formatting and some handle some domains better than others.
• 17:53
  So I'm curious what your experimentation process is like for those aspects.
• 17:59
  Do you also couple those model changes with some like prompt tuning, or do you like to keep those separate? What is your workflow and reasoning maybe for like deciding those aspects of it so first of all i want to try just same prompt same everything don't change anything like a different model how does it work differently because you learn something from it. It might be even that more expensive model is worse than a cheaper model.
• 18:28
  Like, for example, if you use a cheaper model, normally you need to have more direct prompting and repeating and silly things.
• 18:36
  If you don't do this, you go to jail kind of hoops.
• 18:39
  But also the newer models are actually converging a lot and it's less and less and less.
• 18:46
  So models are actually coming pretty close.
• 18:51
  They all do XML well.
• 18:52
  They all do JSON well.
• 18:53
  They all follow directions pretty well.
• 18:56
  And with Flash 2.5, for example, the difference between Flash 2.5 and Sonnet 4 in certain cases is not that big.
• 19:09
  Yeah, there are very different models, but if you just do categorization and you have like a chain of thought and prompted, they're not that different.
• 19:17
  They're not like the price difference.
• 19:19
  Like I say, like Flash 2.5 was ten times cheaper, but the other is not 10 times better.
• 19:30
  So I always start with that, but I do improve prompts as well. And actually, I used to write prompts myself.
• 19:34
  I don't write prompts anymore.
• 19:36
  What I do is I have Opus rewrite my prompts.
• 19:40
  And how I do this is I use Leva as well.
• 19:44
  So I just say to an agent in Cloud Code like hey this is the prompt this is the input this is the data set I want to run can you run it and look at all the failures and see why they fail analyze the chain of thought so I always include the chain of thought because then it's easier to debug and can you rewrite the main prompt yeah until it runs always correctly.
• 20:07
  Or run it 10 times and make sure that all the 10 times I run this, it's actually giving the same output.
• 20:15
  Because sometimes you run it once and it's green and it passes, but the next time it's not.
• 20:19
  So like, it's very important to run it multiple times.
• 20:22
  So I don't write it as much anymore.
• 20:25
  Like I think AI is better at writing prompts now than humans.
• 20:30
  And using the Anthropic Console prompt improver, that's always how I start.
• 20:36
  So I just start jamming what the prompt should do, then put it in there.
• 20:41
  That's the baseline.
• 20:42
  And adding real world data and see if it actually works or not. There is some sale craft to it where I do add things, but it's less and less.
• 20:53
  I do it less and less. Yeah. In the early days, I definitely started out asking, asking it to redo my prompts and, you know, with having mild success, but it definitely has gotten better now i agree with you have you tried any of the tools like some of these d spy frameworks to do the prompt optimization based on your runs or anything like that a little bit yeah a little bit i just want to keep things simple but yeah i'm thinking about including a similar thing in Leva as well, because there is data and it makes sense to improve prompts automatically and track it.
• 21:31
  So it is a very natural place to put it in.
• 21:35
  Yes.
• 21:36
  You mentioned using cloud code.
• 21:39
  I'm always tempted to just hook up the web appCode and see like what kind of cost that generates.
• 21:49
  Yes. It's very tempting. I know it's maybe not the most secure thing to do, but you do some like kind of wild stuff with PlotCode. I'm curious, what have you found like you mentioned like repeating, you know, runs and stuff like that. What do you find the most optimal use cases specifically for Ruby and Rails development while you're going and churning? Where do you see it giving you the most benefit? It can do so many things. It's very powerful. It can do many things, yet it's also a tool that you need to learn.
• 22:33
  And really where I see the most power is it's on the intersection of writing prompts, testing prompts, and writing the code, and getting data from real runs runs and iterating on that.
• 22:48
  Like it's that kind of work all together is insane.
• 22:53
  Obviously, it's great for research and creating issues in GitHub and then implementing those and adding tests and reviewing pull requests. like it does all of those things as well it's basically my intern and the intern is pretty legit but really the powerful part like if you're building like i don't know if everyone is building with ai or for ai or both like i think it's really special when i call this kind of compound engineering or like compounding engineering where you have a problem and you don't solve the problem, but you solve the meta layer, the meta have a bug and I need to debug it.
• 23:45
  Either you go debug and do the bug or you write a prompt that then can run and debug.
• 23:53
  And then the next time you need to debug something, you just do the prompt.
• 23:57
  So investing in like the meta layers and the compounding things, I think that's where it's very special and people should try to do that more.
• 24:08
  You should not use cloud codes to debug.
• 24:11
  You should figure out how it can debug the next time for you.
• 24:16
  That is where it's really special, where you can really build, yeah, like compounding effect.
• 24:22
  And it's a similar thing with using an ai to write a prompt for ai like it's so cool and having itself improve and just sit back sip coffee and it just says oh i'm running this now and like four out of ten times it failed let me look at the chain of thought and then having four sub agents look at every single run and then synthesize it at the chain of thought. And then having four sub-agents look at every single run and then synthesize it at the end.
• 24:48
  And I say, oh, it got tripped here.
• 24:50
  So maybe we should not mention CEO or blah, blah, blah there because it removes it and then run it again.
• 24:56
  Let's see.
• 24:56
  And just going like that, I think that's so magical and it's so powerful.
• 25:02
  And I wish everyone in the world tries that out because if you can get there that's a true hook or aha moment for like this is amazing yeah you're so right i liken it a lot to learning how to use the command line in general yeah there are just like so many commands and so many commands you can make and aliasing and all this stuff that's just available that you just don't know and it's like it takes some getting used to like how the piping works and similar to loms like all these little tricks of the meta aspects of like the lm can do all these things the command line can do too you like compound it yeah it can get overwhelming also because the workflows can change.
• 25:46
  You'll find another workflow that works better for you.
• 25:48
  And like, I'm loving how it's like truly getting creative where you're like, you can like make the workflow just work for how you work.
• 25:56
  Yeah, absolutely.
• 25:58
  Yeah.
• 25:58
  It's so flexible.
• 25:59
  And like, you don't have to do it how anybody else does, right? It's also very overwhelming and very scary to a lot of people because people do like their habits and they are comfortable a certain way.
• 26:12
  And you do need to have a certain mindset to say, actually, I am not using an IDE anymore.
• 26:19
  I'm going to use the terminal.
• 26:21
  Like some people use the terminal their whole lives.
• 26:23
  But like for me, I'm a visual person.
• 26:25
  So just giving up a visual thing and going into the CLI in a terminal, like, yes, I know the terminal, but it's just where my stuff runs or I don't have Tmux and Vim and whatever stuff.
• 26:39
  I don't do that, but I am more comfortable now with Cloud because yeah, you need to experiment. And if i am more comfortable now with claude because yeah you need to experiment and if you get more comfortable using it and seeing what it does and how it works you can truly utilize all of these things that are in your terminal already that i have no idea about but yeah it's really cool because it can learn all these things and it can use all those things and it's very powerful because yeah what you say yes you could if you have an app that's not very security prone you could just say oh yeah you can ssh into this machine and debug the production data like why not because that's what i would do either way then it It's funny, all this hype of Docker almost seems like Docker was made specifically for like gearing up for this moment.
• 27:30
  We can just like, you know, I'll spin up a container and just run this thing like in isolation, right?
• 27:35
  And if it destroys itself, who cares?
• 27:37
  Yeah, you can do whatever you want in this environment.
• 27:39
  Just do something.
• 27:41
  Yeah, absolutely.
• 27:41
  Yeah, you can do many things.
• 27:43
  It's very overwhelming, but also very, very fun.
• 27:46
  So what I encourage is just to experiment and see what works and also follow people that do cool experiments.
• 27:56
  There are so many people on X that just do amazing things.
• 28:00
  And I'm like, oh, or read some documentation on Anthropix website.
• 28:05
  They have very good documentation about how to use cloud code as well.
• 28:09
  I want to dig into the fun aspect because 100% agree with you there.
• 28:13
  It just unlocks so much when you get to that creative mode.
• 28:16
  But before we do, digging back into the UI aspect of things, because I'm a visual person as well.
• 28:22
  What are you using to visualize maybe your workflows or the evaluations you already have like this great UI for?
• 28:31
  What aspects of the runs or things like that and your workflow, are you using tools to help you visualize those aspects of work getting done?
• 28:41
  Are you just using GitHub?
• 28:42
  What are you finding lately?
• 28:44
  I use GitHub for just the builds. of like work getting done? Are you just using GitHub? What are you finding lately?
• 28:48
  I use GitHub for just the builds.
• 28:50
  I think for workflows.
• 28:54
  So I use this gem called Stepper Motor by Ulic, which was inspired by Heia from Honeybadger, but just made it more multipurpose, which is you create a journey and there are steps in a journey.
• 29:04
  It's basically a workflow.
• 29:06
  And it's really nice.
• 29:08
  So I use that for my more complex LM workflows because he's also working on a UI, I heard through the grapevines.
• 29:17
  So then I can just look at a UI where I can see, hey, my AI workflow is in step there. This is the average for these steps it filled here. And I think I've looked at many, many background process things, but I, I like a stepper motor. So I can recommend looking into that. He's actively working on that.
• 29:40
  I think it's a missing piece. If you are doing some more complex LLM workflows, you just need some workflow management. You cannot just call a job that calls another job that calls another job. It will be a mess. So yeah, and you have some visibility also, like a mission control for your workflows, basically. So that's what I use for workflows now. I just started using it.
• 30:07
  I did job calling a job calling a job before, and it's terrible.
• 30:11
  Yeah, I agree with you there.
• 30:12
  It does make me wonder, like, I'd love to hear your thoughts on like the process of like single agent, you know, spawning other agents.
• 30:19
  Or just like a single threaded agent that does work in steps kind of thing.
• 30:25
  Are you leaning more toward the latter in that way?
• 30:28
  Like breaking down stepwise things, even if they happen in parallel, but like having a main process that manages that?
• 30:36
  It depends for what use case.
• 30:37
  I think you need both.
• 30:39
  Very practical for me, workflows are cheaper.
• 30:43
  I run a business.
• 30:45
  I don't want to pay hundreds of thousands of dollars per year in LM costs.
• 30:50
  Today, models are still too expensive to just have an agent run and do whatever you want.
• 30:56
  Like it is too expensive, even though it is cool.
• 31:00
  So for Quora, I use both.
• 31:02
  I think for engineering, clearly agentic.
• 31:06
  Like I don't think workflow is anything you actually need to do for development because you want flexibility, you want the creativity.
• 31:13
  I use agentic workflows more for like memory extraction or analysis of what kind of personality because you need more creativity.
• 31:24
  You need something that's less generalized and more creative and adapting to what it needs at the moment.
• 31:31
  But if an email comes in and needs a category, it just needs a category.
• 31:35
  Like that is a very fine workflow.
• 31:38
  And also, I don't categorize everything.
• 31:41
  Like I get a tag from Google that says this is a promotion. Well, I'm happily using that tag and the LLM spent and models that Google uses for the promotion to also take that into account.
• 31:54
  So yeah, there's that.
• 31:57
  And there's obviously a hybrid of the two where you have like an orchestration layer where there's an agent that can run subagents and the subagents could actually be just workflows because you already know what the steps are going to be and what you want out so yeah you can have both i think you should familiarize yourself with all but people that say this is the way i don't think there is a way i think it just depends on what you're trying to solve and i know that's like the cloth code team and the devon team the cloth code team and the devon team had like at the same day set opposing things where workflows are the only way it works now and the other ones that agentic is the only way it works and in the end they were like oh, actually this is just more of a clickbaity thing and we all agree.
• 32:47
  And we are inspired by all and everything.
• 32:49
  So they all agree that there's no one way and there's very healthy stuff to be learned from either side.
• 32:57
  Yeah.
• 32:58
  I am also torn on this.
• 32:59
  It definitely works for some workflows and some not.
• 33:03
  Every time I, somebody asks me or I ask somebody else, it does always come back to the same old, it depends response. And I feel like this may be true of just how to work with AI in general. Yeah. I think one thing to add is I think workflows are more deterministic than agent-agentic flows because there are more checks in place which if the goal is you want it to be more deterministic then if you go for a workflow because of that reason i would say think again because agents plus evaluations might also be good maybe the quality with an agent and a good evaluation data set is actually higher than a workflow.
• 33:50
  But if you go for a workflow because you want to control the output, I think that's the wrong reason to go for a workflow.
• 33:55
  You should go for a workflow because it's cheaper.
• 33:59
  You should go for a workflow because it's better.
• 34:02
  It's not because it's more controlled. You should go for an agent because it's better. It's not because it's more controlled.
• 34:05
  Like you should go for an agent if that's better for you, but you need to have an evaluation framework in place because that is kind of what a workflow also tries to do in a way.
• 34:16
  It's like, just make sure that things stay in shape and be controlled, which is the same as an evaluation framework.
• 34:23
  So if you go for that because of that reason, then think again. I'm curious, the fun aspect here. I personally love experimenting with AI and Ruby in general because it's fun. And it's so expressive that it lets you do these wild things that I feel like is a little more challenging in other languages and maybe just not as fun because you can't read it as easily. Right. So I'm curious, like, where is your fun level? What are you working on that's fun or what have you seen that's fun?
• 34:54
  I think the most fun part is that I have an idea or like a thought about a gem that I would wish existed. And you can just decide, okay, today I'm going to have fun building a gem.
• 35:09
  And then at the end of the day, the gem is done and is usable.
• 35:14
  Like for me, just creating that is amazing.
• 35:18
  And for me, the fun part is actually going into the details and seeing how the architecture is like, I'm not vibe coding here i'm just lazy coding but i have a very clear vision on how i want to build gems and how it should be structured but i'm just relaxing looking at it do code a whole gem and i'm like oh we could add this layer here or we could all do all these things there or this abstraction or adapter pattern or like can you research like five ways to improve this and then i'm like oh can i learn something that's interesting like a service object way or adapter or like whatever patterns i'm a self-taught engineer i studied classical composition was a film composer before. That's my background.
• 36:05
  I've never learned any of these patterns or things. I've been an engineer for many, many, many years. So I have lots of experience working with frameworks, but I never really learned these things. So for me, it's fun creating something like that because I learn new things or how things work, or I use it to explore an already existing gem from someone i said oh i really like this gem what is the kind of the design they follow and i just check it out with gits and jump in with claw to say what kind of vibe has this gem or what kind of design language is this i really like this what is it called and then it's starting to explain these things to me so like for me it opens the doors like even to rails or like other libraries like to just dive in the source code more and it holding my hands because yeah i can do it and read it but like it's so much easier with an lm and then say okay i like all these things can you summarize everything?
• 37:05
  What the design pattern is?
• 37:06
  Can you like try to follow these patterns in my own gem and then see what it does?
• 37:12
  So for me, it's so amazing.
• 37:13
  Yeah.
• 37:14
  Yeah.
• 37:15
  How does this work is like one of my most common things just jumping at me like, this is like a 500 line class.
• 37:21
  Like I'm not going to read through that.
• 37:23
  Like I know all the things and yeah just okay well how does this work summarize the main points like how does this work with other classes right like that aspect yeah yeah and if you extract that out to higher level concepts like how does concurrency work and Ruby yeah huge value for sure yeah change it's what you think also fun yeah for me that's fun too like I love creating so for me anything that I can create is a lot of fun value for sure. Yeah. Change it to what you think. Also fun. Yeah.
• 37:45
  For me, that's fun too.
• 37:46
  Like I love creating.
• 37:47
  So for me, anything that I can create is a lot of fun.
• 37:50
  I think the fun part is mostly because it's not forced.
• 37:54
  Yes.
• 37:55
  I love building a company.
• 37:57
  That's also fun, but like I also have to do certain things and if there's a bug, I need to solve the bug.
• 38:03
  So it's also the not having the pressure.
• 38:06
  So I think if you want to try things, just do side projects, lots of side projects and just do stuff.
• 38:13
  Yeah.
• 38:14
  I mean, the value of all this generative stuff is you can keep generating.
• 38:17
  So, you know, you can start a side project and let it go and come back to it at any time or just restart it. Yeah, I find myself often just completely deleting whatever it created and starting over because it's cool.
• 38:32
  Yeah, 100% it just saves time, I think, going down the wrong path.
• 38:36
  Yeah, absolutely.
• 38:36
  Yeah, I've done that too.
• 38:37
  And those moments can be frustrating.
• 38:40
  Yeah, and I think in those moments also, lean into your strengths as like a software engineer and your intuition and you can make sure that doesn't happen by just doing the stuff you would do as a normal engineer for example do research have a erd ready or like some plan or like do some kind of research yourself or at least think through things and if you don't understand everything make sure you understand everything and don't just let it do things because that's what it does like it's really good at just doing things but make sure whatever the lm or cloth or whatever you use is producing aligns with your vision. That's why I like to start with the readme because a readme is a pretty clear vision, at least the DSL or the design philosophy is a vision.
• 39:34
  Yeah.
• 39:35
  Yeah, I love that readme aspect.
• 39:38
  Before AI, I would often like create a Ruby interface first before it existed, just as like a way to visualize yeah how the things will work together and then go back and like test drive those aspects of it.
• 39:56
  I love Gary Bernhardt's like the destroy all software I definitely drew a lot of inspiration there in that way. I'm curious like now, now that you've created this business, Ruby, imagining Rails 2, where do you find the value of working within Rails or Ruby that aligns well with building these AI systems?
• 40:16
  Where are the true, oh yeah, I wouldn't do this in another framework or language, maybe because it does this, right?
• 40:24
  Like it works well in this way.
• 40:26
  I love writing Ruby and using Rails.
• 40:30
  So I have another engineer working with me now, but I did it alone for months.
• 40:36
  So like I want to work in something that I like to work in.
• 40:39
  Like that's my personal take.
• 40:42
  But also it's very well aligned with the single founder paradigm like like is a single person framework and that is only enhanced by adding ai because that means as a single person you can just do way more work which is even more cool. So it just augments the values of Rails already in a philosophical sense.
• 41:08
  Like you can do more, even more using AI with this single person framework.
• 41:13
  So in that way, it aligns very well.
• 41:16
  But also if you write good code and good abstractions and you test things, like it's very good and the beauty is it's clear what the right way is because we have conventions already and you can just say to an lm just follow these conventions yes you can use standard rb and stuff like that as well but it's not wild it's not going wild and it kind of knows what the conventions are and if it doesn't you just say go to the rails documentation and research best practices on this thing and it just searches the web and like they're not like very opposing views on most things like it's pretty clear there are a few options for most things, and it works very well for that.
• 42:06
  I did start with Jumpstart Pro as a starter template, which I love.
• 42:13
  It's from Chris Oliver, and it just means it already has accounts built in and all of that stuff.
• 42:20
  And I don't need to deal with all of that.
• 42:22
  It saves so much time.
• 42:23
  It saves so much time.
• 42:24
  And starting actually with a starter project is the way to go.
• 42:28
  I think people say, actually, you don't need starter projects because AI can do it so quickly.
• 42:34
  I disagree because it will crash very quickly or I'm sure it can do it, but it's very easy to make a wrong turn somewhere.
• 42:42
  And if you have a starter project it's already grounded in good taste and decisions to a certain degree and it's yours to add your value or your business on top of it which i think is also very good yeah i'm curious to like contact seven as an example like all these mcp stuff they are just so useful and almost like ragging the ecosystem in a way on a granular level even of, you know, hey, check out these Rails guides for this aspect, like look up the documentation.
• 43:14
  To be honest, Rails is so well documented.
• 43:18
  I think historically there has been some blowback in the community on like, oh, we need better documentation and things like this, but like compared to a lot of other things.
• 43:27
  Yeah, I think like the starter, like how to learn Rails from the first time, I think that was always a little bit lacking, but that's good now as well.
• 43:36
  So yes, CodeDoc 7 is great for pulling in documentation and grounding.
• 43:41
  What I've used most is actually just saying to Cloud in my Cloud MD, use bundle open and just open the gem and see how it works. Because there's yards, like it works so great because, for example, StepperMotor has documentation, but also it is updated every day almost, or every week.
• 44:02
  So, why not just open the gem?
• 44:05
  So it's very, very good at just opening the source code and just understanding how it works.
• 44:11
  It's my favorite way to add context now.
• 44:14
  And it just does it automatically now if it needs to know how certain things work.
• 44:19
  I think I actually did see your ex-post about that and adjusted my own workflow.
• 44:25
  It does.
• 44:26
  It really does improve it.
• 44:28
  Yeah.
• 44:28
  Because you don't even need documentation at that point.
• 44:31
  It just knows how it works because it just.
• 44:33
  Self-documenting workflow.
• 44:34
  Yeah.
• 44:35
  It's basically self-documenting.
• 44:37
  And if you write good code and if you use good names and good structure, like it will only improve.
• 44:44
  Yeah.
• 44:44
  So if you use weird gems that do weird stuff probably that's not working but if you have a nicely structured and maybe yards in there as well have you started like documenting in the code like signals for LLMs to pick up on and read as an example the yard like notation can help help in that way.
• 45:07
  Have you found other cheats?
• 45:10
  Hey, look at this.
• 45:11
  So for service classes, for example, I just use plain old Ruby objects, but I always add class-level Yard documentation with examples.
• 45:22
  And that works great.
• 45:23
  I wouldn't have done that before, but now you just say whenever you're building it, you just say add yards at class level with examples and it's done.
• 45:33
  Like it's very easy. I know the examples are definitely key. Yeah, you need to have the examples because then you can like communicate business logic or like some contextual thing that's a little bit higher this is how you would use this class from different places and call it in different ways so i think that works very well and it's also really great for reviews so like if someone or an lm does that i can review that and basically you're reviewing the dsl and how it will be used, which is nice.
• 46:05
  Like that's nice to review and you can always go into detail, but it gives a nice overview of what it does.
• 46:10
  There's so much more I would want to dig in with you here, more specifically around evals too, which I love that the LevaGem kind of just has like this Ruby execute way of encapsulating what metric you want to create even really interesting and and awesome so i'm curious like for people starting out like they have these prompts and things that they're doing and they want to measure something about it where do you recommend them digging in and where should people be focusing their attention on you need to implement your own evaluator so you need to know a way how to say if something is good or not i'm not doing any of that i just provide you a framework and the hardest part is honestly is like finding a number or metrics that you want to optimize for and categorization is easy because it's either the right category or not.
• 47:06
  But if it's like writing a response to an email, like how do you measure if that's good or not, right? And how I do that is we track what people actually send versus the drafted version.
• 47:22
  And then we have a Jakar similarity score, which is like how many words match or like how similar is the message they actually sent for the message that we generated, that's a number, so it's a number from zero to one, which is great because a number from zero to one is very good for evaluations.
• 47:40
  So think of ways like, hey, how can I capture value or an output in a number of zero to one and one being the best zero the worst so that's really how you need to do it but you can do it other ways you can use an lm as a judge this is another way where you just have an lm with a prompt that says hey you're a judge to see if this is well written, blah, blah, blah, blah, blah.
• 48:07
  Okay, just make your own prompt.
• 48:09
  And this is what is written.
• 48:10
  And you have to rate this on a score from 1 to 10.
• 48:13
  And then you just normalize it to 0 to 1 and give me the reasons why or like have 10 things that you check for.
• 48:22
  And then that's an LM as a judge.
• 48:24
  And people do that.
• 48:26
  I never did that because I feel that's another layer of, it's another thing that you need to check again because does that prompt work?
• 48:36
  You don't know. So I would always steer towards like just hard coded more logic evaluation.
• 48:42
  Getting a discrete metric out of it is always better.
• 48:45
  It's always better.
• 48:46
  Yeah, but sometimes you cannot do that and you can use an LLM as a judge and you can set it up that it works pretty well, but it's just more work and more things can go wrong.
• 48:57
  Is there somewhere that you look for maybe inspiration on how to get these metrics?
• 49:03
  As an example, like summarization, like is there like a place that you look, oh, like is there an algorithm that already generates this kind of metric to see how well that it summarized the aspects of this lengthy document?
• 49:15
  Yeah.
• 49:16
  Where's your thought process lead?
• 49:17
  Yeah, I just jam with Claude or Chet GPT to see if there are any things out there and just do deep research on what is the science behind these things and Jakar score was one of those things that I've never heard of it but I was like well okay that sounds like what we need so let's do that so that's how I arrived there but at the same time there's also an art to it and and that's fine like it is fine to spot check things and vibe check things and just feel things too it's not all numbers like it is okay to read something and say ah this doesn't feel right and just go in and change little things and see if it feels better like it's also that it's both and i think that's important it's not only numbers because if the number says a certain thing, it doesn't mean it's good if your number is not good.
• 50:09
  So just realize it's just one piece of the puzzle, but it's a very important piece and you need all pieces.
• 50:17
  You need the art of understanding what's good and not as well.
• 50:21
  Yeah.
• 50:22
  It fits together.
• 50:23
  Do you find yourself like reviewing maybe the evaluations?
• 50:27
  Do you have like an annotation process?
• 50:29
  Is there like any aspect of like the review and that vibe spot checking where like you like feed it back into itself or where do you fall in that category?
• 50:40
  So Leva doesn't have annotation built in, but I have it in my own app.
• 50:45
  I just add records to certain data sets, which is kind of the annotation.
• 50:49
  So yes, I do annotate things and either it's automatic where there is a signal from a user that returns something to their inbox, for example.
• 50:58
  That's a signal that the categorization was not right so it will be added to a data set and then you can run an experiment on that new data set where maybe i say okay maybe i change something here will that increase everything here and then i add some emails that were right so there's a good mix of both does it still stay good there as well does it overall improve so there's that but there's also like i run it i see the one that failed and i see did it feel for like a very bad reason or like a reason that's like reasonable and if it's a very bad reason like like I go in and fix it. But if it's like, okay, for example, if it says it's categorized as it needs to stay in your inbox, but we don't know how to draft a response because we don't have context.
• 51:54
  It's not too bad because yeah, maybe I argue there is enough context, but it's not a bad thing.
• 52:00
  Like it's the wrong category.
• 52:01
  Like it's as wrong as something that was very important and got unsubscribed from but that's a very bad one so the spot checking isn't like yeah the number it's not the right category but like how bad is that miscategorization and if there is one i just jump in and levi you can experiment on that specific one and you go to the prompts that it used and you can iterate on the prompt and make some new versions so yeah that's how i do it do you find like that triaging aspect like can be solved by an llm eventually or not something that you would even foresee like worthwhile as it and i'm gonna yes i'm already like half doing this by an LLM so yeah absolutely it's just that I need to like Leva can make it a little bit easier where you can maybe have a button that says hey look at all the failures analyze what went wrong like those should already be in or run this in Gemini 2.5 but also run this in 4.1 or something like that.
• 53:08
  And explain to me what the differences are or like, what are the most different things?
• 53:14
  Like, I would love these things to be in Leva as well, because that's what you want to do.
• 53:19
  That's what you need.
• 53:20
  And then after that, like from those learningss can you actually rewrite the prompt obviously and then rerun it and there can be a loop there can be a gentic part to it i think that will be in leva very soon or sooner than later because i do that analyzing swaths of disparate information is like what is great. Exactly.
• 53:45
  Yeah.
• 53:45
  I'm always looking for ways that integrate that.
• 53:49
  It seems like somebody out there is going to make a Rails gem that just can watch the logs or active support and just like create analysis over, you know, yeah, exactly.
• 54:01
  Fix this or alerting just an observer for the entire Rails stack.
• 54:06
  It's coming.
• 54:07
  I have it.
• 54:08
  Oh, yeah.
• 54:08
  It's already there in ways.
• 54:11
  So I have like I use AppSignal and I just link to their API with a custom MCP in Cloud.
• 54:18
  For me, I already do that.
• 54:19
  And I can also live till the logs and analyze things in Cloud Code, which is also nice.
• 54:27
  So I personally already have that, but I'm still figuring things out.
• 54:31
  But yeah, I do that already.
• 54:32
  But there should be a tool that makes it this easy.
• 54:35
  Like AppSignal, for example.
• 54:37
  And see the UI, you know?
• 54:38
  Yeah.
• 54:39
  The visual.
• 54:40
  It's all about the visual.
• 54:41
  And Rails makes it easy to make those.
• 54:43
  So I'm looking forward to just seeing, honestly, what people are creating.
• 54:46
  Yeah, let us know.
• 54:47
  We want to use it.
• 54:48
  We want to pay.
• 54:49
  We do.
• 54:49
  We totally want to use it.
• 54:51
  It's not hard to create new things.
• 54:53
  Yeah.
• 54:53
  Please build.
• 54:54
  We've gone through a lot here, Kieran.
• 54:56
  And I appreciate you coming on and telling everybody about your insights, building the one-person business.
• 55:02
  I imagine you're maybe a little bigger than one person at this point.
• 55:06
  Two.
• 55:06
  But maybe not.
• 55:07
  Two people.
• 55:07
  Two, okay.
• 55:09
  Still impressive what Rails can enable people to do and AI too.
• 55:13
  It's really cool.
• 55:14
  We launched to general public last week and I built a whole company in less than a year, which is really, really cool.
• 55:23
  I had some help on the way for sure, but like I wrote 80% of the code.
• 55:29
  You know, I started using it myself just to see how it works.
• 55:32
  That's honestly works great.
• 55:33
  It does everything I wanted it to do.
• 55:35
  So if you can get it to work in the Hey app too, that would be awesome.
• 55:40
  But you know, I know like Gmail's probably your broader audience.
• 55:44
  For now, we'll add every other tool, but the MailChimp has IMAP support.
• 55:49
  So everything is built around the MailChimp.
• 55:53
  So, oh, that's nice.
• 55:55
  So it's coming soon.
• 55:56
  We have enough people with Gmail for now, but yeah, we will support other providers like Outlook and IMAP as well.
• 56:04
  Well, we're coming to a close here.
• 56:06
  Is there anything else you wanted to mention before we move in to kind of like our pick segment?
• 56:10
  No, thanks for doing an AI and Ruby podcast, because I feel we need more energy and more enthusiasm in the Ruby and AI space.
• 56:19
  I know there are a bunch of people that are really cool, but I think there can be always more.
• 56:26
  Yeah, 100%.
• 56:27
  I've been trying to get this started for over a year now.
• 56:30
  I met the right people.
• 56:31
  Yeah, I'm just a talking head.
• 56:34
  So I love meeting people that are working on stuff that I'm not even thinking about and just exposing it to the rest of the Ruby world because that's what, honestly honestly Ruby is great at.
• 56:45
  Yeah.
• 56:45
  I miss kind of like the Ruby mailing list aspects of spreading stuff.
• 56:50
  And it's just moving too fast kind of for the mail media.
• 56:54
  Well, at the end of the show, we like to just let you share anything that you want, something that you found useful or that you want others to know about.
• 57:02
  Anything at all.
• 57:03
  I'm going to share something that is not cloud code but it is an agent before cloud code came out with their unlimited plan i didn't use it as much because i was like i'm not going to spend five dollars on a git commit it's just too much because it was pretty expensive before. And I got connected with someone through every where I work as well.
• 57:27
  And they said, ah, try this out.
• 57:28
  This is a genetic builder and it's called Friday.
• 57:33
  And it's kind of fun because Friday is also similar to cloud code to see light tool, but it's very opinionated.
• 57:41
  It's like, not like cloud where it is like, you can do whatever you want. It's very opinionated. It's not like Cloud where it is like you can do whatever you want.
• 57:45
  It's very modular.
• 57:48
  They just say, no, it's not modular.
• 57:50
  It's very opinionated.
• 57:52
  This is how we do things.
• 57:53
  But I kind of like it and I love it.
• 57:56
  So I use it as well.
• 57:58
  And it's really good at just giving an issue and say, just implement this and go like YOLO mode.
• 58:04
  And I have moments that I use it for implementing things with Figma.
• 58:08
  They have like a Figma integration.
• 58:10
  They don't expose every MCP, but they said, no, the Figma one will add.
• 58:15
  And they make decisions.
• 58:17
  So if you want to try out a different kind of agent, like give us a week free trial as well.
• 58:23
  Like it's interesting to see how others think about agentic work and yeah there are many but i think this one is kind of cool because it's so opinionated and it might resonate with real developers where we are also very opinionated about how to do things so that's my random pick. It's also a very small team.
• 58:45
  There are like a few people working on it, which I think it's always cool.
• 58:49
  Small team doing things that compare with multi-billion dollar companies.
• 58:55
  So that's awesome.
• 58:56
  Yeah, I'm going to have to check that out.
• 58:57
  It looks fun.
• 59:00
  Codewithfriday.com is the URL.
• 59:03
  Awesome.
• 59:03
  Thanks for sharing that.
• 59:04
  It looks fun. I'm going for sharing that. It looks fun.
• 59:05
  I'm going to share this.
• 59:07
  It was shared in the Ruby AI Builders Discord.
• 59:09
  I don't know if it's an LLM on its own or not.
• 59:12
  Yeah, I think it is a model.
• 59:14
  It's called Morph LLM, but it basically is a way to analyze like diffs of files and apply them specifically for usage in LLMs.
• 59:25
  And it's like super fast.
• 59:27
  I think, yeah, 4,500 tokens per second.
• 59:30
  And so I would highly recommend checking it out.
• 59:33
  I've been using it in a couple of things just as part of my workflow, and it's been working great.
• 59:37
  So it's morphllm.com.
• 59:40
  Well, thanks for coming on, Kieran.
• 59:42
  It was great hearing all the lovely contributions you're making to the community with Leva and Quark's computer. Looking forward to seeing all the great stuff that comes out of that.
• 59:51
  Thank you so much. Yeah, happy to share, inspire, and everyone go build stuff and try things out.
• 59:59
  Have fun!