Latest / Logixly Live Unsupervised Learning Podcast / JEV And The Rise Of Decision Models

Transcript

JEV And The Rise Of Decision Models · Logixly Live Unsupervised Learning Podcast

• 0:06
  SPEAKER_01: I guess he was here.
• 0:08
  SPEAKER_01: Anyway, that was never mine.
• 0:13
  SPEAKER_02: Yeah, small town.
• 0:14
  SPEAKER_01: Yeah, small town is right.
• 0:20
  SPEAKER_02: I'm excited for the topic today, I feel like.
• 0:24
  SPEAKER_02: I don't know.
• 0:25
  SPEAKER_02: I love topics like this where it's uh you just feel these changes.
• 0:41
  SPEAKER_00: And so it's just the the idea that it's not us introducing a new topic, it's sort of people people are aware of the topic, you know.
• 0:47
  SPEAKER_02: Which is so interesting thinking back on like like when when there when a lot of what we did was education about review.
• 0:56
  SPEAKER_02: Oh man.
• 0:57
  SPEAKER_02: And now and now it's just it's the 180.
• 1:00
  SPEAKER_02: It's so great.
• 1:02
  SPEAKER_02: Much better.
• 1:02
  SPEAKER_02: It's much better.
• 1:03
  SPEAKER_02: Yes, it's much better.
• 1:04
  SPEAKER_02: Um but anyway, I'm gonna welcome everybody to Logics Lee Live, uh, where we hit all of the the topics in AI.
• 1:11
  SPEAKER_02: And I won't say we have a special edition today, but we do have a special guest today.
• 1:16
  SPEAKER_02: We got Ted back and and myself back, and uh and Ben Webster is joining us today.
• 1:21
  SPEAKER_02: Ben Webster has been with Logics since literally the beginning, very first employee.
• 1:28
  SPEAKER_02: And um, and I would say hasn't changed a bit, but obviously, but obviously I don't mean that, I mean that in in a in a good way.
• 1:36
  SPEAKER_02: Ben is Ben is one of my favorite people, he knows that.
• 1:39
  SPEAKER_02: Um but anyway, welcome to the show, Ben.
• 1:41
  SPEAKER_02: Yeah, so happy to be here.
• 1:42
  SPEAKER_02: Thanks, guys.
• 1:43
  SPEAKER_02: Yeah.
• 1:43
  SPEAKER_02: So I'm gonna jump right in because I'm uh I know you know we we want to hit the hit the topics and get to it, but today's topic is Jev.
• 1:52
  SPEAKER_02: If you have not heard about Jev, we're gonna talk about it a lot today.
• 1:55
  SPEAKER_02: But uh, I just want to give you uh kind of a high level and explain what JEV is kind of relative to the other language models we're all used to dealing with.
• 2:05
  SPEAKER_02: Uh so Jev is is a commercial product from a um from a company called TypeSafe AI, the founder of TypeSafe AI.
• 2:15
  SPEAKER_02: I won't get his bio exactly right, but he worked for ChatGPT, worked for a big lab.
• 2:20
  SPEAKER_02: He's on a lot of their published research.
• 2:23
  SPEAKER_02: And of course, like a lot of the the group did from OpenAI, a lot of them have moved on and founded other businesses.
• 2:28
  SPEAKER_02: So TypeSafe AI is this is this business.
• 2:31
  SPEAKER_02: Their core product is JEV, and it is a model that is focused on making decisions.
• 2:38
  SPEAKER_02: Okay, so it's it's not a chatbot like uh like we're all used to.
• 2:42
  SPEAKER_02: It's kind of a narrower interface, you know, it's kind of text or state as input, and then it is like a discrete set of things on the output.
• 2:51
  SPEAKER_02: So it's it's you know giving you um you know a set of options or maybe a probability of a yes or no.
• 2:58
  SPEAKER_02: So it's a very constrained output.
• 3:00
  SPEAKER_02: They're calling it a system one model, which uh they describe as a way to make it's designed to make fast focused judgments inside of software, which I think is a key piece.
• 3:12
  SPEAKER_02: Like reading that sentence, I was like, oh yeah.
• 3:15
  SPEAKER_02: Okay, so today's language models are kind of built for both.
• 3:20
  SPEAKER_02: We're using them as knowledge work, general knowledge work.
• 3:23
  SPEAKER_02: And then we also kind of wrestle with them to get them to interact properly with software.
• 3:29
  SPEAKER_00: Yeah.
• 3:30
  SPEAKER_02: Uh and I remember, I'm looking at you now, Ben.
• 3:32
  SPEAKER_02: I remember the first time uh we, you know, playing around with Chat GPT, and it was like, oh, wait a second, can I get you to make a prediction across a class of labels?
• 3:43
  SPEAKER_02: And can I ask you for a probability?
• 3:46
  SPEAKER_02: Uh and and just feeling that was like, man, that is weird and different.
• 3:50
  SPEAKER_02: Feels like I'm misusing a system, but it also feels like there's some value in this.
• 3:54
  SPEAKER_00: Yeah, I mean, I would say, you know, you and I both got exposed to all of this stuff at the same time when it first started coming out.
• 4:02
  SPEAKER_00: And you actually used a line when talking to me about it that stuck with me.
• 4:06
  SPEAKER_00: You said, all of these things feel very brittle when this we were in the early days of this.
• 4:11
  SPEAKER_00: And the reason why is because a lot of the times people were trying to use this as embedded in their software, and we really wanted very uh deterministic results that we wanted to know and trust.
• 4:24
  SPEAKER_00: And um, everybody knew that the LLM belonged there, but it just it it it wasn't uh solving those problems in the beginning, you know.
• 4:32
  SPEAKER_00: This this is our attempt at getting uh a little more stable there.
• 4:36
  SPEAKER_02: Yeah, and I I think what's I mean to me, what what I think is interesting is is this Jev to me puts a concept, a more concrete concept, to like something I think um we've kind of been feeling, which is um the you know, and maybe this is in hindsight, but it's kind of like there's really two use cases.
• 4:57
  SPEAKER_02: There's the the chat use case that sits on everyone's desktop, you know, work with the human to be an assistant and help them do things if it's write code or if it's write an email or write a proposal.
• 5:10
  SPEAKER_02: And that really is different than I want to automate a business process.
• 5:16
  SPEAKER_02: The process is deterministic, but I have these points where I need decisions.
• 5:21
  SPEAKER_02: And to me, I feel like this Jev is saying, hey, we're gonna focus on narrowly focus on helping software make decisions.
• 5:31
  SPEAKER_02: And it it it it feels like, hey, is this the beginning of like an unbundling of all of the capabilities in those big frontier models?
• 5:40
  SPEAKER_02: Is it time and it's worth doing to pull out some of the functionality and unbundle them and and kind of focus them to improve speed, cost, and quality?
• 5:49
  SPEAKER_01: Yeah, and then you were talking earlier today and talking about you know CPU versus GPU, and we all live through that, right?
• 5:58
  SPEAKER_01: Um, I guess 10, 12 years ago now.
• 6:01
  SPEAKER_01: Right.
• 6:01
  SPEAKER_01: Um, where you know the CPUs were more general purpose, and then the GPUs we started to do more and more focused modeling to solve business problems.
• 6:11
  SPEAKER_01: So it kind of feels like back to the future, yeah.
• 6:14
  SPEAKER_01: Or back to the past, or one of those.
• 6:16
  SPEAKER_02: I mean a hundred percent.
• 6:16
  SPEAKER_02: And to me, like that's almost the story of software where it's like, you know, CPUs clearly came first, super valuable, did a lot of workloads, did them well.
• 6:26
  SPEAKER_02: And then what happens?
• 6:27
  SPEAKER_02: Well, you got these specialized use cases with GPUs, certainly gaming, but clearly deep learning right behind it, where it's like, no, here it's worth specializing.
• 6:36
  SPEAKER_02: I uh to me, databases is another one.
• 6:39
  SPEAKER_02: Like, hey, you know, it used to be a file system, it used to be this, and then it's like, no, hold on.
• 6:43
  SPEAKER_02: There's a there's a specialization here to create a piece of software that does this really well.
• 6:49
  SPEAKER_02: I I feel like I agree with that.
• 6:50
  SPEAKER_02: It's like this is a new kind of specialization, Ben.
• 6:53
  SPEAKER_02: And we were talking about how well this aligns with kind of the spectrum that we see when it comes to automation.
• 7:02
  SPEAKER_02: I don't know if you're gonna give some color to that.
• 7:04
  SPEAKER_00: Yeah, sure.
• 7:04
  SPEAKER_00: So, so um, you know, one of the things that happens quite a bit um with LLMs today is they're they're optimized for engagement.
• 7:12
  SPEAKER_00: Somebody asks a question and the LLM, you know, whether it answers it distinctly or not, it's gonna ask follow-up questions.
• 7:18
  SPEAKER_00: It's gonna try and engage in a conversation.
• 7:21
  SPEAKER_00: And and the thing is that a lot of the use cases that people talk about um are really just I'm gonna ask a question, I need to answer, and then I need to move on.
• 7:30
  SPEAKER_00: And so I remember when they first started talking about tiers on the GPT models, and they always told you to go for the light tier when you had the simple questions, when you just wanted to ask sentiment or a yes or no or something like that, you would always use the light models.
• 7:45
  SPEAKER_00: And I remember that really felt weird to me because the light models weren't not as smart, which means they're not gonna be as accurate.
• 7:52
  SPEAKER_00: And these classification tasks, when you think about it on an enterprise pipeline, being right when you classify is super important.
• 8:00
  SPEAKER_00: And so, why are you going to farm that out to a lighter model that might be wrong 20% of the time or something like that?
• 8:06
  SPEAKER_00: You know, and so like this is taking on that simpler class of tasks where it's just classify or say yes or no, but you're not sacrificing and saying, Oh, I'm gonna use a simpler model as a result.
• 8:20
  SPEAKER_00: Like, one thing that that I guess needs to be uh um clarified here is like Jeb is not saying we're gonna use a dumbed-down model, like it is it is a a high quality smart model used to answer a very simple output.
• 8:34
  SPEAKER_00: And I guess that's the combination dynamic.
• 8:37
  SPEAKER_02: Right.
• 8:38
  SPEAKER_02: And you know, just to think about because it does feel it it feels clear to me that uh we're using frontier models in too many places across the enterprise.
• 8:50
  SPEAKER_02: We're using it everywhere, right?
• 8:52
  SPEAKER_02: And it's not needed everywhere, it's it's expensive in a lot of places.
• 8:56
  SPEAKER_02: I think we all feel that.
• 8:57
  SPEAKER_02: Um so in this world of unbundling, I I mean, my opinion is it it makes a lot of sense to build language models like JEV for software, not for humans.
• 9:10
  SPEAKER_02: That's right.
• 9:10
  SPEAKER_02: And in this unbundling world, like I I I definitely feel like if I were to make a prediction, if I was gonna put something upon Kelshi, I would say within weeks or months, I would say OpenAI and Anthropic are probably gonna have a JEV competitor.
• 9:28
  SPEAKER_02: And it might be multiple JEV competitors.
• 9:31
  SPEAKER_02: Like I JEV is focused at across industries, but for decisions.
• 9:35
  SPEAKER_02: Like it I feel like you had a dis you had an industry matrix, you had other types of tasks or other types of software integrations.
• 9:45
  SPEAKER_02: And I could see to me, I could see this being a whole area of um of of like a model catalog that are probably targeted for agents and software.
• 9:57
  SPEAKER_00: Yeah, anybody who doesn't come out with their version of it, they're gonna they're gonna miss out.
• 10:02
  SPEAKER_00: Because I I bet you if you profile any of our complex solutions that are out there, most of the the um steps along the way are steps that are just individual decisions.
• 10:13
  SPEAKER_00: And so, like uh anybody who has a really big cloud bill who's looking at optimization tasks, who doesn't have a JE-like option within their native cloud provider, they're going to be looking elsewhere as as we start to see just the impact on how much this can save people and their cloud bills.
• 10:30
  SPEAKER_02: Yeah.
• 10:31
  SPEAKER_01: Yeah.
• 10:32
  SPEAKER_01: I like how it's getting, we're kind of drifting from it's less magical and becoming more operational.
• 10:40
  SPEAKER_01: Like it's like I didn't make that up.
• 10:42
  SPEAKER_01: That was when I was doing some research.
• 10:43
  SPEAKER_01: I saw that from someone else.
• 10:45
  SPEAKER_01: But it really did, it it does feel like that because everyone's been talking about AI and magic, and we're all gonna die in the next few months and everything.
• 10:53
  SPEAKER_01: And it's like, no, it's shifting to to be more um specialized.
• 10:58
  SPEAKER_01: I I feel like things that that we've done for 15 years is stringing together um different models to perform tasks and very narrow, but but pretty darn accurate.
• 11:08
  SPEAKER_02: Well, and I would say like we we've cost, quality, speed are always part of the equation at scale.
• 11:15
  SPEAKER_02: Like you you can't and I and I and I feel like the market is saying frontier models are expensive and they're and they're probably slower than you'd like.
• 11:25
  SPEAKER_02: And now to us, I feel like it feels so natural because that's the world we live in.
• 11:28
  SPEAKER_02: But I'll Ted, your point on the magical piece.
• 11:32
  SPEAKER_02: Um, that's what I find so interesting is AGI to me is like magic, right?
• 11:37
  SPEAKER_02: You it's not well defined, it seems like this thing, and I love the dichotomy of OpenAI saying, you know, we're in the window of AGI right now, it's here.
• 11:47
  SPEAKER_02: And then at the same time, we have a, you know, a topic like JEV, which is clearly counter to that, saying, hey, wait a second.
• 11:56
  SPEAKER_02: Actually, there is so much more nuance and and detail when it comes to really building solutions or software impacting business.
• 12:05
  SPEAKER_02: It's not just this Oracle model that you just plug in and solve every problem, right?
• 12:10
  SPEAKER_01: Yeah, no, absolutely.
• 12:11
  SPEAKER_01: I love I love how it's more focused on break what we do every day, and every business person's out there, you break down tasks, you build process around it, you know, you have quality, you have other things, all of those functions that go into it is really focusing, you know, it's focusing on that.
• 12:27
  SPEAKER_01: And that's what that's what I'm excited about.
• 12:29
  SPEAKER_00: And yeah, there's there's also this ability to leverage uh the confidence from out of the model in order to make decisions, which uh, you know, some of the uh classic LLM approaches, it doesn't give you that level of um clarity in order for you to conditionally route and stuff like that.
• 12:45
  SPEAKER_00: So what it is presenting to you feels closer to a an actual fine-tuned model that was built on your own data.
• 12:52
  SPEAKER_00: And I guess that that would be something that I would be curious about is that, you know, um at Logics, we spend a lot of time with very high volume use cases where we are able to collect a lot of good training data.
• 13:05
  SPEAKER_00: And eventually we always want to move to that custom model because custom models are gonna be more accurate at scale, they're always gonna be cheaper.
• 13:13
  SPEAKER_00: And so, this ability for JEV to give you some of that stuff that you get from the custom model with the classification with the confidence score and everything like that, it's it's getting us one step closer to that thing that we always want to build.
• 13:27
  SPEAKER_00: And and the big advantage is a lot of times clients don't want to wait for collecting labeled data and monitoring and auditing all of that stuff.
• 13:35
  SPEAKER_00: So, one step in the right direction as people are trying to weigh the options between having your own model and then and then just using a generic one.
• 13:44
  SPEAKER_02: And I guess I don't know, just to make sure that point comes across, it's like uh to me, this is really interesting because we're talking about probabilities and confidences out of a model.
• 13:56
  SPEAKER_02: Yeah.
• 13:56
  SPEAKER_02: And you got to help me if if I'm not um, you know, right on this.
• 14:01
  SPEAKER_02: But I mean, to me, for the most part, when we talk about confidence and probability in a machine learning standpoint, it's usually with the understanding that this has been a model that's been trained on your data.
• 14:15
  SPEAKER_02: I mean, for the most part, right?
• 14:16
  SPEAKER_00: Yeah, it's trained on your data, it understands the expected distribution from what you see in your population.
• 14:23
  SPEAKER_00: And so we go through a lot of effort to calibrate it to make sure that when it says 72% probability, that about seven out of ten times it's going to be right.
• 14:34
  SPEAKER_00: Right.
• 14:34
  SPEAKER_00: And and then people can really make good decisions off of that.
• 14:37
  SPEAKER_00: And today's models, when you just ask them how confident are you, and it says 42%, like it's just a number that it's spit out, right?
• 14:44
  SPEAKER_00: It's very far away from what people want to think when they hear the term confidence.
• 14:49
  SPEAKER_00: Um, but Jev, being that it is trained specifically for these style tasks, yeah, it's still not gonna be calibrated to your use case, right?
• 14:58
  SPEAKER_00: But it is calibrated to the task.
• 15:00
  SPEAKER_00: Um so so therefore it's closer to the right probability, and it's getting people into a system where they're actually gonna use the probability to make decisions, which is helping them on their way to justifying fine-tuning with their own data and everything like that.
• 15:15
  SPEAKER_01: Right.
• 15:15
  SPEAKER_01: Can I ask a quick question?
• 15:16
  SPEAKER_01: Yeah.
• 15:17
  SPEAKER_01: I'm gonna put you guys on the spot.
• 15:18
  SPEAKER_01: Yeah.
• 15:19
  SPEAKER_01: Um what type of training environment is because we all know about the LLMs.
• 15:25
  SPEAKER_01: You've got a big data centers and massive amounts of data and everything.
• 15:29
  SPEAKER_01: What is what do you estimate or would you guess it goes into training a JEV model?
• 15:35
  SPEAKER_00: So we don't know because that's not really public information.
• 15:38
  SPEAKER_00: But um, you know, there there is a similar model that just uh it's been around for a long time, hasn't gotten quite the same hype, and it's called Gliner.
• 15:46
  SPEAKER_00: And uh, in order for you to fine-tune Gliner for these style of tasks on your own data, it's actually shockingly small, the amount of data that you need in order for those patterns to be learned and and applied.
• 16:00
  SPEAKER_00: Um, so I wouldn't be surprised to find out that this is a relatively small data set just done well.
• 16:06
  SPEAKER_00: Uh, I would actually be quite shocked if it was, you know, billions of records kind of a thing.
• 16:11
  SPEAKER_00: Yeah.
• 16:12
  SPEAKER_02: Which is exciting because then if you think again, you think about the enterprise, if you're taking even like 60 or 70, let's let's say, let's say you have a workload and you have decisions and you can use models like JEV and they are small enough to run on local hardware or not on you know large GPUs, like that that to me is gonna be a game changer.
• 16:34
  SPEAKER_02: And I'll say this, Ted, I gotta tell this example because I loved it.
• 16:38
  SPEAKER_02: But uh so not not on how they train Jeb, but on on um reinforcement learning with human feedback in general, is the the you know, the the the founder of Jev was making this point, like basically, hey, these models, all the models we use in ChatGPT are are as assistance, like they were trained to interact with you, the human, okay?
• 16:58
  SPEAKER_02: And they're trained to they are trained to optimize to human preferences, okay, not necessarily outcomes and tasks and be well calibrated in certain dimensions, but to be aligned with with human preferences.
• 17:13
  SPEAKER_02: And so the example was um, and you have to you have to just play along here, but the example was this guy who's proven a point, he's using Chat GPT, and he had a recording of fart noises.
• 17:25
  SPEAKER_02: Okay.
• 17:26
  SPEAKER_02: That's good.
• 17:26
  SPEAKER_02: I know.
• 17:28
  SPEAKER_02: I'm not the only one that has recordings of, but but anyway, so he takes this, he takes the recording and he uploads it into Chat GPT, and he's like, hey, I've been I've been working on creating some music.
• 17:39
  SPEAKER_02: I've uploaded this audio file.
• 17:41
  SPEAKER_02: Can you take a listen and give me some feedback?
• 17:44
  SPEAKER_02: And of course, ChatGPT was like, oh, this is this is really like dark and mysterious music, and I think you've done a good job.
• 17:51
  SPEAKER_02: And it's like, but you understand that that's like to me, that's the that's a system that's optimized for humans, right?
• 17:59
  SPEAKER_02: It's not like there is no truth.
• 18:01
  SPEAKER_02: There's to me, there's no truth in that, right?
• 18:03
  SPEAKER_02: Like, like that, is that a system you want making decisions as part of critical software processes?
• 18:09
  SPEAKER_02: Yeah.
• 18:10
  SPEAKER_02: Seems weird, right?
• 18:11
  SPEAKER_01: It does seem weird.
• 18:12
  SPEAKER_01: And yeah, it's just it's telling you what you want to hear.
• 18:14
  SPEAKER_01: Yes.
• 18:15
  SPEAKER_01: And and it's it's it's kind of parroting that.
• 18:17
  SPEAKER_01: You were nodding, Ben.
• 18:18
  SPEAKER_01: You were gonna say so.
• 18:19
  SPEAKER_01: You weren't gonna do a fart joke.
• 18:20
  SPEAKER_01: You were gonna actually say something intelligent.
• 18:22
  SPEAKER_00: No, no, that that I was I was gonna say what you said, Ted.
• 18:24
  SPEAKER_00: You you got the intelligent ahead of me.
• 18:26
  SPEAKER_00: Uh, I wanted to hear it.
• 18:27
  SPEAKER_00: So it's it it really is like like um all of these models have this uh way of agreeing with you or telling you that your idea was good and then following it up with a nuanced opinion.
• 18:41
  SPEAKER_00: And one of the things is when you just give it the ability to say class A or class B, and it's not a model made for that, you have to think that the bias of its wanting to agree with you and make you feel good is going into the way it's choosing to answer.
• 18:55
  SPEAKER_00: And and those biases at scale on enterprise data, like that's a problem, right?
• 19:01
  SPEAKER_00: And so to have a model that isn't trying to be your friend and isn't measuring its success by your engagement level with it, but it's just a question of is this right or wrong?
• 19:13
  SPEAKER_00: Honestly, that feels closer to the world we lived in prior to Chat GPT.
• 19:17
  unknown: Yeah.
• 19:18
  SPEAKER_02: And built for humans versus built for software and automation, right?
• 19:23
  SPEAKER_00: Yeah, because prior to Chat GPT, we were never worried about personal productivity tools.
• 19:27
  SPEAKER_00: Right.
• 19:28
  SPEAKER_00: Right?
• 19:28
  SPEAKER_00: We were worried about enterprise business problems, which is a series of decisions made at scale.
• 19:33
  SPEAKER_02: Right.
• 19:33
  SPEAKER_02: And we weren't worried about human preferences necessarily.
• 19:37
  SPEAKER_02: We were worried about outcomes and correctness of next action.
• 19:41
  SPEAKER_00: Honestly, when we aggregate it at scale, we were normally trying to get rid of the human preferences that happened in individual people.
• 19:49
  SPEAKER_00: Yeah.
• 19:50
  SPEAKER_00: So yeah, who's your target?
• 19:51
  SPEAKER_02: Yeah, man.
• 19:52
  SPEAKER_02: All great points.
• 19:52
  SPEAKER_02: I love it.
• 19:53
  SPEAKER_02: So uh we're wrapping up this episode of Logics Lee Live.
• 19:57
  SPEAKER_02: Obviously, thanks, Ben, for joining us.
• 20:00
  SPEAKER_02: Certain that Ben's gonna be be back joining us again here shortly.
• 20:03
  SPEAKER_02: Um, but I know there's a lot going on in AI.
• 20:05
  SPEAKER_02: If there's other topics uh anybody wants to hear about either for next uh episode or or questions about this episode or other things we didn't cover, just reach out to any of us really.
• 20:16
  SPEAKER_02: Uh we're all on LinkedIn, we're all super available.
• 20:19
  SPEAKER_02: Um so anyway, thanks everybody for joining us.
• 20:21
  SPEAKER_02: And until next time, keep learning even when it's unsupervised.