Latest / Logixly Live Unsupervised Learning Podcast / The State of AI in 2026: What's Working and What's Not

Transcript

The State of AI in 2026: What's Working and What's Not · Logixly Live Unsupervised Learning Podcast

• 0:01
  SPEAKER_00: Well, with with that kind of segue, let's let's jump into uh today's topic.
• 0:06
  SPEAKER_00: So, welcome everybody to Logixly Lee Live, uh, where we try to hit the biggest stories in AI each each week.
• 0:12
  SPEAKER_00: And I I say I think we've done this 10 episodes, 11 episodes now, and I'm repeatedly amazed at how there's always more happening than it seems like you can even talk about.
• 0:23
  SPEAKER_01: Right.
• 0:23
  SPEAKER_01: I constantly think that man, we could do this for hours a day.
• 0:27
  SPEAKER_01: I know it's awesome.
• 0:28
  SPEAKER_00: Not sure we'd have any listeners, but I know you know that's fun.
• 0:30
  SPEAKER_00: Uh and today, honestly, so so this is um it's one of my favorite reports.
• 0:35
  SPEAKER_00: I know there's a lot of groups that do sort of summary state of reports, but I uh I really like this one.
• 0:42
  SPEAKER_00: This is the state of AI report, is what it's called, very novel.
• 0:46
  SPEAKER_00: And we'll put this in the show notes so uh it'll be linked to the content here, but it is Air Street Capital, and it is Nathan, and I'm already gonna say, I forgot how to say his last name.
• 0:57
  SPEAKER_01: I think we tried Benait.
• 0:59
  SPEAKER_00: Benait.
• 1:00
  SPEAKER_00: Uh, and this is October 8th.
• 1:02
  SPEAKER_00: So, what they do every year, they've been doing it.
• 1:04
  SPEAKER_00: I want to make sure I get this right too.
• 1:06
  SPEAKER_00: They've been doing it since 2018, kind of the state of AI, and it really is a deep dive.
• 1:11
  SPEAKER_00: It's really a meaty report.
• 1:13
  SPEAKER_00: It's it's well over a hundred pages and it's full of novel kind of insights.
• 1:18
  SPEAKER_00: And so Katie and I went through here and just kind of pulled out some of the ones that um that caught her eye on kind of the first the first time through.
• 1:28
  SPEAKER_01: For sure.
• 1:28
  SPEAKER_01: So I I think the coolest thing I gotta say, like I learned stuff from this.
• 1:32
  SPEAKER_01: You know, you see a lot of state of AI reports that are like, oh, here's just uh, you know, the facts that everybody's seen.
• 1:38
  SPEAKER_01: If you've ever read the news or you follow AI at all, but like this, it's genuinely stuff that I was like, wow, I I had no idea.
• 1:45
  SPEAKER_01: Like it's fantastic, really well done.
• 1:48
  SPEAKER_00: Okay, the reason why I like it is because, you know, and I I actually like all the reports.
• 1:51
  SPEAKER_00: So I uh, you know, Deloitte, all the all the all the groups put out reports.
• 1:56
  SPEAKER_00: In my mind, I feel like they all have, you know, they're all probably promoting something else.
• 2:03
  SPEAKER_00: Right.
• 2:04
  SPEAKER_00: I like this because I I do feel like this is more independent.
• 2:07
  SPEAKER_00: Like I really just think these are data points, and you can read the research, read the facts, and see if you agree with them and or not.
• 2:15
  SPEAKER_01: And it's not driving you to something, it really is just yeah, here is cool stuff in AI.
• 2:20
  SPEAKER_00: Right.
• 2:21
  SPEAKER_00: So Katie and I both love side quests, and so uh there's a um uh slide 88 actually in this deck, and again, we'll link it in the content.
• 2:31
  SPEAKER_00: There's a quote from OpenAI that says, We cannot miss this moment because we are distracted by side quests.
• 2:38
  SPEAKER_00: And what they're talking about and showing in this chart is uh on the um on the left hand side, they show all of the OpenAI projects, if you will.
• 2:50
  SPEAKER_00: So from Dolly to ChatGPT plugins, the assistant API, Canvas, the evals platform, they show each one of these projects, and there's probably 10 or 11 of them there, and they show the time frame that each project was active for because all of these have been have been axed.
• 3:08
  SPEAKER_00: Um, and what is there, 10 or 12?
• 3:10
  SPEAKER_01: It's a whole bunch.
• 3:11
  SPEAKER_01: And honestly, like look at Sora, where I mean, if you were into creating videos, like Sora was the future, right?
• 3:18
  SPEAKER_01: Everyone's like, oh my god, we gotta all get into Sora, and it was seven months, right?
• 3:22
  SPEAKER_01: And they put a ton of work into Sora, right?
• 3:25
  SPEAKER_01: It was a really well-built platform, right?
• 3:27
  SPEAKER_01: You know, I think if we look back at like Dolly 2, I don't know that anybody was super psyched, like you can make an avocado couch, like okay, and it was an innovation we needed at the time, but like Sora really sort of was its own thing.
• 3:41
  SPEAKER_01: Like each of these were their own little piece, and they really like open AI has taken a windy path to get there.
• 3:48
  SPEAKER_01: Like, obviously, you know, Chat GPT has been there the whole time, enterprise and codex are there, but like when you compare it to Claude, Cloud's got the same set of things, they've not wavered.
• 3:58
  SPEAKER_00: 100%.
• 3:59
  SPEAKER_00: Yeah, and it's so it kind of boils down to side quests and and not, and the intersection on what open AI and Anthropic have been focused on is basically chat, co-work, and coding assistance.
• 4:13
  SPEAKER_00: Right.
• 4:13
  SPEAKER_00: Like those are the three that have are still first and foremost.
• 4:17
  SPEAKER_00: And I don't know, to me, I I read this and I feel like this is kind of a different version of, and I think we referenced this previously about just because you can build something maybe doesn't mean you should.
• 4:30
  SPEAKER_00: Like in today's world, we're you you know you can go prompt to prototyped pretty darn quickly, but kind of the nuance of are you building the right things in the right order for the right customers for sure.
• 4:44
  SPEAKER_01: I mean, I have really mixed feelings on it though, because if I look at the open AI world, some of these really were revolutionary.
• 4:51
  SPEAKER_01: Like, I think if you would have asked me a year ago to use Claude for anything related to images, I would have told you this is the biggest waste of time.
• 5:00
  SPEAKER_01: It can't read images, it's not gonna do a good job.
• 5:02
  SPEAKER_01: Like they really just had not focused in that area.
• 5:06
  SPEAKER_01: And open AI just constantly pushed the envelope in that.
• 5:10
  SPEAKER_01: Right.
• 5:10
  SPEAKER_01: So you have to give some credit to open AI and like these side quests, they m they look like side quests, and man, they really polish up their side quests, but they also did push the envelope.
• 5:22
  SPEAKER_01: So like we need that innovation, right?
• 5:24
  SPEAKER_01: But yeah, it kind of distracted them where you look at you know, cloud code versus codex, like really different world.
• 5:31
  SPEAKER_01: So interesting.
• 5:32
  SPEAKER_00: I honestly, I before we move on, I kind of wonder, you know, even Chat GPT was the product that wasn't supposed to be.
• 5:39
  SPEAKER_00: Right.
• 5:40
  SPEAKER_00: You know, so it's like they they kind of released it on a whim, it goes viral, and the world does never look back.
• 5:46
  SPEAKER_00: And I I kind of wonder when you look at Sora or you look at at some of these other products that have been shelved, like how much of that was well, we need to ship this stuff fast because maybe there's there's something here that was um, you know, worth shipping it kind of like the the original was.
• 6:04
  SPEAKER_01: Right.
• 6:04
  SPEAKER_01: 100%.
• 6:05
  SPEAKER_01: Like I think if you go and look at the the Dolly stuff as well, like right now, you would be really disappointed in Chat GPT if you had said, Hey, can you help me with this PowerPoint?
• 6:14
  SPEAKER_01: And it couldn't come up with a way to construct images.
• 6:18
  SPEAKER_01: And the only way that that happens is through innovation through Dolly and through these models where you do have a side quest.
• 6:25
  SPEAKER_01: And so super interesting.
• 6:27
  SPEAKER_01: And then I think the next one looking on there um was the token spending.
• 6:31
  SPEAKER_01: So on slide 85, you've got OpenAI and Anthropic capture 96% of the token spending trapped tracked by RAMP.
• 6:39
  SPEAKER_01: But again, this is just who's connected to ramp, so you're sort of limited on the exposure of what you're seeing.
• 6:44
  SPEAKER_01: But what I think is interesting is there's sort of this dichotomy where OpenAI was taking up more of the share, and Cloud is slowly taking up more and more of the share.
• 6:53
  SPEAKER_01: And so the question that I would get to there is the immediate thing that you'd think is, man, I guess people like Cloud more than they do Chat GPT.
• 7:03
  SPEAKER_01: But is it just that, well, we know Cloud reasons more, right?
• 7:07
  SPEAKER_01: Like Cloud is gonna spend more tokens.
• 7:09
  SPEAKER_01: Like that's sort of where you're at looking at the spending and the cost of these models.
• 7:14
  SPEAKER_01: So, you know, it's it's an interesting way to track, but I don't know that that's you know, the only thing I would track.
• 7:20
  SPEAKER_01: Just neat.
• 7:21
  SPEAKER_00: Yeah, I mean, it is interesting.
• 7:22
  SPEAKER_00: So the the things that jump out of me, and I agree with you, Katie.
• 7:25
  SPEAKER_00: It's like uh knowing that some model types are have a propensity to generate more tokens of different types, and that those all cost different.
• 7:34
  SPEAKER_00: But it's interesting that you know, I look back at at this year, I feel like it was over Christmas into February that Claude really took over for an open AI.
• 7:47
  SPEAKER_00: Yeah, yeah.
• 7:48
  SPEAKER_00: Okay, so it's interesting that even here, like Jan 26, according, you know, again, through RAMP, um, already Anthropic had 50% of the tokens.
• 7:58
  SPEAKER_00: And what I was feeling from January, February, March, uh push that up to almost looks like 70%.
• 8:06
  SPEAKER_00: Yeah.
• 8:07
  SPEAKER_00: And then now, yeah, I mean, I I do.
• 8:09
  SPEAKER_00: I feel like between Codex playing catch up, you know, where um where this has them is is back to more, almost to more of a 50-50 kind of a split.
• 8:19
  SPEAKER_00: But it looks like OpenAI has some momentum here.
• 8:21
  SPEAKER_01: For sure.
• 8:22
  SPEAKER_01: And I mean, you've got to look at from the the other standpoint, you know.
• 8:25
  SPEAKER_01: Of course, this is because it's just connected to RAM, but it's not every token that's spent, you know, you've got to look at what's happening with Google as well.
• 8:32
  SPEAKER_01: Like, I mean, if you go to the leaderboards right now, the ones that are sort of ELO focused, where you look like the arenas and it is people voting, Argon is top, like Argon is king, you know, it's in preview right now.
• 8:44
  SPEAKER_00: Right.
• 8:45
  SPEAKER_00: But that's Gemini, that's the new Gemini family, right?
• 8:48
  SPEAKER_01: The new Gemini that's coming out, and it looks like it's gonna be amazing.
• 8:52
  SPEAKER_01: Yeah, and they're doing it at it's half price right now.
• 8:54
  SPEAKER_01: Yeah, but their overall price is cheap, yeah.
• 8:58
  SPEAKER_01: Even when it's not half price, like right, so you know, it'll be interesting to see how you know changes in this evolve.
• 9:05
  SPEAKER_00: So, what's interesting too about that, I think, is the leaderboards versus like the perception of people find useful.
• 9:12
  SPEAKER_00: And if I could just tell a really quick anecdote.
• 9:15
  SPEAKER_00: So um traveling this weekend with my son, and we're you know, uh Tesla just rolled out the Grokbot kind of features.
• 9:23
  SPEAKER_00: Okay, so my son is constantly correcting me with his, you know, wide, impressive uh sports knowledge, always impressing, you know, always trying to correct me, and I give it to him.
• 9:34
  SPEAKER_00: He's usually right.
• 9:36
  SPEAKER_00: And so this weekend I was fact-checking him on he, you know, he's a huge Vikings fan, loves JJ McCarthy.
• 9:43
  SPEAKER_00: Obviously, JJ went to uh the Giants, but he was Jack was quoting that JJ McCarthy was good.
• 9:50
  SPEAKER_00: He went six and three last year, is what my son had said.
• 9:55
  SPEAKER_00: And Grok we were chatting with in the background, and as Jack is making his point, right after he says JJ went six and three last year, Grok chimes in and says he actually went six and four.
• 10:08
  SPEAKER_00: Just real-time fact checking.
• 10:10
  SPEAKER_00: I mean, I gotta say, you know, I know we talk about like what we think AI should be.
• 10:14
  SPEAKER_00: Right, that hit the mark for me.
• 10:16
  SPEAKER_00: And I and I'll just link it to what you said.
• 10:18
  SPEAKER_00: Like, there was no leaderboard.
• 10:20
  SPEAKER_00: There's no, it really is that was an experience where I actually I'll probably never forget that because that was an experience where it's like I feel like there is a third, there is a, there is third person, but there is like a there is a valuable thing in my car that is helping with this conversation.
• 10:37
  SPEAKER_00: It's also proven my son wrong, but also like like like helping, like this is I didn't ask it, right?
• 10:43
  SPEAKER_00: It heard something, it knew the conversation, and it correct, like I thought this is this is like one of those experiences.
• 10:50
  SPEAKER_00: I thought it was so great.
• 10:52
  SPEAKER_00: I you know, I know that's just an anecdote, but it's like there's no leaderboard.
• 10:56
  SPEAKER_00: How do you capture that on a leaderboard?
• 10:58
  SPEAKER_01: Well, so that's where I think you look at like the ELO based ones, right?
• 11:01
  SPEAKER_01: Where it is it is voting that is saying like this is the public opinion, but it's not truly public opinion, right?
• 11:07
  SPEAKER_01: It is opinion of people that are very, very well informed in the space, right?
• 11:11
  SPEAKER_01: Versus like, let's go and figure out like what did what does the beat on the street say?
• 11:16
  SPEAKER_01: You know, you gotta go and ask, like, what are people thinking about these you these tools?
• 11:22
  SPEAKER_01: Like, where where do you stand?
• 11:23
  SPEAKER_01: And honestly, I don't care who writes the best math paper 99% of the time.
• 11:28
  SPEAKER_01: I want the one that's gonna fact check, you know, my my sports facts.
• 11:32
  SPEAKER_01: I want to know how Jack reacted, though.
• 11:34
  SPEAKER_00: Well, we all, I mean, we laughed.
• 11:35
  SPEAKER_00: It was just so um, it was so funny, you know, because he could, you know, my son, he couldn't get mad at me because I didn't say it, right?
• 11:44
  SPEAKER_00: So I mean it was great, but but it's also if I could link to the next topic, it's also like, and I think we've kind of hit on this lately, but it's almost like, you know, the focus, so much focus on the model, the model, the model, the model.
• 11:57
  SPEAKER_00: And then, but but you know, you we sometimes I think kind of think differently or don't recognize the harness, like all of the stuff around the model, all of the instructions and and you know, whatever, whatever's in Grokbot that said, you know, and interject and or however I know I'm sure there's a lot to that, but I mean just just just the the thinking that and some of the data in the report here is pretty neat because it's you know, they're making the case on a few slides, specifically looking at slide 17 here, is really saying, you know, if you fix the models, like if you had if you had a hundred bucks to invest on making improvements in Chat GPT, I think of what some of this is saying is you might want to that the low-hanging fruit is actually to make a better harness and not focus on the model.
• 12:49
  SPEAKER_01: 100%.
• 12:50
  SPEAKER_01: And I think like, okay, I'm gonna tell you the dumbest thing I've said at NLP Logixly.
• 12:55
  SPEAKER_01: Um I guess that's an objective thing, but to me, the dumbest thing I've said at NLP Logixly was today.
• 13:02
  SPEAKER_01: 13 years ago, okay, and I remember, you know, I've I've just got my stats degree.
• 13:08
  SPEAKER_01: I want to talk about stats and machine learning, and I don't really care about anything else.
• 13:12
  SPEAKER_01: Um and having a conversation uh with Robbie at the time, and it was he wanted us to build dashboards.
• 13:23
  SPEAKER_01: And I said, we are a machine learning company, we are not a dashboard company.
• 13:30
  SPEAKER_01: And that is the dumbest thing I've ever said because when you look at it, even you know, take out the the large language models.
• 13:36
  SPEAKER_01: When you look at machine learning models in general, the thing that makes them incredible is all of the stuff around it.
• 13:43
  SPEAKER_01: Because it turns out if I just give you a CSV with a bunch of predictions, that is pretty much useless, right?
• 13:50
  SPEAKER_01: Like you need all of the stuff around it of like, how do I use it?
• 13:54
  SPEAKER_01: What am I gonna do with it?
• 13:55
  SPEAKER_01: What do I need?
• 13:56
  SPEAKER_01: Like all it's just continuing with the large language models, which is amazing.
• 14:01
  SPEAKER_00: Yeah.
• 14:01
  SPEAKER_00: So interesting too, there, you know, and that one of the data points here, I should probably quote this so you can get a feel for it, but it was basically saying, you know, they were uh on one of the evals, they were holding the uh model fix, looks like haiku 4.5, and they were with improvements only to the harness, they were able to improve the performance on this benchmark from 27.5% to 37.6%.
• 14:27
  SPEAKER_00: And some of this to me, Katie too, the other thing I look at is like when you know, when we build machine learning models, and this is since the beginning of time, right?
• 14:37
  SPEAKER_00: You know, we never called it a harness, but but the logic surrounding the pre-processing, post-processing, maybe a couple of business rules, you know, like really in hindsight, that's a little bit of a harness.
• 14:49
  SPEAKER_00: It's like it's like, hey, we have the core guts of a model, but there's things we know that we can take advantage of either on the inputs or the outputs that can make the outputs more um accurate even without retraining or doing something different.
• 15:05
  SPEAKER_01: 100%.
• 15:06
  SPEAKER_01: It's when I look at things like a cutoff and in computer vision, like I want to make a cutoff, and you know, I mean, you get the sort of default, I'm gonna, oh, 50%.
• 15:14
  SPEAKER_01: No, we're not using 50%.
• 15:15
  SPEAKER_01: But when it's really shiny, maybe I want it to be 95%.
• 15:19
  SPEAKER_01: And when it's dull, I want it to be 96% because they represent different things, and like those business logic and rules, that's what drove value.
• 15:28
  SPEAKER_01: Like that's always been the case.
• 15:30
  SPEAKER_01: And I think coolest fact on this, yeah, to me, this is like the Katie learned, like mind blown.
• 15:35
  SPEAKER_01: I think I told like six people out on the floor, like, oh my god, you guys have got to know.
• 15:38
  SPEAKER_01: Um, so from slide 14, Claude Code removed 80% of the system prompt for its advanced models, and there was no measurable loss in the coding.
• 15:48
  SPEAKER_01: So, like, this is going back to the, I mean, probably six months where everybody was convinced that prompt engineer was the most important job that would ever exist.
• 15:57
  SPEAKER_01: And, you know, if you give your prompt all of this context and all of these details, and you know, here's all of the things that you have to provide in order for this to be a good prompt, it kind of turns out that's not right.
• 16:08
  SPEAKER_01: You don't need like overwhelming the prompt isn't solving anything, it's just making it more complex and not really giving you a better result.
• 16:16
  SPEAKER_00: Yeah.
• 16:16
  SPEAKER_00: And and I do feel like this is I'm not saying this is making the opposite point of the point we just made, but it to me it just shows that there are cases where what once were prompts and even things in a harness can get pulled and baked into the model.
• 16:32
  SPEAKER_00: And it, you know, and it takes me back too, and we were talking about this internally, and I think everybody talks about this in some form, which is you know, how do you build all of the stuff around the models in ways that is, you know, forward and backward compatible, at least as much as possible.
• 16:49
  SPEAKER_01: And scalable.
• 16:50
  SPEAKER_01: I mean, it's it when you go like going back to the computer vision thing, right?
• 16:53
  SPEAKER_01: If I say, well, if the amount of you know R pixels is, you know, whatever, like if you go through your RGB and you write really distinct rules for all of those, you can memorize a data set, it doesn't make the model better.
• 17:07
  SPEAKER_00: Right.
• 17:07
  SPEAKER_01: But if you have really sort of broad rules, then you get the value without having to really overspecify.
• 17:14
  SPEAKER_01: So, you know, it's this thing where you're going and saying, like you could really make this complicated.
• 17:20
  SPEAKER_01: And it's kind of like, you know, if you go back to like the world of Kaggle contests right now, where back when it was about what features can you engineer, like what is the nuance that you can drive?
• 17:31
  SPEAKER_01: Like that's where we got awesome stuff that you could think about productionalizing, versus you know, now just like how do I make it as complex as possible?
• 17:40
  SPEAKER_01: And sure it might take 18 hours to get a prediction, but it's 0.01% better.
• 17:45
  SPEAKER_01: That's that's not the real world.
• 17:47
  SPEAKER_01: Right.
• 17:48
  SPEAKER_01: So I think it's awesome to see like no, we don't need to add complexity, like simple is always better.
• 17:54
  SPEAKER_00: Right.
• 17:55
  SPEAKER_00: Yeah, 100%.
• 17:56
  SPEAKER_00: Any, you know, simplify, simplify, simplify.
• 17:59
  SPEAKER_00: And then the last, you know, one of the last areas, I feel like this is kind of related to some of the stuff we've been talking about, real at least adjacently related to the topic last week of you know, how we're making progress on math.
• 18:13
  SPEAKER_00: And math is it, it's just so interesting, right?
• 18:15
  SPEAKER_00: It's like math is hard.
• 18:18
  SPEAKER_00: I think everybody who, you know, except for you who majors in math.
• 18:22
  SPEAKER_00: I mean, like every right, like you ask a parent what's the subject your students or your children struggle with.
• 18:28
  SPEAKER_00: Oh, it's math, right?
• 18:29
  SPEAKER_00: They they get geometry, whatever.
• 18:30
  SPEAKER_00: Like everybody has their thing.
• 18:31
  SPEAKER_00: So, but it's so that's why I feel like it's so interesting that you know these language models, OpenAI anthropic, they're making great progress on these really hard math problems that we've had hundreds hundreds, hundreds of years to work on as humans.
• 18:48
  SPEAKER_00: We're making progress, but not at the pace that obviously AI can make a progress at.
• 18:54
  SPEAKER_00: And then it's like, you know, for us, at least for me, if I were to explain that to someone, like say I were to go home and and explain it to you know, my wife, to her, I think that's gonna sound like genuine novel innovation.
• 19:06
  SPEAKER_01: Right.
• 19:07
  SPEAKER_00: Right.
• 19:07
  SPEAKER_00: And then and and I get that because it it is it is the the the proof is novel, but it's like it I mean, how would you explain that?
• 19:15
  SPEAKER_00: Because it it is novel, but but what but when you look at some of the other tests for how AI can be used to invent things or to do novel things in other domains that humans are doing, it seems like it's it's not like it's just across the board.
• 19:33
  SPEAKER_01: It's the Einstein quote, right?
• 19:35
  SPEAKER_01: Like it's not about you know, how complex can you make this?
• 19:39
  SPEAKER_01: Can you teach this?
• 19:39
  SPEAKER_01: Can somebody else understand it?
• 19:41
  SPEAKER_01: When you're looking at a research paper, yeah, I should be able to go if I'm an expert in that area.
• 19:46
  SPEAKER_01: Like, obviously, if you don't know that area, like you're not gonna go and look at it, but if I'm an expert in that area, I should be able to read your paper and go, oh my god, why didn't I think of that?
• 19:54
  SPEAKER_00: And is that like the bar?
• 19:56
  SPEAKER_01: I mean, I I feel like you gotta go and you I should be able to read this and understand what's happening.
• 20:01
  SPEAKER_00: Okay, so it and because I have no idea how this works.
• 20:04
  SPEAKER_00: So if if you're if I'm reviewing a paper for you and I'm in your field and you're you want to get this paper submitted, I know there's this process with peer review.
• 20:15
  SPEAKER_00: When I read your paper, should I be thinking, oh, I that's the sign of a good paper is me being like, I should have thought of this.
• 20:23
  SPEAKER_01: Yeah, or you should be able to at least understand, like read through it and go, Yeah, that totally makes sense.
• 20:28
  SPEAKER_01: That all checks out.
• 20:29
  SPEAKER_01: If you're if you're a human and you write a really convoluted paper and it has a good proof in it, but it's really just like, why would you even take that step?
• 20:37
  SPEAKER_01: Like, this makes no sense.
• 20:38
  SPEAKER_01: Like, you're gonna get comments that are like, go back and clean this up.
• 20:42
  SPEAKER_01: Okay, go fix this.
• 20:43
  SPEAKER_00: Yeah.
• 20:43
  SPEAKER_01: And so, like, that's what people are saying right now about open AI and the papers that they're putting out, they're like, Yeah, this is uh it's it's possibly a proof, but it also is gobbledygook that like I have to get open AI to explain, like, hey, can you tell me what you're trying to say here?
• 21:01
  SPEAKER_01: Right?
• 21:02
  SPEAKER_01: Like, yeah, so I it goes sort of goes back last week.
• 21:05
  SPEAKER_01: We talked about just because you've got code that solves a problem doesn't mean it's good code, right?
• 21:09
  SPEAKER_01: Same thing here.
• 21:10
  SPEAKER_01: Like, just because you can show a proof doesn't mean that it's a beautiful proof or even a good proof, right?
• 21:16
  SPEAKER_01: Like it's just like okay, you kind of got a result, yeah.
• 21:18
  SPEAKER_01: But like this is so interesting because they are saying, I think it was for NeuroIP submissions, and they had agents that got three thousand dollars of API credit, GPUs, open web, go do your whole thing.
• 21:33
  SPEAKER_01: And the engineering was good, like the actual underlying work was good, but the papers are terrible, like they got awful reviews, right?
• 21:43
  SPEAKER_00: Like, yeah.
• 21:45
  SPEAKER_00: So this is so this is slide seventeen.
• 21:47
  SPEAKER_00: So this is in uh in NeuroIps to me is I always think of it as the machine learning conference in November, December focused on deep learning.
• 21:59
  SPEAKER_00: And You know, going back 10, 15 years, like this is where all of the novel work in deep learning would have been presented annually.
• 22:08
  SPEAKER_00: Right.
• 22:08
  SPEAKER_00: Yep.
• 22:09
  SPEAKER_00: So what they what they did, um, and I don't know if this was this year or last year, but they they it says that they gave an agent a central research question of a high quality, unpublished neuroip submission.
• 22:23
  SPEAKER_00: So basically saying there's no way that anyone could cheat and find relevant information.
• 22:27
  SPEAKER_00: So this was novel work that a human had done, and they let the paper's original authors grade the output as reviewers.
• 22:36
  SPEAKER_00: So then they they asked um the agent to go work on this research, and they let the paper's original authors grade the output as reviewers.
• 22:44
  SPEAKER_00: And that was getting back to my reviewer question to you, Katie.
• 22:47
  SPEAKER_00: Right.
• 22:47
  SPEAKER_00: So the agent got six days,$3,000 of API credits, GPUs, and like you said, the open web.
• 22:53
  SPEAKER_00: And honestly, the the results, quality, clarity, significance, or originality, overall confidence of the papers all were too low to get published, I would say.
• 23:03
  SPEAKER_01: Right.
• 23:04
  SPEAKER_01: I love the statement in clarity.
• 23:06
  SPEAKER_01: We've got one that's a one out of four and one that's a two out of four, and it is dense, unclear writing, and it's hard to tell what matters.
• 23:12
  SPEAKER_01: And it is, I will always go back, and I'm sure it's better now.
• 23:15
  SPEAKER_01: But like one of my things that I know about basketball, I don't know much about basketball because honestly, it's tall people showing off.
• 23:22
  SPEAKER_01: Um so I don't know a ton, but I, you know, grew up in Utah, and so John Stockton is a name that I know.
• 23:29
  SPEAKER_01: Oh, yeah.
• 23:30
  SPEAKER_01: And I took a basketball data set and I gave it to JadGPT.
• 23:33
  SPEAKER_01: This is probably like early fours, and said, you know, hey, can you tell me like the best stats?
• 23:38
  SPEAKER_01: And of course, the first question, like, you know, give me LeBron versus Jordan, and you know, arguably Jordan.
• 23:44
  SPEAKER_01: Um, thank you, Chat GPT.
• 23:46
  SPEAKER_01: I'm glad you sided with me.
• 23:47
  SPEAKER_01: Um, but when you look at the um the question of like what is John Stockton's best stat, it came back and said, minutes played.
• 23:59
  SPEAKER_01: And it's like, yeah, he did have a lot of minutes played.
• 24:02
  SPEAKER_01: That's really great.
• 24:03
  SPEAKER_01: Um, it's not really what I'm thinking of when I'm thinking of John Stockton.
• 24:08
  SPEAKER_01: You know, I might be looking for assist or something along that line, but you know, we're gonna go and it it doesn't always know, like this is this is actually what matters.
• 24:18
  SPEAKER_01: Sometimes reading a chat GPT response to me, I don't know.
• 24:22
  SPEAKER_01: Do you do you ever go and read recipes online?
• 24:25
  SPEAKER_01: I feel like that's a really unfair question.
• 24:27
  SPEAKER_01: I mean, not recently.
• 24:28
  SPEAKER_01: But like, I'll go and read a recipe, and it's like, I just want to make some cookies.
• 24:32
  SPEAKER_01: I don't need your life story.
• 24:33
  SPEAKER_01: Yeah, and they go and they're like yeah, yeah, it's like 50 pages of just irrelevant stuff.
• 24:38
  SPEAKER_00: Scroll more ads, scroll below the ads.
• 24:41
  SPEAKER_00: Right, and then there's the second half, scroll.
• 24:43
  SPEAKER_00: Yep.
• 24:44
  SPEAKER_01: And then at the end, here's a recipe, and it's like, I really just needed this one chunk, and that is what OpenAI, it's what Claude, it's what all of the models constantly do, is like, don't I only care about this one piece that matters, but like, man, you gotta give me the whole backstory in order to get there.
• 25:02
  SPEAKER_01: And it turns out humans don't really like reading that.
• 25:05
  SPEAKER_00: Right.
• 25:05
  SPEAKER_00: So I'm curious in in this research paper, is it and I and I'm curious if if we know the answer from slide 17 here, is it it it might not get submitted because it was written poorly, but but is there the novel contribution that they were looking for?
• 25:23
  SPEAKER_00: Is that is that was that part achieved?
• 25:28
  SPEAKER_01: Right.
• 25:28
  SPEAKER_01: I like I from reading this, yeah, it seems like all of the engineering was done, right?
• 25:33
  SPEAKER_01: So like that piece is there, and like theoretically it's high quality and unpublished, so okay, there is something there.
• 25:39
  SPEAKER_01: Yeah, but when it writes for originality, like we got a three out of four and a two out of four.
• 25:44
  SPEAKER_00: Right.
• 25:45
  SPEAKER_01: Um, so you know, it's it's I'd be really interested to see how the human papers scored on this.
• 25:51
  SPEAKER_01: Yeah.
• 25:51
  SPEAKER_01: So when a human writes the context, did it come out differently?
• 25:54
  SPEAKER_00: I guess it'd be interesting too to know more details about the problem.
• 25:57
  SPEAKER_00: But I I thought that was interesting just to see that.
• 25:58
  SPEAKER_00: The other one I think is interesting.
• 26:00
  SPEAKER_00: So this is slide 44 in the deck, and it's talking about um using the bit the title is The House Wins, Every Model Loses Money on Kelly Bench Sports Betting.
• 26:12
  SPEAKER_00: And um, so Kelly Bench is obviously a benchmark on um using uh AI to um you know maximize gains on wagering specifically in sports.
• 26:25
  SPEAKER_00: I find this one interesting because I I think that there is a lot of talk and probably doing about how to leverage AI for trading and other you know automated, um automated uh, you know, any kind of markets, right?
• 26:42
  SPEAKER_00: Everybody, what everybody wants to do is to create the algorithm that lets them make money repeatedly in markets without doing anything.
• 26:49
  SPEAKER_00: And I just think it's interesting that you know, even um, you know, I think every financial institution has some form of AI-based, you know, trading recommendations.
• 27:02
  SPEAKER_00: 100%.
• 27:03
  SPEAKER_00: Yeah, and and when you back test them, it's like it, you know, it it the edges are maybe not coming from these algorithms, you know, and then and they're coming from elsewhere.
• 27:13
  SPEAKER_00: So I just I I like it's like the dichotomy, like I feel like we're doing this a lot, where it's like, you know, you you you you ha we have these amazing innovations in mathematics, and yet at the same time we have these shortcomings in research or shortcomings and areas where maybe you would think AI is just always gonna be better.
• 27:33
  SPEAKER_01: Right.
• 27:33
  SPEAKER_01: I mean, like if you look at it from you've been doing this 15 years, how many times have you met somebody and they've said, Yeah, can you build me a stock model?
• 27:42
  SPEAKER_00: I can't tell you how many, you know, I really could not.
• 27:45
  SPEAKER_00: I I it's it's it's probably some version of that, yeah, is probably over 15 years, I would say at least two to three times a year.
• 27:55
  SPEAKER_00: That that that's not a that's not even a joke, that somebody is looking for some sort of algorithmic.
• 28:01
  SPEAKER_01: Um, but even as a joke, like if I'm sitting at a bar, it's gonna be like, hey, can you uh build me a model to predict who wins the game?
• 28:07
  SPEAKER_01: And it's like, you know, honestly, like I pull this back to, you know, I'll always talk about wounds.
• 28:12
  SPEAKER_01: Like, we did a a model to predict whether or not a wound's gonna heal.
• 28:15
  SPEAKER_01: And you we could get like AUC got to like high 70s, so like it did a pretty good job of of explaining like this is who's likely to heal and who's not, but there's factors you're never capturing, and it's things like do you go home to a clean house?
• 28:31
  SPEAKER_01: Do you have a good diet?
• 28:32
  SPEAKER_01: Like these are things that you know, they're not available in the data, but they're really critical to the outcome.
• 28:38
  SPEAKER_01: And I think that like you do not have the ability in data, like in sports.
• 28:44
  SPEAKER_01: I mean, I I think we we asked somebody once because their dad was a horse trainer, and it's like, hey, can you tell um you know if a horse is doing good?
• 28:52
  SPEAKER_01: And he's like, Oh, if you meet it, like if you go and see the horse, yeah, from from day one, you know if this is gonna be a good horse or not, because you can see like the desire to win and things like that.
• 29:02
  SPEAKER_01: If you watch, you know, uh go to a lot of hockey games, you can see from the first minute, like this is gonna, you're gonna get some fights here, this is gonna get chippy.
• 29:10
  SPEAKER_01: But uh it's from like human intuition, it's not just data.
• 29:14
  SPEAKER_01: Like the these uh the Premier League is not just, you know, here's 17 pieces of data.
• 29:20
  SPEAKER_01: And if you look at all of those, then you're gonna get the perfect answer.
• 29:22
  SPEAKER_01: Like it's it is so much more complex than that.
• 29:25
  SPEAKER_00: Right.
• 29:25
  SPEAKER_00: And and I should also say, too, because it's not like you know, you have organizations like Jane Street and everyone else that does this really well.
• 29:33
  SPEAKER_00: And I I can take a guess that they're not putting this into Chat GPT off the shelf, that they have their own highly customized algorithms.
• 29:43
  SPEAKER_00: And honestly, to your point, Katie, the two examples you just said, like the horse example is or the racing example is there's other data points that maybe you don't have, but somebody else has, or your healing standpoint of like those are data points.
• 29:58
  SPEAKER_00: Right.
• 29:59
  SPEAKER_00: If you could get them, you could make better decisions.
• 30:02
  SPEAKER_00: And certainly, you know, organizations, hedge funds that that do stuff like this, you know, the you know, the build algorithms to work in markets, they also deal with a lot of other invest heavily in other things that give them advantage, right?
• 30:17
  SPEAKER_00: Data points, you know.
• 30:19
  SPEAKER_01: I mean, I think the coolest sort of math story of all time, and this is actually a fun fact.
• 30:25
  SPEAKER_00: Yeah.
• 30:25
  SPEAKER_01: Um, I went and saw you speak before I started NLP.
• 30:29
  SPEAKER_01: Really?
• 30:29
  SPEAKER_01: Yeah, it was like it was Ben that just started here and he was like, Hey, you want to come see my boss?
• 30:33
  SPEAKER_01: He's pretty cool.
• 30:33
  SPEAKER_01: And I was like, Yeah.
• 30:35
  SPEAKER_01: So um, and you talked about a topic that I've always found really fun, and I was like, Okay, this guy's pretty great.
• 30:41
  SPEAKER_01: And it was long-term capital management.
• 30:42
  SPEAKER_00: Oh, yes.
• 30:43
  SPEAKER_01: And so, like long-term capital management had like the money printing machine.
• 30:47
  SPEAKER_01: Yes, I mean, they were 40%, 40%, you know, 40, 42, and then like 25, I think.
• 30:52
  SPEAKER_01: And then all of a sudden they collapsed the global economy, and it's like, well, you weren't considering what's happening in the Russian market, you know, like it's these like tiny nuanced pieces that a human is watching for that models don't.
• 31:07
  SPEAKER_00: Yep.
• 31:08
  SPEAKER_01: And I don't know that we're gonna get to a point where AI can understand that.
• 31:13
  SPEAKER_01: Right.
• 31:13
  SPEAKER_01: Like it's just there's there's things, and like maybe we will.
• 31:17
  SPEAKER_01: Like, what do I know?
• 31:18
  SPEAKER_01: But you know, there's these little nuances that, like, yeah, I'd rather it correct your son in the car than uh replace, you know, the joy of sports betting.
• 31:28
  SPEAKER_00: Well, I love the fact that it's not just me correcting my son in the car.
• 31:32
  SPEAKER_00: Because that goes, it goes a lot further when it's somebody else, even if it's even if it's AI.
• 31:37
  SPEAKER_00: But I have a feeling we'll be revisiting more um state of AI, um, more parts of the report, like you said, hundreds of pages, lots to dig into.
• 31:46
  SPEAKER_00: We might come back to it.
• 31:47
  SPEAKER_00: It would surprise me if we don't.
• 31:49
  SPEAKER_00: Um, but we're gonna leave it there.
• 31:50
  SPEAKER_00: So that's it for today for Logixly Lee Live.
• 31:53
  SPEAKER_00: Um, obviously want to thank Katie and and I'm gonna thank Sarah and Wendy today because they they get to listen to us all the time.
• 32:00
  SPEAKER_00: I'm sure they get really tired of us.
• 32:02
  SPEAKER_00: But uh, if we if we if there's anything else you want to hear, any topics we don't cover, please reach out.
• 32:07
  SPEAKER_00: We love to connect.
• 32:08
  SPEAKER_00: And until next time, keep learning, even when it's unsupervised.