Latest / In the Loop AI / OpenRouter: The Only AI Tool You'll Ever Need | Founder Alex Atallah

Transcript

OpenRouter: The Only AI Tool You'll Ever Need | Founder Alex Atallah · In the Loop AI

0:03
Ejaaz: What if I told you there was a single website you could go to where you can
0:07
Ejaaz: chat to any major AI model from one single interface?
0:11
Ejaaz: It's kind of like chat GPT, but instead every prompt gets routed to the exact
0:16
Ejaaz: AI model that will do the best job for whatever your prompt might be.
0:20
Ejaaz: Well, on today's episode, we're joined by Alex Atala, the founder and CEO of Open Router AI.
0:25
Ejaaz: It's the fastest growing AI model marketplace with access to over 400 LLMs,
0:31
Ejaaz: making it the only place that really knows how people use AI models,
0:34
Ejaaz: and more importantly, how they might use them in the future.
0:37
Ejaaz: It's at the intersection of every single prompt that anyone writes and every
0:42
Ejaaz: model that they might ever be.
0:43
Ejaaz: Alex Atala, welcome to the show. How are you, man?
0:46
Alex: Thanks, guys. Great. Thanks so much for having me on.
0:48
Ejaaz: So it is a Monday. How does the founder of OpenRouter spend his weekend?
0:53
Ejaaz: Presumably you know out and about chilling relaxing not at all focused on the company oh
0:58
Alex: I usually i love weekends with no
1:01
Alex: meetings planned and i just go to a coffee shop and just have tons of hours
1:06
Alex: stacked in a row uh to do things that require a lot of momentum build up so
1:11
Alex: i did that at coffee shops on saturday and sunday and then i watched blade runner again.
1:20
Ejaaz: Again okay um well so
1:23
Ejaaz: when we were preparing for this episode alex
1:27
Ejaaz: um i couldn't help but think that you've had a pretty insane decade of startup
1:35
Ejaaz: foundership right um so open router is kind of like your second major thing
1:40
Ejaaz: that you've done but prior to doing that you were the founder and cto of OpenSea,
1:45
Ejaaz: the biggest NFT marketplace out there.
1:48
Ejaaz: And now you're focused on one of the biggest AI companies out there.
1:51
Ejaaz: So it sounds like you're at kind of like the pivot point of two of the most
1:56
Ejaaz: important technology sectors over the last decade.
1:59
Ejaaz: Can you just give us a bit of background as to how you ended up here?
2:04
Ejaaz: And more importantly, where you started.
2:06
Ejaaz: Walk us through the journey of OpenSea and how you ended up at OpenRouter AI.
2:10
Alex: Yeah, so I co-founded OpenSea with Devin Finzer the very beginning of 2018, very end of 2017.
2:20
Alex: It was the first NFT marketplace. And...
2:25
Alex: It was not dissimilar to OpenRouter in that there was a really fragmented ecosystem
2:33
Alex: of NFT metadata and media that gets attached to these tokens.
2:40
Alex: And it was the first example of something in crypto that could be non-fungible,
2:47
Alex: meaning it's a single thing that can be traded from person to person.
2:51
Alex: Most things in the world are non-fungible. A chair is non-fungible. a
2:54
Alex: currency is fungible so it was
2:58
Alex: back back in 2018 no
3:02
Alex: one was really thinking about crypto in terms of non-fungible goods
3:05
Alex: and uh and the problem with the non with
3:08
Alex: non-fungible goods is that there weren't any real standards set up
3:11
Alex: um there was a lot of heterogeneous like
3:15
Alex: implementations for how to get uh like
3:18
Alex: a non-fungible item represented and tradable in a decentralized way So OpenSea
3:24
Alex: organized this like very heterogeneous inventory and put it together in one
3:33
Alex: place. We came up with like a metadata standard.
3:36
Alex: We did a lot of like a lot of work to really make the experience super good for each collection.
3:45
Alex: And you see a lot of those a lot of similarities with how AI works today,
3:50
Alex: too, where there's also just a very heterogeneous ecosystem.
3:53
Alex: On a lot of different APIs and different features supported by language model providers.
4:00
Alex: And Open Router similarly does a lot of work to organize it all.
4:06
Alex: Um i was at open sea uh until 2022 um when i was kind of feeling the itch to do something new.
4:16
Alex: And um i'm at the very end of i left in august and then chat gpt came out a few months later.
4:24
Alex: And uh and my biggest question around that
4:27
Alex: time was whether it was going to be a winner take all market
4:30
Alex: because opening i was very far ahead of
4:33
Alex: everybody else and um you know
4:36
Alex: we had cohere command we had a couple open source models um
4:39
Alex: but opening i was the only really usable one i
4:42
Alex: was doing little projects to experiment
4:45
Alex: with the gpt3 api and uh
4:48
Alex: and then llama came out in january um really
4:52
Alex: exciting about a tenth the size one on a
4:55
Alex: couple benchmarks but it wasn't really chattable yet and
4:59
Alex: uh and it wasn't until uh a few
5:02
Alex: months later that somebody a team at
5:05
Alex: stanford distilled it into a new
5:09
Alex: model called alpaca um distillation means
5:12
Alex: you you take the model and you customize it or fine tune it
5:15
Alex: on a set of synthetic data that they
5:18
Alex: made using chat gpt as a research project
5:21
Alex: and uh and that was it was
5:24
Alex: the first successful major distillation that i'm aware
5:27
Alex: of um and it was an actually usable model i
5:30
Alex: was like on the airplane talking to him i was like wow this
5:33
Alex: is if it only took six hundred dollars to make something like this then you
5:36
Alex: don't need ten million dollars to make a model there might be like tens of thousands
5:40
Alex: hundreds of thousands of models in the future and suddenly this started to look
5:44
Alex: like a new like economic primitive a new building block that people that kind
5:51
Alex: of deserve their own place on the internet.
5:52
Alex: And there wasn't one. There wasn't a place where you could discover new language
5:56
Alex: models and see who uses them and why.
5:59
Alex: And that's how OpenRouter got started.
6:02
Josh: That's amazing. So one of the things that we're obsessed with on this channel
6:05
Josh: in particular is exploring frontiers and how to properly see these frontiers
6:10
Josh: and analyze them and understand when they're going to happen.
6:12
Josh: And when I was going through your history, you have this talent consistently over time.
6:16
Josh: And even as far back as early on, I read you were hacking Wi-Fi routers in a hackathon.
6:21
Josh: You're very early to that. You were early to the NFTs. You were early to understanding
6:25
Josh: AI and the impact that it would have. And what I'd love for you to explain is
6:28
Josh: the thought process and the indicators you look for when exploring these new
6:31
Josh: frontiers, because clearly there's some sort of pattern matching going on.
6:34
Josh: Clearly you have some sort of awareness of what will be important and why it
6:38
Josh: will be important, and then inserting yourself into that narrative.
6:41
Josh: So are there patterns? Are there certain things that you look for when searching
6:45
Josh: for these new opportunities and that led you to make these decisions that you have?
6:49
Alex: I think there's there's a lot to be said for finding enthusiast communities
6:55
Alex: and and seeing if you're going to join it.
6:58
Alex: Like, can you be an enthusiast with them?
7:02
Alex: Like whenever something new comes out that has like some kind of ecosystem potential,
7:08
Alex: there's there are going to be enthusiast communities that pop up.
7:11
Alex: And the Internet has made it self-certain. You could just join the communities.
7:17
Alex: Um discord i think is a incredible
7:20
Alex: and super underrated platform because
7:24
Alex: the communities feel kind of private you're
7:27
Alex: like getting you don't feel like you're you know
7:30
Alex: seeing somebody trying to get s you
7:33
Alex: know like advertise something for seo juice there's
7:36
Alex: no seo juice in discord um it's it's
7:40
Alex: just people talking about what they're passionate about and and it
7:43
Alex: goes it gets really niche um and when
7:45
Alex: you find a like an interest group in discord that
7:49
Alex: like has to do with some some new
7:52
Alex: piece of technology that's just being developed right now and doesn't really
7:57
Alex: work very well at all um you get people who are just trying to figure out what
8:02
Alex: to do with it and how to make it better and i think that's like that's the first
8:07
Alex: core piece of magic that jumps to mind,
8:11
Alex: there's got to be like a willingness to be weird because if you jump into any
8:17
Alex: of these communities at face value it's stupid.
8:21
Alex: Like oh this is like just a game or it's like a really weird game I mean I'm
8:26
Alex: not really a collectible game so I'm going to leave right now and yeah.
8:34
Alex: Not only do you have to be aware, but you have to be creative.
8:36
Alex: Like, okay, these are just cats on the blockchain, and people are just trading cats back and forth.
8:43
Alex: You can't look at the community as simply that.
8:49
Alex: Think about what you could do with it.
8:52
Alex: Like, what is this unlock that wasn't achievable before?
8:56
Alex: Um and uh
8:59
Alex: and and i think there are there are people who
9:02
Alex: just are good who will do this and they'll join the communities
9:05
Alex: and and brainstorm live and you can see everybody
9:08
Alex: brainstorming uh in real time but like
9:11
Alex: another incredible example of this was the mid-journey discord
9:15
Alex: you know it became the
9:18
Alex: biggest biggest server in discord by
9:21
Alex: far uh and you know
9:25
Alex: why did that happened well you could it started with
9:28
Alex: something weird silly maybe
9:32
Alex: not super useful but you could see all the
9:34
Alex: enthusiasts like remixing and
9:38
Alex: brainstorming live how to turn it into something beautiful
9:41
Alex: and how to how to make it useful and um
9:45
Alex: and then you know just explode it like i it's the most it's the it's the most
9:53
Alex: incredible like niche community uh i think that discord has ever seen because
9:59
Alex: of like how useless it started and how insanely exciting it became.
10:06
Alex: So um like i mean i i
10:09
Alex: think i saw big sleep i was like playing around with this model
10:12
Alex: called big sleep in 2021 that uh
10:16
Alex: let you generate images that
10:19
Alex: look kind of like deviant art okay and
10:22
Alex: uh you could see you could like they're all
10:25
Alex: animated images and they none of them really made sense but you could get some
10:29
Alex: really cool stuff not like potentially something you'd want to make your desktop
10:32
Alex: wallpaper and if you're really like deep in some deviant art communities you
10:38
Alex: know you kind appreciate it and so and that that that was like oh there's like
10:41
Alex: a kernel of something here,
10:43
Alex: and uh it took like a like another year or two before mid-journey started to
10:49
Alex: like pick up but that was like.
10:51
Ejaaz: Where were you seeing all of this alex like where were you scouring just random
10:56
Ejaaz: forums or just wherever your nose told you to go
10:59
Alex: But basically there's this twitter account I'm trying to remember what it's
11:03
Alex: called that posts AI research papers and and like kind of tries to show what you can do with them.
11:13
Alex: And I discovered this Twitter account in like 2021.
11:19
Alex: And I.
11:21
Alex: I think it was not it was it wasn't at all like related to crypto but it was
11:27
Alex: a way you know big sleep was like the first thing i saw that used ai to generate
11:32
Alex: things that could potentially be nfts,
11:35
Alex: so i started experimenting around like how how much you could direct it to make
11:42
Alex: an nft collection that would make any sense it was very very difficult um but
11:49
Alex: that was how uh that was like the first generative and.
11:54
Ejaaz: This was before you were even thinking about starting open router right
12:01
Alex: Um yeah yeah this was back this was when i was
12:04
Alex: full-time at openc um oh is
12:07
Alex: yeah i got the it's a colic
12:10
Alex: this twitter account all right
12:13
Alex: i really recommend it they basically post papers and like explainate and explore
12:20
Alex: how this paper gets useful um they post animations uh like they make they make
12:27
Alex: ai research like kind of fun to engage with and that was that was my first experience.
12:33
Ejaaz: Okay, so I mean that's a massive win for X or formerly as it was known back
12:39
Ejaaz: then, Twitter as a platform, right?
12:41
Ejaaz: It gave birth to kind of like two of the biggest technologies crypto,
12:44
Ejaaz: also known as crypto Twitter, and now apparently all the AI research stuff which
12:49
Ejaaz: kind of put you on to the path that led you to OpenRatter.
12:53
Ejaaz: So if I've got this right, Alex, you were full-time at OpenSea with a
12:59
Ejaaz: multi-billion dollar company loads of important stuff to do there,
13:02
Ejaaz: but you still found the time to kind of scour this fringe technology because
13:07
Ejaaz: that's what AI was at the time.
13:09
Ejaaz: Prior to kind of GPT-2 or GPT-3, no one really knew about this.
13:13
Ejaaz: And you were playing around with these gen AI models, these generative AI models
13:17
Ejaaz: that would create this magical little substance and maybe it came in the form
13:22
Ejaaz: of a pitcher or a weird little cat.
13:25
Ejaaz: And you kind of like jumped into these niche forums of enthusiasts,
13:28
Ejaaz: as you say, and kind of explored that further.
13:32
Ejaaz: And it sounds like you kind of like honed that even beyond your journey from OpenSea when you left.
13:37
Ejaaz: I remember actually meeting you in this kind of like this abbess between you
13:43
Ejaaz: leaving OpenSea and starting OpenRouter where you were kind of brainstorming
13:48
Ejaaz: a bunch of these ideas. And I remember a snippet from our conversation
13:53
Ejaaz: In like one of the WeWorks here, where you just kind of like had whiteboarded a bunch of AI stuff.
13:58
Ejaaz: And one of those things was kind of like the whole topic of inference.
14:02
Ejaaz: And if I'm being honest with you, I had no idea what that word even meant back then.
14:07
Ejaaz: I was extremely focused on all the NFT stuff and all the crypto stuff,
14:12
Ejaaz: my background's in all of that.
14:13
Ejaaz: But I just found that fascinating that you always had your nose in some of the
14:17
Ejaaz: early communities. And I think that's a really important lesson there.
14:21
Ejaaz: I want to pick up on something that you actually brought up when you said you
14:27
Ejaaz: discovered kind of like your path to open router, Alex.
14:31
Ejaaz: And that is, you said you were playing around with these early AI models.
14:36
Ejaaz: So not the GPTs before Claude was even created.
14:40
Ejaaz: You're playing around with these random models that you would find either on
14:43
Ejaaz: forums, on Twitter, or on Reddit, right? and you would experiment with them.
14:49
Ejaaz: And I find it fascinating that back then, even when GPT became a thing,
14:54
Ejaaz: you were convinced that there would be hundreds of thousands,
14:57
Ejaaz: or did you say hundreds of thousands of AI models?
14:59
Ejaaz: Back then, that wasn't a normal view.
15:02
Ejaaz: Back then, everyone was like, you need hundreds of millions of dollars.
15:06
Ejaaz: Maybe it was tens of millions of dollars back then. And it was going to be a rich man's game.
15:11
Alex: Yeah, it was basically the Alpaca Project that kind of put me over the sack.
15:18
Alex: On there being many, many, many models instead of just a very small number.
15:24
Ejaaz: And can you explain what the Alpaca project is for the audience? Yeah.
15:29
Alex: So the Alpaca project, after Lama came out, you really could not chat with it
15:38
Alex: very well. It was a text completion model.
15:41
Alex: There were a couple benchmarks where it beat GPT-3.
15:46
Alex: And... It was about a tenth the size of what most people thought GPT-3 was sized at.
15:54
Alex: So it was a pretty incredible achievement.
15:57
Alex: But it wasn't really like, the user experience wasn't there.
16:01
Alex: And the Alpaca project took ChatGPT and generated a bunch of synthetic outputs.
16:09
Alex: And then they fine-tuned Llama on those synthetic outputs.
16:14
Alex: And this did two things to Llama. It taught it style, and it taught it knowledge.
16:21
Alex: It taught it, like, the style is like how to chat, which was the big user experience gap.
16:26
Alex: And it made it smarter.
16:30
Alex: Like, you can, fine-tuning transfers both style and knowledge.
16:34
Alex: And the model would, like, respond to things that it had, you know,
16:37
Alex: like, the content of the synthetic data, like, was reflected in the model's
16:42
Alex: performance on benchmarks after that point.
16:44
Alex: So um so if you can do
16:47
Alex: that without revealing all
16:50
Alex: the data that goes in um now now
16:54
Alex: there's like a way you could sell data via api without
16:57
Alex: like like just dumping all the data out to the world and then never being able
17:02
Alex: to to like monetize it again so there's like a brand new business model around
17:07
Alex: data that emerges um yet like the ability to create just like work towards open intelligence,
17:16
Alex: and uh and build like new
17:20
Alex: architectures test them more quickly and and and
17:23
Alex: uh uh fine-tune them quickly basically you
17:26
Alex: can build on top of the work of giants i mean
17:29
Alex: you don't have to start from zero every time a lot
17:32
Alex: of like the biggest developer experience innovations just involve like giving
17:38
Alex: developers a higher stair to start walking up so they don't have to start at
17:43
Alex: the bottom of the staircase every single time um and you know that was like the the the big.
17:52
Alex: Like generous give that llama had for the community um and it wasn't you know
17:58
Alex: that wasn't the only company doing open source models, Mastral,
18:02
Alex: came out with 7B Instruct a few months later. It was an incredible model.
18:08
Alex: Then they came out with the first open-weight mixture of experts a few months later.
18:15
Alex: It felt like actual intelligence, but completely open.
18:19
Alex: And all of these provide higher and higher stairs for other developers to kind
18:24
Alex: of like, basically to crowdsource new ideas from the whole planet.
18:30
Alex: Uh and and let these new ideas build on
18:33
Alex: top of really good foundations so and
18:37
Alex: you know when that when that like whole picture started
18:40
Alex: to form into place um it felt like okay this is going to be like a huge inventory
18:46
Alex: situation you kind of like nft collections were a huge inventory situation obviously
18:51
Alex: completely different really different market dynamics really different type
18:55
Alex: of of goal that buyers have.
18:59
Alex: And so a lot of like my early experimentation, like I made like a Chrome extension called Window AI.
19:06
Alex: I did like a few other things were just about learning how the ecosystem works
19:11
Alex: and like what makes it different and how the like, like what people really want,
19:16
Alex: what developers really want.
19:17
Josh: So that leads us to OpenRouter itself, right? So I kind of want you to help
19:21
Josh: explain to the listeners who aren't familiar with OpenRouter what it does.
19:24
Josh: Because I think a lot of people, the way they interact with an AI is they send
19:27
Josh: a prompt to their model of choice.
19:29
Josh: They use ChatGPT or they use the Grok app or they're on Gemini and they kind
19:33
Josh: of live in these siloed worlds.
19:34
Josh: And then the next step up from the people are those kind of who use it professionally,
19:38
Josh: who are developers. They're interacting with APIs.
19:40
Josh: Maybe they're not interfacing with the actual UI, but they're calling a single model.
19:44
Josh: And OpenRouter kind of exists on top of this, right? Can you walk us through
19:47
Josh: how it works and why so many people love using OpenRouter?
19:50
Alex: Open Router is an aggregator and marketplace for large language models.
19:56
Alex: You can kind of think of it as like a Stripe meets Cloudflare for both of them.
20:04
Alex: It's like a single pane of glass. You can orchestrate, discover,
20:09
Alex: and optimize all of your intelligence needs in one place.
20:14
Alex: One billing provider gets you all the models.
20:17
Alex: Uh there's like 470 plus now uh
20:21
Alex: like all the models like they sort of implement features
20:24
Alex: but they do it differently and they also there's
20:28
Alex: a lot of like intelligence brownouts as andre carpoffi calls them yeah where
20:32
Alex: models just go down all the time even the you know even the top models like
20:36
Alex: anthropic and gemini and and open
20:39
Alex: ai um so what we do is you know we like developers need a lot of choice.
20:48
Alex: CTOs need a lot of reliability.
20:50
Alex: CFOs need predictable costs. CISOs need complex policy controls.
20:55
Alex: All of these are inputs to what we do, which is build a single pane of glass
21:03
Alex: that makes models more reliable, lower costs, gives you more choice, and,
21:11
Alex: and then and helps you choose between all the options for where to source your intelligence.
21:17
Josh: How does it work uh because i would imagine like what
21:20
Josh: each as and i on the show we frequently talk about benchmarks right where
21:23
Josh: a certain model is the best at coding and that infers that maybe you should
21:27
Josh: go to that model to do all of your coding needs because it's the best at it
21:30
Josh: but it would appear as if it's not true if you're routing through a lot of different
21:34
Josh: providers so how do you consider which provider gets routed to when and how
21:39
Josh: to get the best result for what you're asking
21:41
Alex: So we've taken a different approach so
21:44
Alex: far which is instead of like focusing on
21:48
Alex: a production router that picks
21:50
Alex: the model for you um we try
21:53
Alex: to help you choose the model so we
21:57
Alex: we build lots we create lots of analytics both on
22:01
Alex: your account and uh and on our
22:03
Alex: rankings page to help you browse and discover the models that
22:06
Alex: like the power users are really using successfully on
22:10
Alex: a certain type of workload um because we
22:13
Alex: think like developers today primarily want to
22:16
Alex: choose the model themselves um switching between all
22:19
Alex: families can result in like a lot like very
22:22
Alex: unpredictable behavior but once you've
22:25
Alex: chosen your model um we try to
22:28
Alex: help developers not need to think about the provider there are
22:31
Alex: like sometimes dozens of
22:34
Alex: providers for a given model uh all kinds
22:38
Alex: of companies including the hyperscalers like aws google vertex and azure um
22:46
Alex: and uh like scaling startups like together fireworks deep infra um and a long
22:54
Alex: tail of providers that provide,
22:57
Alex: like very unique features,
23:00
Alex: very like exceptional performance.
23:03
Alex: There's all kinds of differentiators for them.
23:06
Alex: So what we do is we collect them all in one place. And if you want a feature,
23:10
Alex: you just get the providers that support it.
23:12
Alex: If you want performance, you get prioritized to the providers that have high performance.
23:18
Alex: If you really are cost sensitive, you get prioritized to the providers that
23:22
Alex: are really low cost today. and we basically create all these lanes. There's.
23:31
Alex: Innumerable ways you could get routed but
23:35
Alex: you're in full control of the of the overall user
23:38
Alex: experience that you're aiming for and that's
23:41
Alex: what that's what we found that was missing from the
23:44
Alex: whole ecosystem was just a way of doing that and uh
23:47
Alex: and you know we get like between on average five to ten percent uptime boosts
23:53
Alex: over going to um providers directly just by load balancing and sending you to
24:01
Alex: the top provider that's up and able to handle your request.
24:10
Alex: We really focus hard on efficiency and performance.
24:14
Alex: We only add about 20 to 25 milliseconds of latency on top of your request.
24:19
Alex: It all gets deployed very close to your servers up the edge.
24:25
Alex: We overall get just We stack providers.
24:29
Alex: We figure out what you can benefit from that everybody else is doing and just
24:37
Alex: give you the power of big data as a developer just accessing your model choice.
24:44
Josh: So it kind of allows you to harness the collective knowledge of everybody, right?
24:48
Josh: You get all of the data, you have all of the queries, you know which yields
24:51
Josh: the best result, and you're able to deliver the best product for them.
24:54
Josh: Now, in terms of actual LLMs, EJ has actually pulled this up just before, which is a leaderboard.
24:59
Josh: And I'm interested in how you guys think about LLMs, which are the best,
25:03
Josh: how to benchmark them, and how you route people through them.
25:06
Josh: Is there a specific... Do you believe that benchmarks are accurate,
25:10
Josh: and do you reflect those in the way that you route traffic through these models?
25:13
Alex: In general, we have taken the stance that we want to be the capitalist benchmark for models.
25:25
Alex: What is actually happening?
25:27
Alex: And part of this is that I really think both the law of large numbers and the
25:36
Alex: enthusiasm of power users are really, really valuable for everybody else.
25:42
Alex: Like when you're routing to
25:45
Alex: um like clod in
25:50
Alex: let's say you're routing to clod 4 and you're
25:52
Alex: based in europe um there you
25:56
Alex: know all of a sudden there might be like a huge variance in in throughput from
26:00
Alex: one of the providers and you're only able to detect that if like some other
26:05
Alex: users have discovered it before you and so we route around the provider that's
26:09
Alex: like running kind of slow in Europe and send you,
26:13
Alex: if your data policies allow it,
26:16
Alex: to a much faster provider somewhere else.
26:18
Alex: And that allows you to get faster performance. So, like, um...
26:23
Alex: That's, like, on the provider level, how, like, numbers help.
26:26
Alex: On the, like, model selection level, like, what you see on this rankings page
26:30
Alex: here, power users will, like, when we put up a model, like, we put up a new
26:36
Alex: model today from a new model lab called ZAI,
26:40
Alex: like, the power users instantly discover it.
26:43
Alex: We have this LLM enthusiast community that dives in and really figures out what
26:51
Alex: a model is good for along a bunch of core use cases.
26:54
Alex: The power users figure out which workloads are interesting, and then you can
26:59
Alex: just see in the data what they're doing. And everybody can benefit from it.
27:04
Alex: That's why we open up our data and share it for free on the rankings page here.
27:11
Ejaaz: I'm seeing this one consistent unit across all these rankings,
27:16
Ejaaz: Alex, which is tokens, right?
27:18
Ejaaz: And Josh and I have spoken about this on the show before, but I'm wondering
27:23
Ejaaz: how, like you've chosen this specific unit to measure how good or effective
27:28
Ejaaz: these models are or how consumed or used they are.
27:32
Ejaaz: Can you tell us a bit more as to why you picked this particular unit and what
27:35
Ejaaz: that tells you as like the open router platform as to how a user is using a particular model?
27:41
Alex: Yeah, I think dollars is a good metric too.
27:45
Alex: The reason we chose tokens is primarily because we were seeing prices come down really quickly.
27:57
Alex: Open Router has been around since the beginning of 2023.
28:02
Alex: And I didn't want a model to be penalized in the rankings just because the prices
28:11
Alex: are going down really dramatically now like there's a,
28:17
Alex: There's a paradox called Jevons paradox, which is that when prices decrease like 10x,
28:26
Alex: users' use of some component of infrastructure increases by more than 10x.
28:35
Alex: And so maybe they didn't get 10x at all.
28:39
Alex: But I thought there were some other advantages to using tokens,
28:42
Alex: too. Tokens don't have this penalty and don't rely on Jevon's Paradox,
28:47
Alex: which can have a lot of lag.
28:49
Alex: They also are a little bit of a proxy for time.
28:53
Alex: A model that is generating a lot of tokens and doing so for a while across a lot of users.
29:02
Alex: It means that a lot of people are reading those tokens and actually doing something with them.
29:06
Alex: And same goes for input. But if I really want to send an enormous number of
29:11
Alex: documents and the model has a really, really, really tiny prompt pricing,
29:16
Alex: I think that's still valuable and something that we want to see.
29:19
Alex: We want to see that this model is processing an enormous number of documents.
29:23
Alex: That's a use case that should show up in the rankings.
29:27
Alex: And so we decided to go with tokens. We might like add dollars in the future,
29:33
Alex: but I think tokens are, you know, they don't have this like Jevons Paradox lag.
29:42
Alex: And there wasn't anything else. Like nobody was doing any kind of like overall analytics.
29:48
Alex: We didn't see any other company even do it until Google did a few months ago
29:54
Alex: where they started publishing the total amount of tokens processed by Gemini.
30:00
Alex: So we'll see which use cases really need dollars.
30:06
Alex: But tokens have been holding up pretty well.
30:09
Ejaaz: Yeah, I mean, this dashboard is awesome. And I recommend anyone that's listening
30:14
Ejaaz: to this that can't see our screen to get on OpenRouter's website and check it out.
30:20
Ejaaz: I've been following it for the last two weeks kind of pretty rigorously, Alex.
30:25
Ejaaz: And what I love is you can literally see...
30:28
Ejaaz: So two weeks ago Grok 4 got released right
30:31
Ejaaz: and Josh and I were making a ton of videos on this we were
30:34
Ejaaz: using it with pretty much everything that we could do and
30:38
Ejaaz: then this other model came out of China pretty much a few days after called
30:42
Ejaaz: Kimi K2 and I was like oh yeah whatever this is just some random Chinese model
30:47
Ejaaz: I'm not going to focus on it and then I kept seeing it in my feed and I thought
30:51
Ejaaz: okay maybe I'll give this a go and I kind of like went straight to open rather than just
30:55
Ejaaz: almost gauge the interest from a wider set of AI users. And I saw that it was skyrocketing, right?
31:03
Ejaaz: And then I saw that Quen dropped their models last week.
31:07
Ejaaz: And again, I came to Open Router and it preceded the trend, right?
31:11
Ejaaz: People had already started using it. So I love how you describe Open Router
31:14
Ejaaz: as this kind of like prophetic orb,
31:18
Ejaaz: basically, where the enthusiasts and the community itself can kind of like front
31:22
Ejaaz: run very popular trends. And I think that's a very powerful moat.
31:26
Ejaaz: And kind of on this path, Alex, I noticed that a lot of these major model providers
31:33
Ejaaz: see the value in this, right?
31:35
Ejaaz: So if I'm not mistaken, OpenAI kind of like used your platform to kind of secretly
31:41
Ejaaz: launch their Frontier model before they officially launched it, right?
31:46
Ejaaz: Can you walk us through, you know, how that comes about and more importantly,
31:51
Ejaaz: why they want to do that and why they chose OpenRoddy to do that?
31:54
Alex: Uh open ai will sometimes
31:58
Alex: give uh early access
32:01
Alex: to their to models to some of their customers for
32:04
Alex: testing and we asked them if they
32:08
Alex: wanted to try a stealth model with us which we had never done before um it involved
32:14
Alex: like launching it as under another name and seeing how users respond to it without
32:21
Alex: having any bias or sort of inclination for against the model at the onset.
32:30
Alex: And it would be like a new way of testing it and a new way of...
32:35
Alex: It was like an experiment for both us and them.
32:38
Alex: And they generously decided to take the leap of faith and try it. And we...
32:48
Alex: Launched gpt 4.1 with
32:51
Alex: them at and we called it quasar alpha and
32:55
Alex: it was a million uh
32:59
Alex: token context length model opening us first very
33:02
Alex: very long context model and it was also optimized
33:07
Alex: for coding and the incredible
33:11
Alex: there were a couple incredible things that happened first
33:14
Alex: we have this community uh of benchmarkers
33:18
Alex: that run open source benchmarks and we give
33:20
Alex: a lot of them grants to help fund the benchmarks
33:23
Alex: grants of open router tokens they'll just run the
33:27
Alex: suite of tests against all the models and some of them are very creative like
33:31
Alex: there's one that tests uh like the ability to generate fiction there's one that
33:37
Alex: tests um like how like whether it can make a 3d object project in Minecraft called MCBench.
33:45
Alex: There are a few that test different types of coding proficiency.
33:50
Alex: There's one that just focuses on how good it is at Ruby, because Ruby is,
33:54
Alex: turns out a lot of the models are not great at Ruby.
33:56
Alex: There are a lot of like languages that all the models are pretty bad at.
34:00
Alex: And so we have this like long tail of very niche benchmarks,
34:04
Alex: And all the benchmarkers ran, you know, for free their benchmarks on Quasar
34:11
Alex: Alpha and found pretty incredible results for most of them.
34:16
Alex: And so the model got like, you know, OpenAI got this feedback in real time.
34:21
Alex: We kind of like helped them find it.
34:25
Alex: And they made another snapshot, which we launched as Optimus Alpha.
34:32
Alex: And they could compare the feedback that they got from the two snapshots.
34:36
Alex: Um, and, and then they, and then like two weeks later, they launched GPT 4.1 live for everybody.
34:43
Alex: So it was like, uh, uh, was it an experiment for us?
34:48
Alex: And, and we've done it, um, again since, uh, with, uh, another model provider
34:54
Alex: that, uh, that's still working on it.
34:58
Alex: Um, and it, and it's kind of like a cool way of learning of like crowdsourcing,
35:03
Alex: uh, benchmarks that you wouldn't have expected. and also getting unbiased community sentiment.
35:09
Josh: That's great. So now when we see a new model pop up and we want to test GPT-5,
35:13
Josh: we know where to come to to try it early.
35:16
Josh: We'll see because rumor is it's coming soon. So we'll be, we're on your watch list.
35:20
Josh: But having, I do want to ask you about open source versus closed source because
35:23
Josh: this has been an important thing for us. We talk about this a lot.
35:26
Josh: You have a ton of data on this.
35:28
Josh: I'm looking at the leaderboards there. There are open source models that are
35:30
Josh: doing very well, closed source.
35:32
Josh: What are your takes in general? How do you feel about open source versus closed
35:36
Josh: source models, particularly around how you serve them to users?
35:40
Alex: Both models, both types of models have supply problems, but the supply problems are very different.
35:47
Alex: Typically, what we see with closed source models is that there's there's very
35:50
Alex: few suppliers, usually just one or two.
35:54
Alex: Like with Grok, for example, there's Grok Direct and there's Azure.
35:58
Alex: Um with anthropic there's anthropic direct there's google vertex there's aws
36:04
Alex: bedrock um and then we also like deploy it in different regions like we have
36:08
Alex: an eu deployment um for customers who'd like only want their data like to stay in the eu,
36:15
Alex: and uh and we do custom deployments for
36:18
Alex: the for the closed source models too to just kind of guarantee good
36:21
Alex: throughput high and high rate limits for people um
36:25
Alex: but uh the
36:29
Alex: like a tricky part is that like the the demand usually the like the closed source
36:39
Alex: malls are doing most of the tokens on open router um it's it's dominant you
36:44
Alex: know it's probably 80-ish 70 to 80 percent closed source tokens today.
36:49
Alex: But the open source models have a much more fragmented supply, like cell supply.
36:58
Alex: Side order book um and and like
37:02
Alex: the rate limits for each provider is
37:05
Alex: like a like less stable on average um it
37:09
Alex: usually takes a while for the hyperscalers to serve a
37:12
Alex: new closed source a new open source model um so we so the load balancing work
37:18
Alex: that we do on um open source models tends to be a lot more valuable the load
37:24
Alex: balancing work that we do for closed source models tends to be very focused
37:28
Alex: on caching and feature awareness,
37:31
Alex: making sure you're getting clean cache hits and only transitioning over to new
37:36
Alex: providers when your cache is expired.
37:39
Alex: For open source models, there's way less caching. Very, very few open source
37:45
Alex: models implement caching.
37:48
Alex: And so switching between providers becomes more common. and
37:52
Alex: uh like we we also track a
37:55
Alex: lot of quality differences between the the open
37:58
Alex: source providers some of them will deploy at lower
38:00
Alex: quantization levels which means like it's kind of like a way of compressing
38:04
Alex: the model um generally doesn't have an impact on the quality of the output uh
38:11
Alex: but and yet we still see some odd things from some of the open source providers.
38:18
Alex: And so we run tests internally to detect those outputs. And we're building up
38:25
Alex: a lot more muscle here soon.
38:27
Alex: So that like, they get pulled out of the routing lane and don't affect anyone.
38:33
Josh: So closed source accounts for 80% or something like that, a very large amount.
38:36
Josh: Do you see that changing?
38:38
Josh: Because that post we just had, it's at nine out of the 10 fastest growing LLMs
38:41
Josh: last week, they were open source.
38:43
Josh: And every time it seems like China comes out with another model,
38:47
Josh: it was Kimmy K2 a week or two ago, it kind of really pushes the frontier of open source forward.
38:54
Josh: And the rate of acceleration of open source seems to be as fast,
38:58
Josh: if not faster than closed source, where it's just, it's making these improvements very quickly.
39:02
Josh: It has the benefit of being able to compound in speed because it's open source
39:06
Josh: and everyone can contribute.
39:08
Josh: Do you think that starts to change where the percentage of tokens you're issuing
39:11
Josh: are from open source models versus closed source?
39:14
Josh: Or do you continue to see a trend where it's going to be Google,
39:17
Josh: it's going to be OpenAI that are serving a majority of these tokens to users?
39:20
Alex: In the short term, we're likely to see open source models continue to dominate
39:25
Alex: the fastest growing model category on OpenRouter.
39:29
Alex: And the reason for that is that a lot of users who come for a closed source
39:36
Alex: model, but then decide they want to optimize later,
39:41
Alex: either they want to save on costs or try out a new model that's supposed to
39:49
Alex: be a little bit better in some direction that their app cares about or their use case cares about,
39:55
Alex: then they leave the closed source model and go to an open source model.
39:58
Alex: So open source tends to be like a last mile optimization thing,
40:03
Alex: making a big generalization because the reverse can happen too.
40:08
Alex: And so because it's a last mile optimization thing,
40:12
Alex: the jump from this model is not being used at all to this model is really being
40:17
Alex: used by a couple of people who have
40:19
Alex: left Claude 4 and want to try some new coding use case will be bigger.
40:29
Alex: Than the closed-source models, which start at a really high base and don't have
40:34
Alex: growth quite as dramatic.
40:36
Alex: So the other part of your question, though, was whether there's going to be like a flippening of.
40:43
Josh: Close or some sort of like chipping it away at that monopoly of close source tokens.
40:49
Alex: It's hard to predict these things because, you know,
40:52
Alex: I think like the the biggest problem today with open source models is that the
40:58
Alex: incentives are not as strong like the model lab and the model provider.
41:05
Alex: Um they've you know they're they're
41:08
Alex: sort of established incentives for how to
41:10
Alex: grow as a company and attract good high quality um ai talent and um and giving
41:19
Alex: the model weights away impairs those incentives now like we might see yeah this
41:27
Alex: is where we might see like decentralized providers,
41:30
Alex: helping in the future.
41:33
Alex: A way for like,
41:36
Alex: uh you know like a really good incentive scheme that
41:40
Alex: like allows high quality talent
41:43
Alex: to work on an open source model um
41:46
Alex: that remains open weights at least uh like could fix this i like i you know
41:54
Alex: i stay pretty i try to stay close to the decentralized providers um and like
41:58
Alex: learn a lot from them there's some like cool on the provider side on like on
42:02
Alex: running inference i I think there's some really cool incentive schemes being worked on.
42:07
Alex: But on actually developing the models themselves, I haven't seen too much, unfortunately.
42:12
Alex: So I think if we see one, flipping in the radar. And until we do, I personally doubt it.
42:20
Josh: TBD, do you have personal takes on how you feel about open source versus closed source?
42:24
Josh: Because this has been a huge topic we've been debating too. It's just the ethical
42:28
Josh: concerns around alignment and closed source models versus open source.
42:32
Josh: When you look at the competitors, China, generally speaking,
42:35
Josh: is associated with open source, whereas the United States is generally associated with closed source.
42:39
Josh: And we saw Llama and Meta release the open source models, but now they're raising
42:45
Josh: a ton of money to pay a lot of employees a lot of money to probably develop a closed source model.
42:49
Josh: So it seems like the trends are kind of split between US and China.
42:53
Josh: And I'm curious if you have any personal takes, even outside of OpenRouter,
42:55
Josh: of which you think serves better for the long term outlook on,
43:00
Josh: I mean, the position of the United States or just the general safety and alignment
43:05
Josh: conversation around AI?
43:06
Alex: I mean, like a very simple fundamental difference between the two is that an
43:12
Alex: innovation in open source models can be copied more quickly than an innovation
43:17
Alex: in closed source models.
43:20
Alex: So in terms of velocity and like how far ahead one is over the other,
43:26
Alex: that is like a massive structural difference.
43:29
Alex: That means that closed source models should be theoretically always ahead until
43:37
Alex: a really interesting incentive scheme develops, like I mentioned before.
43:41
Alex: Uh, I think, and I think that's, you know, I don't see like evidence that that's
43:47
Alex: going to change in terms of China versus the U S.
43:52
Alex: Um, it's, I think it's very interesting that China has not had like a major closed source model.
44:01
Alex: Um and i don't really
44:04
Alex: see a great reason why i'm
44:08
Alex: not aware of any reasons that's not that's not going
44:11
Alex: to be going to be the case in the future um my prediction
44:14
Alex: is that there's going to be a closed source model from china um
44:18
Alex: and uh you know if uh uh you know if like it's possible that DeepSeas and Moonshot
44:29
Alex: and Gwen have built up really sticky talent pools.
44:36
Alex: But generally with talent pools, after enough years have passed,
44:41
Alex: people quit and go and create new companies and build new talent pools.
44:49
Alex: And so we should see some of that. It's not the case that the AI space has NDAs
44:54
Alex: or non-competes that the hedge fund space has.
45:00
Alex: That might happen in the future too. But assuming that the current non-compete
45:04
Alex: culture continues, there should be more companies that pop up in China over time.
45:11
Alex: And I'm betting that some of them will be closed source.
45:13
Alex: And my guess is that the two nations will start to look more similar.
45:18
Ejaaz: Yeah, I guess that's why you have Zuck dishing out 300 mil to a billion dollar
45:24
Ejaaz: salary offers to a bunch of these guys, right?
45:27
Ejaaz: One more question on China versus the US. I kind of agree with you.
45:31
Ejaaz: I didn't really expect China to be the one to lead open source anything,
45:37
Ejaaz: let alone the most important technology of our time.
45:41
Ejaaz: Do you think is their secret source to building these models, Alex?
45:44
Ejaaz: And I know this might be out of the forte of
45:48
Ejaaz: open router specifically but as someone who has studied this technology for
45:52
Ejaaz: a while now i'm struggling to figure out you know what advantage they had you
45:59
Ejaaz: know they're discovering all these new techniques and maybe the simple answer
46:03
Ejaaz: is like constraints right they don't have access to all of
46:06
Ejaaz: nvidia's chips they don't have access to infinite compute so then maybe they're
46:11
Ejaaz: forced to kind of like figure out other ways around the same kinds of problems
46:14
Ejaaz: that western companies are focused on But it's pretty clear that America, with all its funding,
46:20
Ejaaz: hasn't been able to make these frontier breakthroughs.
46:23
Ejaaz: So I'm curious whether you are aware of or know some kind of technical moat
46:29
Ejaaz: that Chinese AI researchers or these AI teams that are featuring on Open Rata
46:33
Ejaaz: day in and day out have over the U.S.?
46:41
Alex: Well, I don't know.
46:46
Alex: There are certainly some that they've come up with that like DeepSeek had a
46:52
Alex: lot of very cool inference innovations that they published in their paper.
46:58
Alex: But a lot of what they published in the original R1 paper were things that OpenAI
47:04
Alex: had done independently themselves many months before.
47:07
Alex: So uh i like
47:11
Alex: on the inference side and on
47:14
Alex: uh some of the model side i think like deep seek we we
47:18
Alex: had talked to their team for years before r1 came
47:21
Alex: out they had many models before that and
47:24
Alex: they were always like a pretty sharp optimum like
47:27
Alex: team for doing inference um like they
47:30
Alex: came up with like the best user experience for caching prompts
47:33
Alex: long before deep cpr1 came out and they had very good pricing um they uh they
47:41
Alex: were just they were like you know by far the the strongest chinese team um that
47:47
Alex: we were aware of uh well before that happened and so i'm guessing there was like some talent.
47:53
Alex: Uh accumulation that they were working on in china
47:56
Alex: for people who wanted to stay in china and yeah that's
48:00
Alex: that's a huge advantage like american companies are obviously not
48:02
Alex: doing that there's a duck is very on
48:06
Alex: point that a lot of this is just based on talent
48:09
Alex: um there are a lot of
48:12
Alex: ai is open and out there and just like and
48:16
Alex: very composable like a big tree of knowledge
48:19
Alex: there's a paper that comes out and it cites like
48:22
Alex: 20 other papers and you can go and read all
48:25
Alex: of the cited papers and then you like have kind of
48:28
Alex: a basis for understanding the paper but you really have to
48:30
Alex: go one level deeper and read all the cited papers two levels
48:33
Alex: down to really understand what's going on and it's.
48:37
Alex: Just that no very few people can do that um and
48:40
Alex: it takes like a lot of years of experience to like actually
48:44
Alex: apply that knowledge and learn all these
48:46
Alex: things that have not been written in any paper at all and uh
48:50
Alex: and there's just there's just such such it
48:53
Alex: like a small number of people um who can
48:56
Alex: really lead research on all the different dimensions that
48:59
Alex: go on to making a model and uh um and
49:03
Alex: and like the the border between china and the u.s is
49:06
Alex: is pretty defined you have to leave china move to the u.s
49:09
Alex: and really establish yourself here um so
49:13
Alex: i do think there's like country arbitrage there's like
49:16
Alex: there's you know the head the hedge fund background arbitrage there's uh there's
49:22
Alex: there's hardware arbitrage like there's like a ton of hardware that's only available
49:25
Alex: in china but not here vice versa that creates an opportunity um and this this
49:32
Alex: will just continue to happen.
49:33
Ejaaz: Yeah, I think this arbitrage is fascinating.
49:37
Ejaaz: I read somewhere that there's probably less than 200 or 250 researchers in the
49:44
Ejaaz: world that are worthy of working at some of these frontier AI model labs.
49:49
Ejaaz: And I looked into some of the backgrounds of the team behind Kimi K2,
49:54
Ejaaz: which is this recent open source model out of China, which broke all these crazy rankings.
50:01
Ejaaz: I think it was like a trillion parameter model or something crazy like that.
50:05
Ejaaz: And a lot of them worked at some of the top American tech companies.
50:08
Ejaaz: And they all graduated from this one university in China.
50:11
Ejaaz: I think it's Tsinghua, which apparently is like, you know, the Harvard of AI
50:16
Ejaaz: in China, right? So pretty crazy.
50:19
Ejaaz: But Alex, I wanted to shift the focus of the conversation to a point that you
50:25
Ejaaz: brought up earlier in this episode, which is around data.
50:29
Ejaaz: Okay, so here's the context that like Josh and I have spoken about this at length, right?
50:35
Ejaaz: We are obsessed with this feature on OpenAI, which is memory, right?
50:39
Ejaaz: And I know a lot of the other memory, sorry, a lot of the other AI models have memory as well.
50:44
Ejaaz: But the reason why we love it so much is I feel like the model knows me, Alex.
50:49
Ejaaz: I feel like it knows everything about me. It can personally curate any of my prompt.
50:54
Ejaaz: It just gets me. It knows what I want and it just serves up to me in a platter
50:59
Ejaaz: and off I go, you know, doing my thing.
51:02
Ejaaz: Now, Open Router sits on top of like kind of like the query layer, right?
51:07
Ejaaz: So you have all these people writing all these weird and wonderful prompts and
51:12
Ejaaz: kind of routing it through on towards like different AI models.
51:18
Ejaaz: You hold all of that data or maybe you have access to all of that data.
51:21
Ejaaz: And I know you have something called private chat as well, where you don't have access to it.
51:25
Ejaaz: Talk to me about like what OpenRouter and what you guys are thinking about doing
51:29
Ejaaz: with this data, because presumably,
51:31
Ejaaz: or in my opinion, you guys have actually the best mode, arguably better than
51:36
Ejaaz: ChatGPT, because you have all these different types of prompts coming from all
51:40
Ejaaz: these different types of users for all these different types of models.
51:43
Ejaaz: So theoretically, you could spin up some of the most personal AI models for
51:47
Ejaaz: each individual user if you wanted to.
51:49
Ejaaz: Do I have that correct? Or am I, you know, speaking crazy?
51:54
Alex: No, that's true. No, it's something we're thinking about.
52:00
Alex: By default, your prompts are not logged at all.
52:05
Alex: We don't have prompts or completions for new users by default.
52:09
Alex: You have to toggle it on in settings.
52:15
Alex: But the result, a lot of people do toggle it on. And as a result,
52:21
Alex: I think we have by far the largest multi-model prompt data set.
52:26
Alex: Uh, but what we've done today, we've barely done anything with it.
52:32
Alex: We classify a tiny, tiny, tiny subset of it. And that's what you see in the rankings page.
52:37
Alex: Um, but, uh, what it could be done on like a per account level is really,
52:43
Alex: um, like three main things.
52:45
Alex: One memory right out of the box. You can, you can get this today by like combining
52:51
Alex: open router with like a memory as a service. We've got a couple of companies
52:56
Alex: that do this, like Memzero and SuperMemory.
52:59
Alex: And we can partner with one of those companies or do something similar and just
53:03
Alex: provide a lot of distribution.
53:05
Alex: And that basically gets you a chat GPT as a service where it feels like the
53:09
Alex: model really knows you and the right context gets added to your prompt.
53:16
Alex: The other things that we can do are help you select the right model more intelligently.
53:25
Alex: There's a lot of models where there's like a super clear, like migration decision that needs to be made.
53:33
Alex: And, and we can just see this very clearly in the data.
53:36
Alex: But we right now we just like, you know, we have like a channel or like some
53:41
Alex: kind of communication channel open with the customer, we can just tell them
53:44
Alex: like, hey, and we know you're using this model a ton.
53:48
Alex: It's been deprecated. This model is significantly better. you
53:51
Alex: should move this kind of workload over to it or like
53:54
Alex: this workload you'll get way better pricing if you do this um
53:58
Alex: and and that's basically like that's the
54:01
Alex: only sort of guidance and kind of like
54:04
Alex: opinionated routing we've done so far and it could
54:06
Alex: be a lot more intelligent a lot more out of the box a lot more
54:09
Alex: built into the product um and then
54:13
Alex: the the last thing
54:16
Alex: we can do i mean there's there's probably tons of
54:19
Alex: things we're not even thinking about um but
54:22
Alex: like getting really
54:26
Alex: really smart about how
54:30
Alex: models and providers are responding to prompts and
54:33
Alex: uh showing you just the really coolest
54:37
Alex: data just like telling you
54:40
Alex: what kinds of of prompts um are
54:44
Alex: going to which models and how those models are replying and
54:48
Alex: just like characterizing the reply in all kinds of interesting ways
54:51
Alex: like did the model refuse to answer what's the refusal rate
54:54
Alex: did the model um did the.
54:57
Alex: Model like successfully make a tool call or did it decide to
55:00
Alex: ignore all the tools that you passed in that's a huge one
55:02
Alex: um did the model like pay
55:06
Alex: attention to its context did uh you know did what did did some kind of truncation
55:12
Alex: happening happen before you sent it to the model So there's all kinds of like
55:16
Alex: edge cases that cause developers apps to just get dumber and they're all detectable.
55:25
Ejaaz: I'm so happy you said that because I have this kind of like hot take,
55:31
Ejaaz: but maybe not so hot take, which is I actually think all the Frontier models
55:35
Ejaaz: right now are good enough to do the craziest stuff ever for each user.
55:40
Ejaaz: But we just haven't been able to unlock it because it just doesn't have the context.
55:44
Ejaaz: Sure, you can attach it to a bunch of different tools and stuff,
55:48
Ejaaz: but if it doesn't know when to use the tool or how to process a certain prompt
55:53
Ejaaz: or if the users themselves don't know how to read
55:57
Ejaaz: the output of the AI model themselves, like you just said, we need some kind
56:01
Ejaaz: of analytics into all of this,
56:03
Ejaaz: then we're just kind of walking around like headless chickens almost.
56:07
Ejaaz: So I'm really happy that you said that. One other thing that I wanted to get
56:11
Ejaaz: your take on on the data side of things is, I just think this whole concept
56:15
Ejaaz: or notion of AI agents is becoming such a big trend, Alex.
56:21
Ejaaz: And I noticed a lot of Frontier Model Labs release new models that kind of spin
56:28
Ejaaz: up several instances of their AI model.
56:30
Ejaaz: And they're tasked with a specific role, right?
56:33
Ejaaz: Okay, you're going to do the research. You're going to do the orchestrating.
56:37
Ejaaz: You're going to look online via a browser, blah, blah, blah,
56:40
Ejaaz: blah, blah. And then they coalesce together at the end of that little search
56:44
Ejaaz: and refine their answer and then present it to someone, right?
56:48
Ejaaz: You know, Grok4 does this, Claude does this, and a few other models.
56:52
Ejaaz: I feel like with this data that you're describing, OpenRouter could be or could
56:58
Ejaaz: offer that as a feature, right?
57:00
Ejaaz: Which is essentially, you can now have super intuitive, context-rich agents
57:05
Ejaaz: that can do a lot more than just talk to you or answer your prompts.
57:08
Ejaaz: But they could probably do a bunch of other actions for you.
57:11
Ejaaz: Is that a fair take, or is that something that maybe might be out of the realm of open router?
57:18
Alex: Our strategy is to be the best inference layer for agents.
57:23
Alex: And what I think developers want...
57:30
Alex: Is control over how their agents work.
57:35
Alex: And our developers at least want to use us as a single pane of glass for doing
57:42
Alex: inference, but they want to see and control the way an agent looks.
57:48
Alex: An agent is basically just something
57:52
Alex: that is doing inference in a loop and controlling the direction it goes.
57:57
Alex: So um what what
58:01
Alex: we want to do is just like build incredible docs
58:04
Alex: really good primitives that make that easy
58:08
Alex: to do so that you know like i think like
58:11
Alex: a lot of our developers are just people building agents and so
58:14
Alex: what they want is they want the primitives to
58:17
Alex: be solved so that they can just keep creating new
58:20
Alex: versions and new ideas um without worrying
58:24
Alex: about like you know re-implementing tool calling over
58:27
Alex: and over again and um and and
58:31
Alex: and so like at least for this is like a it's it's
58:34
Alex: a tough problem given how many models there's like a new model or provider every
58:37
Alex: day and uh and people actually want them and use them so uh to standardize this
58:44
Alex: like make make these tools like really dependable um that's kind of like where
58:49
Alex: we want to focus and uh so that like agent developers don't have to worry about it.
58:54
Josh: As we level up towards closer and closer to getting to AGI beyond,
58:58
Josh: I'm curious what Open Router's kind of endgame is.
59:00
Josh: If you have one, what is the master plan where you hope to end up?
59:03
Josh: Because the assumption is as these systems get more intelligent,
59:06
Josh: as they're able to kind of make their own decisions and choose their own tool
59:09
Josh: sets, what role does Open Router play in continuing to route that data through?
59:15
Josh: Do you have a kind of master plan, a grand vision of where you see this all heading to?
59:19
Alex: You're saying like as agents get better at choosing the tools that they use
59:24
Alex: what what becomes our role when like the agents are really good at that yes.
59:31
Josh: Yes and like where do you see open router fitting into the picture and what
59:34
Josh: would be the best case scenario for this this future of open router
59:38
Alex: Right now open routers bring your own tool,
59:42
Alex: platform um we don't have like a
59:45
Alex: marketplace of mcps yet uh and
59:49
Alex: and i i do think like a lot of the i think most of the most used tools will
59:55
Alex: be ones that developers configure themselves agents just work like they're given
1:00:00
Alex: access to it like i think like a holy grail for for open router is that.
1:00:07
Alex: The the ecosystem is going to like basically my
1:00:12
Alex: prediction for how the ecosystem is going to evolve is that um
1:00:16
Alex: all the models are going to be adding state and
1:00:19
Alex: other kinds of stickiness that just make you want to stick
1:00:22
Alex: with them so they're going to add server-side tool calls
1:00:25
Alex: they're going to add like um you know web search that that is stateful they're
1:00:31
Alex: going to add memory They're going to add all kinds of things that try to prevent
1:00:35
Alex: developers from leaving and increase lock-in.
1:00:42
Alex: And OpenRouter is doing the opposite.
1:00:45
Alex: We want developers to not feel vendor lock-in.
1:00:49
Alex: We want them to feel like they have choice and they can use the best intelligence,
1:00:53
Alex: even if they didn't before.
1:00:56
Alex: It's never too late to switch to a more intelligent model. That would be like,
1:01:00
Alex: you know, a good always on outcome for us.
1:01:05
Alex: And so what I think we'll end up doing is, is like partnering with other companies
1:01:13
Alex: or building the tools ourselves if we have to, so that developers don't feel stuck.
1:01:20
Alex: That's how I, you know, there's a lot of ways the ecosystem could evolve,
1:01:23
Alex: but that's how I would put it in a nutshell.
1:01:25
Josh: Okay, now there's another personal question that I was really curious about,
1:01:29
Josh: because I was also right there with you in the crypto cycle when NFTs got absolutely
1:01:34
Josh: huge, was a big user of OpenSea.
1:01:36
Josh: And it was kind of this trend that went up and then went down.
1:01:39
Josh: And NFTs kind of fizzled out, it wasn't as hot anymore, and AI kind of took the wind from the sails.
1:01:44
Josh: And it's a completely separate audience, but a similar thing where now it's
1:01:48
Josh: the hottest thing in the world.
1:01:49
Josh: And i'm curious how you see the trend continuing is this a cyclical thing that
1:01:54
Josh: has ups and downs or is this a one-way trajectory of more tokens every day more
1:01:59
Josh: ai every day is do you see it being a cyclical thing or is this a a one-way
1:02:03
Josh: trend towards up into the right nfts
1:02:06
Alex: Kind of follow uh crypto in a,
1:02:12
Alex: indirect way um when crypto
1:02:15
Alex: has ups and downs nfts generally lag a bit
1:02:18
Alex: but they they have similar ups and downs and um
1:02:22
Alex: and crypto is an extremely long-term play on like building a new financial system
1:02:30
Alex: and there are so many reasons that it's not going to happen overnight um and And they're like,
1:02:40
Alex: it's very, very entrenched reasons.
1:02:43
Alex: Whereas AI, there are some overnight business transformations going on.
1:02:49
Alex: And the reason AI, I think, moves a lot, one of the reasons that AI moves a
1:02:55
Alex: lot faster is it's just about making computers behave more like humans.
1:02:59
Alex: So if a company already works with a bunch of humans, then there's,
1:03:05
Alex: you know, there's some engineering that needs to be done.
1:03:07
Alex: There's some like thinking about how
1:03:10
Alex: to like scale this but
1:03:14
Alex: but in general i think that it's not like
1:03:17
Alex: after seeing what can be possible um inference
1:03:21
Alex: will be the fastest growing operating expense for all companies
1:03:24
Alex: it'll it'll be like oh we can just hire
1:03:27
Alex: high-performing employees at a click of a
1:03:37
Alex: and they they work 24 7 they
1:03:41
Alex: scale elastically it's like you know
1:03:44
Alex: it it's not that hard it's not like huge mental
1:03:47
Alex: model shift it's just like a huge upgrade to the way companies work today um
1:03:51
Alex: in most cases so it's just completely different from crypto there's there's
1:03:55
Alex: like other than both being you know than nfts i mean other than both being new
1:04:00
Alex: they're fundamentally very different changes.
1:04:04
Ejaaz: You're probably one of very few people in the world right now that has crazy
1:04:11
Ejaaz: insights to every single AI model.
1:04:13
Ejaaz: Definitely more than the average user, right? Like I have like three or four
1:04:17
Ejaaz: subscriptions right now and I think I'm a hotshot.
1:04:20
Ejaaz: You get access to like 400 and what is it? 57 models right now on OpenRata.
1:04:25
Ejaaz: So an obvious question that I have for you is
1:04:29
Ejaaz: I'm not going to say in the next couple of years, because everything moves way
1:04:33
Ejaaz: too quickly in this sector.
1:04:34
Ejaaz: But over the next six months, is there anything really obvious to you that should
1:04:41
Ejaaz: be focused on within the AI sector?
1:04:44
Ejaaz: Maybe it's like the way that certain models should be designed,
1:04:47
Ejaaz: or perhaps it's at the application layer that no one's talking about right now.
1:04:52
Ejaaz: Because going on from our earlier part of the conversation, you just pick these
1:04:56
Ejaaz: trends out really early. and I'm wondering if you see anything.
1:04:59
Ejaaz: It doesn't have to be open-racket related. It could just be AI related.
1:05:03
Alex: I've seen the models trending towards caring more about how resourceful they
1:05:11
Alex: are than what knowledge they have in the bank.
1:05:14
Alex: Not all of, I feel like a lot of the applications, I think the model labs maybe,
1:05:21
Alex: a lot of them, I don't know how many of them really deeply believe that,
1:05:24
Alex: but a couple of them uh talk about it and i don't think it's really hit the
1:05:31
Alex: application space yet um because people will will ask chat gpt things and if
1:05:37
Alex: the knowledge is wrong they think the model is stupid,
1:05:40
Alex: and that's just kind of a bad way of evaluating a model um
1:05:44
Alex: like whatever knowledge a person has whatever
1:05:48
Alex: a person like where calls happen at a certain time like
1:05:51
Alex: does not it's not a proxy for how smart they are um
1:05:55
Alex: like the the intelligence and usefulness of a model
1:05:57
Alex: is going to trend towards how good it is at using tools and
1:06:01
Alex: uh and and how good it is at like paying
1:06:05
Alex: attention to its context of a long long long long context and so it's like it's
1:06:11
Alex: it's total memory capacity and accuracy um so i think those two things need
1:06:17
Alex: to be like emphasized more um the.
1:06:23
Alex: Like it might be that that models pull all
1:06:26
Alex: of their knowledge from like online databases
1:06:30
Alex: from like real-time uh scraped
1:06:34
Alex: index indices of the web along with a
1:06:36
Alex: ton of real-time updating data sources um and
1:06:40
Alex: they're never they're always kind of like relying on some some sort of database
1:06:44
Alex: for knowledge but relying on their reasoning process for for tool calling you
1:06:49
Alex: know like we we put it We spend probably the plurality of our time every week
1:06:56
Alex: on tool calling and figuring out how to make it work really well.
1:06:59
Alex: Humans, the big difference between us and animals is that we're tool users and tool builders.
1:07:06
Alex: And that's where human acceleration and innovation has happened.
1:07:11
Alex: So how do we get models creating tools and using tools very,
1:07:19
Alex: very effectively? there's very little,
1:07:22
Alex: There are very few benchmarks. There's very little priority.
1:07:25
Alex: There's the Tau Bench for measuring how good a model is at tool calling.
1:07:29
Alex: But there's, and there's like maybe a few others.
1:07:32
Alex: There's Swee Bench for measuring how good a model is at multi-turn programming tasks.
1:07:39
Alex: It's very, very hard to run, though. It costs like, you know,
1:07:43
Alex: for Sonnet, it could cost like $1,000 to run it.
1:07:48
Alex: And it's like the user experience for kind of like evaluating the real intelligence
1:07:54
Alex: of these models is not good.
1:07:56
Alex: And so like I love, as much as we don't have benchmarks listed on OpenRouter
1:08:00
Alex: today, I love benchmarks.
1:08:02
Alex: And I think like the app ecosystem and like developer ecosystem should spend
1:08:07
Alex: a lot more time making very cool and interesting ones.
1:08:10
Alex: Also, we will give credit grants for all the best ones. So I highly encourage it.
1:08:17
Ejaaz: Well, Alex, thank you for your time today. I think we're coming up on a close
1:08:22
Ejaaz: now. That was a fascinating conversation, man.
1:08:25
Ejaaz: And I think your entire journey from just non-AI stuff, so OpenSea all the way
1:08:31
Ejaaz: to OpenRouter has just been a great indicator of where these technologies are
1:08:37
Ejaaz: progressing and more importantly, where we're going to end up.
1:08:39
Ejaaz: I'm incredibly excited to see where OpenRatter goes beyond just prompt routing.
1:08:45
Ejaaz: I think some of the stuff you spoke about on the data side of things is going
1:08:49
Ejaaz: to be fascinating and arguably one of your bigger features. So I'm excited for future releases.
1:08:53
Ejaaz: And as Josh said earlier, if GPT-5 is releasing through your platform first,
1:08:59
Ejaaz: please give us some credits. We would love to use it.
1:09:03
Ejaaz: But for the listeners of this show, as you know, we're trying to bring on the
1:09:07
Ejaaz: most interesting people to chat about AI and Frontier Tech. We hope you enjoyed this episode.
1:09:14
Ejaaz: And as always, please like, subscribe, and share it with any of your friends
1:09:18
Ejaaz: who would find this interesting. And we'll see you on the next one. Thanks, folks.