Latest / Merryn Talks Money / Why You Should Wait Out AI’s Super-Spending False Start

Transcript

Why You Should Wait Out AI’s Super-Spending False Start · Merryn Talks Money

• 0:02
  Speaker 1: Bloomberg Audio Studios, Podcasts, Radio News. Welcome to Marin Talks Money,
• 0:19
  Speaker 1: the podcast in which people who know the markets explain
• 0:22
  Speaker 1: the markets. I am Maren Summerzet Web and this week
• 0:25
  Speaker 1: I am speaking with doctor Yanushmeretzky, who is an AI
• 0:28
  Speaker 1: partner at Aaron Innovation Capital. Now, as you know, on
• 0:32
  Speaker 1: this podcast, we like to talk about the big forces
• 0:34
  Speaker 1: affecting our economy and markets in general, so it's really
• 0:37
  Speaker 1: no surprise that we keep coming back to the impact
• 0:40
  Speaker 1: of AI. We've talked at different times about the consequences
• 0:44
  Speaker 1: for jobs, for inflation, for interest rates, for tech companies,
• 0:46
  Speaker 1: and whether politicians and indeed policymakers, let alone ordinary workers
• 0:50
  Speaker 1: and investors, are ready for any of this. But what
• 0:53
  Speaker 1: we've never asked is is it actually working? Yanish Welcome
• 0:59
  Speaker 1: to Marri Talks Money.
• 1:01
  Speaker 2: Thank you so much for having me.
• 1:03
  Speaker 1: And it's coming to start with a brief explanation of
• 1:06
  Speaker 1: exactly what it is that we mean when we say AI.
• 1:10
  Speaker 2: Yes, everyone talks.
• 1:11
  Speaker 1: About AI all the time. Aire say that is going
• 1:13
  Speaker 1: to change the world, it's going to solve all our problems,
• 1:15
  Speaker 1: it's going to destroy our jobs, etc. But what do
• 1:18
  Speaker 1: we actually mean when we say AI.
• 1:21
  Speaker 2: So these days, what we mean by saying AI is
• 1:24
  Speaker 2: a system which is approximating certain process. It might be
• 1:30
  Speaker 2: a system which is approximating language. It might be a
• 1:33
  Speaker 2: system which is approximating images. It might be a system
• 1:37
  Speaker 2: which is approximating how a robot moves. By the end
• 1:40
  Speaker 2: of the day, the current generation of AI techniques, those
• 1:44
  Speaker 2: neural networks are the function approximators. This is approximate things.
• 1:49
  Speaker 2: They don't solve intelligence, they approximate. So that's why you
• 1:53
  Speaker 2: may have an illusion that those systems are intelligent. At
• 1:56
  Speaker 2: the end of the day, they are approximating intelligence.
• 1:59
  Speaker 1: And I suppose there's two ways to look at this.
• 2:00
  Speaker 1: What AI as you've just described needs from us and
• 2:04
  Speaker 1: what we really need from it to make it work
• 2:06
  Speaker 1: for us. So if you look at the way that
• 2:09
  Speaker 1: the big hyperscalas are approaching things at the moment, they're
• 2:11
  Speaker 1: building massive data centers to build out their capacity, and
• 2:15
  Speaker 1: that requires vast amounts of energy, It requires lots of coolants,
• 2:19
  Speaker 1: it requires a very large volume of very different types
• 2:21
  Speaker 1: of chips, right, yes, and all those things. Obviously there
• 2:25
  Speaker 1: are troubles at the moment. We're getting all these things
• 2:27
  Speaker 1: with the war in the Middle East, etc. So there
• 2:29
  Speaker 1: are supply restrictions, but nonetheless none of these things are
• 2:32
  Speaker 1: really our long term problem. Given their correct policy choices,
• 2:35
  Speaker 1: all those material elements can easily be not easily, but
• 2:40
  Speaker 1: can be found and built in. Then there's the second
• 2:43
  Speaker 1: bit that we've talked about when we met we last met,
• 2:47
  Speaker 1: which is the data that you require to train a model,
• 2:51
  Speaker 1: and that can't be created in the volumes that are required.
• 2:56
  Speaker 1: And we've hit it. We've hit a supply problem with data.
• 3:00
  Speaker 2: We have hit a supply problem with diverse data. To
• 3:04
  Speaker 2: clarify the thing, because you can just create an infinite
• 3:07
  Speaker 2: amount of data by randomly generating new words, so you
• 3:10
  Speaker 2: can create it. But we're talking about creating high quality,
• 3:13
  Speaker 2: diverse data. And you see we've run out of data,
• 3:17
  Speaker 2: diverse data, not yesterday, not a month ago. We have
• 3:21
  Speaker 2: run out of data three and a half years ago.
• 3:23
  Speaker 2: You can to understand that the last frontier model GBT four,
• 3:28
  Speaker 2: which was not a combination of agents, etc. Just basic LLM.
• 3:34
  Speaker 2: The last LLM that was the most potent was GBT four.
• 3:39
  Speaker 2: It was released when it was released in twenty twenty
• 3:42
  Speaker 2: three in January, However, the training of that model finished
• 3:46
  Speaker 2: at the end of twenty twenty two, so it's three
• 3:48
  Speaker 2: and a half years ago. We have trained the model
• 3:52
  Speaker 2: that used all the publicly available data on the Internet.
• 3:56
  Speaker 2: There is nothing more out there to use to train
• 3:58
  Speaker 2: the model. Hit the data ceiling not a month ago,
• 4:02
  Speaker 2: but three and a half years ago. It's extremely important.
• 4:06
  Speaker 2: And yes, what we're doing right now is we're trying
• 4:09
  Speaker 2: to put together multiple lms. We're trying to have synthetic data,
• 4:13
  Speaker 2: but the performance isn't really there. We've hit diminishing returns
• 4:17
  Speaker 2: not a month ago, but three and a half years ago.
• 4:19
  Speaker 1: Okay, And it's surely new data is created all the
• 4:22
  Speaker 1: time on the Internet. You know, we talk about diverse data,
• 4:25
  Speaker 1: and it's an awful lot more created in the past
• 4:27
  Speaker 1: than there is now. But everyone's use of the Internet
• 4:30
  Speaker 1: surely creates huge volumes of new data all the time.
• 4:33
  Speaker 2: That's even worse. Humans are creating new data on the Internet,
• 4:37
  Speaker 2: but that data falls into certain patterns. How many conversations
• 4:40
  Speaker 2: on the weather you can have every day?
• 4:42
  Speaker 1: A lot, actually a lot. I think I've already had
• 4:46
  Speaker 1: three today, to be honest. You know, it's very cold
• 4:47
  Speaker 1: in Edinburgh.
• 4:48
  Speaker 2: You can have a lot, but there's a bigger, profound
• 4:50
  Speaker 2: problem there I want to mention is that if you
• 4:53
  Speaker 2: look at the data that is currently being created on
• 4:55
  Speaker 2: the open Internet, it is data create by those llms,
• 5:01
  Speaker 2: which number one is inaccurate because it suffers from Halla's hallucinations,
• 5:06
  Speaker 2: and number two, it feeds into itself, so that the
• 5:10
  Speaker 2: new models now train on the entire Internet, the training
• 5:14
  Speaker 2: on the output from other models, which is I said,
• 5:17
  Speaker 2: are making mistakes and hallucinations, and that, using a technical term,
• 5:22
  Speaker 2: slowly ends up leading to something called model collapse, where
• 5:26
  Speaker 2: the models themselves are actually getting dumber.
• 5:29
  Speaker 1: Okay, so if you train a new model on newly
• 5:32
  Speaker 1: created data, you're effectively training on its own nonsense or
• 5:36
  Speaker 1: nonsense created by other similar models.
• 5:39
  Speaker 2: Yeah, well, not only nonsense. We have to understand that
• 5:41
  Speaker 2: those llms using artificial neural networks do you're correct let's
• 5:46
  Speaker 2: say ninety five ninety nine percent of the time, So
• 5:49
  Speaker 2: most of the content is correct, but you no longer
• 5:52
  Speaker 2: know what content is incorrect, which is a significant degradation
• 5:56
  Speaker 2: of the quality of data. It's almost like having access
• 5:59
  Speaker 2: to a calculator which claims to be correct one hundred
• 6:02
  Speaker 2: percent of the time, but in reality it is correct
• 6:04
  Speaker 2: ninety five ninety nine percent of the time. How much
• 6:07
  Speaker 2: would you pay for that calculator? Would you use the
• 6:09
  Speaker 2: output from that calculator to produce novel calculator? No, you
• 6:14
  Speaker 2: wouldn't do that. But we are doing it right now,
• 6:16
  Speaker 2: and we are seeing right now that the performance of
• 6:19
  Speaker 2: those models on benchmark is not increasing anymore plat out
• 6:22
  Speaker 2: three years ago. And there's even a bigger, profound question.
• 6:25
  Speaker 2: We are training those models now using higher quality data,
• 6:30
  Speaker 2: so no longer we're using the entire internet. Now we
• 6:33
  Speaker 2: are filtering from the Internet a lot of garbage, so
• 6:37
  Speaker 2: we're using a smaller training data set, thus producing smaller
• 6:42
  Speaker 2: llms that have the same quality. But as I mentioned,
• 6:45
  Speaker 2: they are smaller, So what is the consequence of it. Well,
• 6:48
  Speaker 2: as we speak, I'm running those models on my laptop.
• 6:50
  Speaker 2: I don't use any data center for it. I never will.
• 6:53
  Speaker 2: I'm running them on my laptop. So the models have
• 6:55
  Speaker 2: gotten better. But by saying better, I mean they are
• 6:58
  Speaker 2: more power efficial.
• 7:00
  Speaker 1: They're not all right, So I mean that they're most specific,
• 7:03
  Speaker 1: so that I get toward a more specific task because
• 7:06
  Speaker 1: they're using a narrower range of data, so they can't
• 7:09
  Speaker 1: be a type of general intelligence they're specific and.
• 7:12
  Speaker 2: Remarkably no, they are general purpose. I mean you can
• 7:16
  Speaker 2: just go online and down on quent three point five
• 7:18
  Speaker 2: and three point six. This is a general purpose model,
• 7:21
  Speaker 2: which is even accessing the internet to give you Summari
• 7:24
  Speaker 2: is reasoning. It's doing this thing on your laptop. It's today.
• 7:27
  Speaker 2: Of course. On top of that, if you want to,
• 7:30
  Speaker 2: you can find tune your model on your proprietary data.
• 7:34
  Speaker 2: But you see even the fine tuning process these days
• 7:38
  Speaker 2: can be done on your laptop, which again raises the question,
• 7:41
  Speaker 2: why do you need to pay for compute? Why do
• 7:44
  Speaker 2: you need all this expansion of the data centers. You
• 7:46
  Speaker 2: really don't. And what's going to happen in year two,
• 7:50
  Speaker 2: three years from now when the majority of laptops out
• 7:53
  Speaker 2: there will be able to run general purpose the most
• 7:56
  Speaker 2: potent language models with access to the Internet. I just
• 8:00
  Speaker 2: I'm just not seeing it. And that's on top of
• 8:02
  Speaker 2: one fundamental thing that I want to mention upfront, because
• 8:05
  Speaker 2: you mentioned what does the AI need from us? And
• 8:09
  Speaker 2: what we need from the AI? So what the AI
• 8:11
  Speaker 2: needs from us are two things, as you mentioned. Number one,
• 8:13
  Speaker 2: compute and yes, we can provide more and more compute
• 8:17
  Speaker 2: as long as oracle CDs is not going down, which
• 8:21
  Speaker 2: it is right now. And number two, we need to
• 8:24
  Speaker 2: provide more data. We've use all of it, so to
• 8:26
  Speaker 2: some extent we cannot produce better AI. Now what do
• 8:29
  Speaker 2: we want to have from the AI. We want to
• 8:32
  Speaker 2: have at least two things and just bury me here.
• 8:36
  Speaker 2: Number one, we want to have systems which are continually learning.
• 8:42
  Speaker 2: So just like during this podcast today, I hope that
• 8:46
  Speaker 2: I will be able to learn something from you. You
• 8:48
  Speaker 2: will be able to learn something from me and remember
• 8:51
  Speaker 2: tomorrow or maybe remembered after Easter. Current generation systems in general,
• 8:56
  Speaker 2: neural networks are not learning anything new you when you
• 9:00
  Speaker 2: interact with them, which is significant limitation. So we're not
• 9:03
  Speaker 2: getting from the AIS what we want to They're not
• 9:06
  Speaker 2: learning from us. It's a fundamental limitation that's not solved.
• 9:10
  Speaker 2: And number two, you want to mention this thing up front.
• 9:13
  Speaker 2: Those systems are stochastic. They are probabilistic. You cannot trust them.
• 9:20
  Speaker 2: They roll the dice whenever they produce output, so to
• 9:23
  Speaker 2: some extent you cannot trust their output. Can you make
• 9:26
  Speaker 2: them deterministic? Yes, of course you can by making sure
• 9:30
  Speaker 2: that they always produce the most likely token. Now, use
• 9:34
  Speaker 2: the word token. But the problem with this thing is
• 9:36
  Speaker 2: that then they will be just copying the data from
• 9:38
  Speaker 2: the training set. So just imagine those lawsuits when you
• 9:41
  Speaker 2: see verbatting copies of all the podcasts books produced as
• 9:46
  Speaker 2: an output of Gemini or open Ai system. So those
• 9:48
  Speaker 2: systems number one are not continually learning, which basically for
• 9:53
  Speaker 2: me is just a no go. And number two, those
• 9:55
  Speaker 2: systems are not to be trusted because whenever they produce output,
• 9:59
  Speaker 2: that output is it stochastic. It's not a terministic it's
• 10:03
  Speaker 2: basically rolling the dice to catch you output.
• 10:05
  Speaker 1: Yeah. Can we talk a bit more about that about
• 10:07
  Speaker 1: how the hallucinations or errors have it build up?
• 10:10
  Speaker 2: Absolutely? I me try to explain using simple terms when
• 10:14
  Speaker 2: you use ch GPT or Gemini or copilot, but this
• 10:19
  Speaker 2: is Copilot is actually open eye system. It just produces texts,
• 10:23
  Speaker 2: so you have an impression that generates one word at
• 10:27
  Speaker 2: the time deterministically, that's what's on the screen, just one
• 10:31
  Speaker 2: word after another. But in reality that's not what those
• 10:34
  Speaker 2: systems produce. If you are a developer like me and
• 10:39
  Speaker 2: my coelagus are developers as well, you can look at
• 10:42
  Speaker 2: the developer output of a large language. Models do you
• 10:45
  Speaker 2: know what it gives you. It gives you a vector
• 10:48
  Speaker 2: of fifty thousand elements, actually fifty two thousand elements, so
• 10:52
  Speaker 2: fifty thousand elements, and each element has a certain probability
• 10:56
  Speaker 2: of being correct zero point ninety five zero one zero
• 11:00
  Speaker 2: zero one zero, zero point three. It's an entire vector.
• 11:03
  Speaker 2: It's not one word has probability one. Everything else is zero. No,
• 11:07
  Speaker 2: there is a little bit of error there. It has
• 11:09
  Speaker 2: to be, and so not is what happens when you're
• 11:12
  Speaker 2: producing one word at the time, or one token at
• 11:15
  Speaker 2: the time. You can think of a token as a
• 11:18
  Speaker 2: ward split by two or three. So when you produce
• 11:21
  Speaker 2: output one token at the time, this system is rolling
• 11:24
  Speaker 2: the dice all the time. It's making a small, tiny
• 11:27
  Speaker 2: error every time it produces a ward s. At the beginning,
• 11:32
  Speaker 2: you may not perceive the error, but over time, after
• 11:34
  Speaker 2: three hundred, five hundred a thousand words, the error is
• 11:38
  Speaker 2: going to be so big that it's going to result
• 11:41
  Speaker 2: in the critical failure of the system. You cannot circumvent
• 11:45
  Speaker 2: it because those systems are probabilistic. It's not like in
• 11:48
  Speaker 2: an Excel spreadsheet, where you can have a chain of
• 11:51
  Speaker 2: aund ten twenty one hundred one thousand formulas and you
• 11:54
  Speaker 2: know that the formulas are going to produce the correct result. Here,
• 11:58
  Speaker 2: if you have a chain of words, every time you
• 12:02
  Speaker 2: produce a word, you accrue a little bit of error
• 12:05
  Speaker 2: and you see this error manifest itself later on.
• 12:08
  Speaker 1: Okay, so is this solvable? I guess that's the key question.
• 12:13
  Speaker 1: Is it possible? And the type of models that we're
• 12:15
  Speaker 1: using at the moment, which are super hyped, is it
• 12:18
  Speaker 1: possible for that hallucination problem or compounding error problem to
• 12:22
  Speaker 1: be solved, or is it simply a systemic shortcoming that
• 12:26
  Speaker 1: is unresolvable.
• 12:28
  Speaker 2: So here I'm again speaking from my experience having two
• 12:31
  Speaker 2: ephds in mathematics and computer science and twenty years of
• 12:34
  Speaker 2: experience in deep mind and IBM Watson research. No, it's impossible.
• 12:39
  Speaker 2: You have to use a different technique for it. Yes,
• 12:43
  Speaker 2: there are different techniques that are emerging right now. In
• 12:47
  Speaker 2: full disclaimer, I'm also a co founder of a startup
• 12:49
  Speaker 2: working on one of those techniques. We call them fractal brain. Yes,
• 12:53
  Speaker 2: there are new techniques on the horizon. However, the existing
• 12:56
  Speaker 2: techniques have a building mechanism so that every time produce
• 13:00
  Speaker 2: an output another word, you have a little bit of error.
• 13:04
  Speaker 2: You cannot eliminate it thing. And there's also a fundamental
• 13:07
  Speaker 2: other thing you mentioned hallucinations. What do we mean by hallucinations?
• 13:10
  Speaker 2: It's number one, it's producing those small errors one word
• 13:14
  Speaker 2: at a time. But there's also another reason for hallucinations
• 13:18
  Speaker 2: when this system just doesn't remember what was mentioned yesterday,
• 13:22
  Speaker 2: a week ago, or a month ago, and goes back
• 13:24
  Speaker 2: to the initial question. You need both. You need to
• 13:26
  Speaker 2: have a system which, like humans, it's continually learning. And
• 13:30
  Speaker 2: number two, you need to have a system which does
• 13:33
  Speaker 2: not make errors when it produces the next word. It
• 13:37
  Speaker 2: shouldn't roll a dice. You need those two and so
• 13:40
  Speaker 2: now to answer your question, can we solve this problem
• 13:43
  Speaker 2: using artificial neural networks? No, there are attempts to. There
• 13:49
  Speaker 2: have been attempts to circumvent it. If you want to
• 13:52
  Speaker 2: have continual learning, you can maybe try to use something
• 13:55
  Speaker 2: like continual backpropagation from rich Sutt only got a two
• 13:58
  Speaker 2: in go word in twenty twenty four. Or we can,
• 14:00
  Speaker 2: but it's not solving the problem. At least he's attempting
• 14:03
  Speaker 2: to find solutions to it. You can retrain the system,
• 14:07
  Speaker 2: of course, right, I mean, why not take GBT for
• 14:09
  Speaker 2: after the end of the podcast. Retrain the entire system.
• 14:12
  Speaker 2: It's going to cost you five million dollars. But you
• 14:14
  Speaker 2: could do that. You could retrain the entire system, or
• 14:18
  Speaker 2: you can find you in your system. But when you
• 14:20
  Speaker 2: find you in your system, the system is forgetting what
• 14:24
  Speaker 2: it was trained on before. So you're suffering something which
• 14:28
  Speaker 2: is called catastrophic forgetting or catastrophic interference. Long story short, No,
• 14:35
  Speaker 2: you cannot solve the outstanding problems of hallucinations and lack
• 14:39
  Speaker 2: of continual learning. Unfortunately. I wish you could, but you
• 14:41
  Speaker 2: need to have different things for it. And it's not
• 14:43
  Speaker 2: just me saying it. Look at the landscape of researchers,
• 14:47
  Speaker 2: leading researchers in the field, my colleagues, I know all
• 14:49
  Speaker 2: of them personally. They have all jumped ship. This is important.
• 14:54
  Speaker 2: Look at, for example, young Lecon from Metaga. He jumped ship.
• 14:58
  Speaker 2: He's working on his new startup am I line apps,
• 15:00
  Speaker 2: not working on llms, saying l lams is a dead
• 15:03
  Speaker 2: end to AGI. Look at Michael Iff Dave Silver from DeepMind.
• 15:07
  Speaker 2: He just left deep Mind, but a couple of weeks
• 15:09
  Speaker 2: ago he formed Ineffable Intelligence. I think they're raising a
• 15:12
  Speaker 2: billion dollars the same thing. He is not a believer
• 15:15
  Speaker 2: of using l lms for general artificial intelligence, and can
• 15:19
  Speaker 2: go on and on Iliasutskiver, Andre Carpaty. And so it's
• 15:24
  Speaker 2: not just me who is saying that we need to
• 15:26
  Speaker 2: go back to research. The leading research and researchers in
• 15:30
  Speaker 2: the field have already jumped ship a few months, maybe
• 15:34
  Speaker 2: even years ago, working on the next generation things. The
• 15:37
  Speaker 2: market still believes we can solve hallucinations, but the leading
• 15:41
  Speaker 2: researchers have jumped ship. That's unbelievable to me that we
• 15:46
  Speaker 2: keep pouring money into bigger data centers, knowing we've used
• 15:50
  Speaker 2: all the data already, and knowing that even if they
• 15:53
  Speaker 2: have more data, you will not solve continual learning, and
• 15:57
  Speaker 2: you will not solve hallucinations. So why is not everyone?
• 16:01
  Speaker 2: Why are people doing it?
• 16:02
  Speaker 1: Okay, So if we know clearly, and it sounds from
• 16:05
  Speaker 1: what you say that we do know very clearly that
• 16:07
  Speaker 1: using more and more computing power and more and more
• 16:10
  Speaker 1: already mildly corrupted data isn't going to get us anywhere.
• 16:14
  Speaker 1: Then the enormous cap expand on vast data centers, the
• 16:19
  Speaker 1: hundreds of billions of dollars that have already gone into
• 16:22
  Speaker 1: this and are still projected to go into This is
• 16:24
  Speaker 1: a catastrophic misallocation of capital.
• 16:27
  Speaker 2: Well that's how you look at it. So obviously you
• 16:31
  Speaker 2: can say the biggest winner of it is Nvidia because
• 16:34
  Speaker 2: it's producing those GPUs or during the gold rush, you
• 16:37
  Speaker 2: should invest in companies that produce the shovels you can
• 16:40
  Speaker 2: still make money. For example, I'm a partner at Urn
• 16:43
  Speaker 2: Innovation Capital in the UK and we are investing in
• 16:48
  Speaker 2: airs mission two companies, one called high Verge and second
• 16:51
  Speaker 2: called Hydra. These companies improve coolings of data centers or
• 16:57
  Speaker 2: these companies produce better algorithms to run on data centers.
• 17:01
  Speaker 2: So you can still allocate your capital wisely, but you
• 17:04
  Speaker 2: shouldn't allocate them in companies which are spending on this compute.
• 17:09
  Speaker 2: You should allocate your capital in companies that are allowing
• 17:12
  Speaker 2: those data centers to run efficiently because those data centers,
• 17:16
  Speaker 2: who knows, maybe they will be used for different purpose
• 17:19
  Speaker 2: at some point. So again, there are going to be
• 17:22
  Speaker 2: winners and losers of the current gold rush in GPUs
• 17:26
  Speaker 2: and AI. I would say companies that have not invested
• 17:31
  Speaker 2: massively in data centers and in front here models are
• 17:34
  Speaker 2: going to be the winners. There's some companies without mentioning
• 17:38
  Speaker 2: the names, that have been accused of not training their
• 17:42
  Speaker 2: own language models. I think these, to me are going
• 17:45
  Speaker 2: to be the winners in today's market.
• 17:47
  Speaker 1: Are we talking about Apple?
• 17:49
  Speaker 2: I will let you determine that thing. But you can
• 17:52
  Speaker 2: see some companies number one, have not invested in the
• 17:56
  Speaker 2: LM front here models, but the research teams have kept
• 17:59
  Speaker 2: public papers saying that those lms stayed on the reason
• 18:02
  Speaker 2: they make mistakes. So let you find those companies. And
• 18:05
  Speaker 2: there are some companies that have borrowed a lot of
• 18:08
  Speaker 2: money to expand those center centers. And it's not me.
• 18:11
  Speaker 2: Look at the market, look at CDs on Oracle. It's
• 18:14
  Speaker 2: the market is just flashing reds saying this is foolish.
• 18:17
  Speaker 2: So we're seeing those signals already. But I want to
• 18:20
  Speaker 2: make sure that if you want to make money in
• 18:21
  Speaker 2: today's market, is given that the governments have to pay
• 18:25
  Speaker 2: eight nine percent of the revenue on servicing the debt.
• 18:29
  Speaker 2: You don't know if the market is going to go
• 18:31
  Speaker 2: up and down, and maybe a capital injection, liquid, the
• 18:33
  Speaker 2: injection from Central Bucks. You don't know those things. So
• 18:35
  Speaker 2: I would not recommend your short or go along any investment,
• 18:39
  Speaker 2: I would recommend maybe doing an arbitrush buy companies that
• 18:42
  Speaker 2: have not wasted money on llms and short companies that
• 18:46
  Speaker 2: have borrowed a lot of money to expand data centers.
• 18:49
  Speaker 2: That will be my suggestion, but I might be wrong.
• 18:51
  Speaker 1: Again. Can you tell us about any of the startup
• 19:16
  Speaker 1: companies that you're invested in or interested in that are
• 19:19
  Speaker 1: taking us to this new frontier and AI away from
• 19:22
  Speaker 1: the LLM model and towards a different model.
• 19:24
  Speaker 2: Absolutely so. In our investment fund, our an innovation capital,
• 19:27
  Speaker 2: we have access to a lot of companies, actually maybe
• 19:30
  Speaker 2: twenty or thirty companies that are developing the next frontier
• 19:33
  Speaker 2: models which are not necessarily using artificial in our networks.
• 19:36
  Speaker 2: And I can speak about three or four of them
• 19:39
  Speaker 2: which are very exciting. One company you can have a look.
• 19:42
  Speaker 2: They are based in Switzerland. They're called Innate AI. Again,
• 19:46
  Speaker 2: I'm advertising them and we're not investing in them. We're
• 19:48
  Speaker 2: not investing them yet. They are developing new version of
• 19:53
  Speaker 2: neural networks which are inspired on the brain. This is
• 19:57
  Speaker 2: an effort that was going on in Europe for more
• 19:59
  Speaker 2: than a decade, the Blue Brain project. They are developing
• 20:02
  Speaker 2: something new which is not an artificial neural network. So
• 20:05
  Speaker 2: that's one kind. Look at another company, for example, PATHWAYAI
• 20:10
  Speaker 2: in the Bay Area. Again another example they mentioned upfront,
• 20:15
  Speaker 2: you need to solve continual learning. You need to solve that.
• 20:18
  Speaker 2: If you don't have it, forget about the solution to AGI.
• 20:21
  Speaker 2: And so they have been developing systems that can learn
• 20:27
  Speaker 2: using something called heavy on learning, which is a local
• 20:30
  Speaker 2: learning technique that happens in the brain, not using bad
• 20:32
  Speaker 2: propagation grid in descent. So this is another example. Another
• 20:36
  Speaker 2: company that I'm actually a co founder of and the
• 20:39
  Speaker 2: CEO called Fractal Brain AI. Have a look at the thing.
• 20:42
  Speaker 2: It's also based on prefrontal cortex, and it's this idea
• 20:45
  Speaker 2: that those networks are continually growing and rewiring themselves. So
• 20:50
  Speaker 2: no longer you have a fixed network with a fixed
• 20:53
  Speaker 2: number of parameters. No, the network is growing expanding themselves.
• 20:58
  Speaker 2: Like today, you're going to probably form a connections after
• 21:00
  Speaker 2: this podcast. Those networks do the same, They create new
• 21:03
  Speaker 2: connections all the time, and on top of that, they
• 21:06
  Speaker 2: are continually learning and thousands of times more power efficient
• 21:10
  Speaker 2: in addition to being data efficient. So these are only
• 21:13
  Speaker 2: some of the examples of companies that I'm personally very
• 21:15
  Speaker 2: excited about. But as ILIOSU Skiverer said the other day
• 21:20
  Speaker 2: on one of those podcasts, that we have gone back
• 21:25
  Speaker 2: from the age of scaling to the age of research,
• 21:30
  Speaker 2: So researchers have gone back to developing the new things.
• 21:33
  Speaker 2: It's just onlin end here me and my teams have
• 21:37
  Speaker 2: started developing, for example, fractals fractal brain twelve years ago.
• 21:41
  Speaker 2: We knew about those outstanding limitations of artificial neural networks
• 21:45
  Speaker 2: more than a decade ago, so we wouldn't invest our
• 21:47
  Speaker 2: time in it.
• 21:48
  Speaker 1: Nevertheless, somehow I got to cut up in this sort
• 21:50
  Speaker 1: of super bubble hype, despite the fact that good scientists
• 21:54
  Speaker 1: aren't you.
• 21:54
  Speaker 2: But this is good. I like the hype because you see,
• 21:57
  Speaker 2: to some extent that hype and the lands they allowed
• 22:02
  Speaker 2: us to understand that it's possible to approximate human language.
• 22:07
  Speaker 2: So now that you know that you can approximate human
• 22:09
  Speaker 2: language with el lamps, you can try to find ways
• 22:13
  Speaker 2: to actually solve the idea of human language. If you
• 22:16
  Speaker 2: can approximate something, you can see the size of it.
• 22:19
  Speaker 2: So it's almost like if someone showed you, hey, there
• 22:22
  Speaker 2: is a rocket there, it flies, you already know the
• 22:24
  Speaker 2: size of the rocket. You can know you can fly,
• 22:26
  Speaker 2: you can start to crack the details of the engine
• 22:29
  Speaker 2: of the rocket. So to some extent, I like the
• 22:31
  Speaker 2: current hype. I like the current generation systems because they
• 22:34
  Speaker 2: allowed us to understand the size of the problem and
• 22:38
  Speaker 2: approximate it. Now that we know that we can in
• 22:40
  Speaker 2: principle approximate human language, let's just solve it, okay.
• 22:44
  Speaker 1: And the other thing I suppose we should say is
• 22:46
  Speaker 1: that while we spent quite a lot of time criticizing
• 22:48
  Speaker 1: this generation of llms, they're still great, still as really useful.
• 22:52
  Speaker 1: It's not like we have a totally pointless technology, something
• 22:55
  Speaker 1: that can remove entry level jobs across the board, which
• 22:58
  Speaker 1: of course comes with its own problems, but none LASS
• 23:01
  Speaker 1: has enormous use from activity and business.
• 23:04
  Speaker 2: Absolutely, I love maybe not just l elms. I love
• 23:07
  Speaker 2: the generative AI. For example. Not sure if you've noticed
• 23:09
  Speaker 2: behind me, I have this amazing landscape of London, but
• 23:13
  Speaker 2: you can tell it's all fake here, like this building
• 23:15
  Speaker 2: is all tilted here. So those systems they produce very
• 23:20
  Speaker 2: pretty graphics. It's inaccurate, but it's okay for me. It
• 23:23
  Speaker 2: still gives me a very nice background. Same thing with text.
• 23:26
  Speaker 2: They will produce beautifully looking poem. They can summarize the document. Yes,
• 23:32
  Speaker 2: there are errors. The are like this building here. You
• 23:34
  Speaker 2: can tell it's all tilted a little bit, But I'm
• 23:37
  Speaker 2: okay with that. So those systems are very good for
• 23:40
  Speaker 2: creating templates, knowing what templates of data, it's so called
• 23:44
  Speaker 2: boiler plate code, creating nice graphics. They're not good for
• 23:48
  Speaker 2: details to put in there. And it's interesting because about
• 23:51
  Speaker 2: two or three years ago, I was giving a talk
• 23:55
  Speaker 2: to high school students. They're asking me what do we
• 23:58
  Speaker 2: use those lms for. I told them for generating templates,
• 24:02
  Speaker 2: templates of presentation, etc. And for summarizing documents. But I
• 24:08
  Speaker 2: misled them. I don't think you should be using those
• 24:10
  Speaker 2: systems for summarization for two reasons. The reason number one,
• 24:15
  Speaker 2: in that summary there might be errors and mistakes, So
• 24:19
  Speaker 2: if you summarize a document, don't throw away the original.
• 24:23
  Speaker 2: And number two, you know, when you're summarizing something, you
• 24:27
  Speaker 2: should know what you care about. For example, if you
• 24:29
  Speaker 2: are to summarize today's podcast, maybe you only care about
• 24:33
  Speaker 2: this tilted building here, which is fake. Maybe that's what
• 24:36
  Speaker 2: you're looking for. LM doesn't know it. So when it
• 24:39
  Speaker 2: produces a summary of text. When it compresses graphics, it
• 24:43
  Speaker 2: doesn't know what you care about really, so it's going
• 24:46
  Speaker 2: to produce a summary maybe lacking the details that you
• 24:49
  Speaker 2: want to know later on.
• 24:51
  Speaker 1: And I suppose the other things with the errors. You
• 24:54
  Speaker 1: should only really be using it for things where you
• 24:57
  Speaker 1: know that you will be able to spot the errors
• 24:58
  Speaker 1: at the end.
• 24:59
  Speaker 2: So this is a actually very interesting I think that
• 25:01
  Speaker 2: the killer use case for generative AI is producing output
• 25:05
  Speaker 2: that you yourself can check for correctness. So for example,
• 25:09
  Speaker 2: if I'm if I want to compute one hundred plus
• 25:11
  Speaker 2: one hundred, LM will give me an answer I know
• 25:14
  Speaker 2: I can check the answer myself the correctness. I like
• 25:17
  Speaker 2: it this way. Unfortunately, people are using those lms to
• 25:22
  Speaker 2: answer a question to which they themselves don't know the
• 25:25
  Speaker 2: answer to. This is a recipe of an absolute disaster.
• 25:30
  Speaker 2: In the worst case, you can use those systems to
• 25:34
  Speaker 2: check whether your output that you produce yourself is correct.
• 25:38
  Speaker 2: You can do it that way. But people using the
• 25:40
  Speaker 2: other way around. They are asking those lms to produce
• 25:43
  Speaker 2: an output they don't know what the output should be,
• 25:46
  Speaker 2: and the output can have one or five percent error, right,
• 25:49
  Speaker 2: why would you even do that?
• 25:50
  Speaker 1: So are we worrying unnecessarily about the job market. We
• 25:53
  Speaker 1: spend most of the talks and podcasts and panels, etc.
• 25:56
  Speaker 1: That I do, the question is always what on earth
• 25:58
  Speaker 1: does my child do for work? In an age of AI.
• 26:01
  Speaker 1: Are we worrying too much about that? Because the human
• 26:03
  Speaker 1: input will remain absolutely compulsory for the next couple of decades.
• 26:08
  Speaker 2: Having the same problem as I mentioned, my son as
• 26:11
  Speaker 2: is thirteen years old, so mis the father to give
• 26:14
  Speaker 2: him advice what to do in the future. I guess
• 26:17
  Speaker 2: being a scuba diver instructor is a good job. Yeah,
• 26:20
  Speaker 2: it is a great job. You're going to need them.
• 26:22
  Speaker 2: I worry about interns entry level of software engineers because
• 26:27
  Speaker 2: to some extent you can automate. It's not replaced. You
• 26:31
  Speaker 2: can automate most of the tasks that you delegate to
• 26:35
  Speaker 2: interns today, like, for example, write a boilerplate code, template code,
• 26:40
  Speaker 2: check some if there's errors in that code, you can
• 26:43
  Speaker 2: automate that. But what's going to happen then is that
• 26:45
  Speaker 2: we are not going to have interns anymore or significantly
• 26:48
  Speaker 2: smaller number of interests and entry level software engineers. So
• 26:52
  Speaker 2: what's the consequence. What's going to happen with the entire
• 26:55
  Speaker 2: promotion cycle. If you're in senior management or a middle management,
• 27:01
  Speaker 2: you're gonna get promoted. Who's going to replace you? Who's
• 27:04
  Speaker 2: going to become a software architect if we're not training
• 27:08
  Speaker 2: the new entry level software engineers. So there's this entire
• 27:11
  Speaker 2: skills gap right now. Some people that I know have
• 27:14
  Speaker 2: chosen not to pursue studies in computer science because for
• 27:17
  Speaker 2: the very reason they worry that we're not going to
• 27:20
  Speaker 2: need software engineers. Yes, we are going to need software engineers,
• 27:23
  Speaker 2: it's just you need to jump immediately to being an
• 27:28
  Speaker 2: architect of a software engineering system. And to make sure
• 27:31
  Speaker 2: verry hard. It is hard. It Typically what you do
• 27:35
  Speaker 2: is you get gain this experience on the job. You
• 27:37
  Speaker 2: go to Google, spend first two three years writing code,
• 27:40
  Speaker 2: but you appear so software architects, and so you learn
• 27:43
  Speaker 2: from them. If we don't have this experience that we're
• 27:46
  Speaker 2: giving to entry level software engineers, they won't be able
• 27:49
  Speaker 2: to have this experience. So this is what always me more.
• 27:51
  Speaker 2: It's not about replacing software engineers, it's about us not
• 27:56
  Speaker 2: having a pipeline of software architects. And see your software engineers,
• 28:01
  Speaker 2: we are not having this pielime anymore. That worries me
• 28:03
  Speaker 2: quite a bit, to be honest.
• 28:04
  Speaker 1: Yeah, and the pipeline problem is just being discussed in
• 28:07
  Speaker 1: a lot of other professions as well, most obviously in
• 28:09
  Speaker 1: a legal profession.
• 28:10
  Speaker 2: Absolutely, again for lawyers, this is very interesting and this
• 28:12
  Speaker 2: is not my profession, so you can discount what I
• 28:14
  Speaker 2: say quite a bit way. Now, you can get the
• 28:18
  Speaker 2: initial blueprints of legal documents very quickly, and we do
• 28:21
  Speaker 2: it all the time at our startup. You can get
• 28:23
  Speaker 2: a blueprint of a legal document, but I would never
• 28:26
  Speaker 2: use that document to get an investor on board in
• 28:29
  Speaker 2: my startup. I wouldn't do that. I still need to
• 28:32
  Speaker 2: send it to an actual human being to at least
• 28:34
  Speaker 2: prove read it. So we're going to have to have
• 28:37
  Speaker 2: those lawyers which can prove readdocuments produced by the generative AI,
• 28:43
  Speaker 2: but aren't doing it already. They already have hundreds of
• 28:46
  Speaker 2: template documents safe on their computers. It's just changing the
• 28:50
  Speaker 2: names of companies investors in those documents. So the legal
• 28:53
  Speaker 2: profession is not going to go away because those generative
• 28:57
  Speaker 2: AI systems, they don't have the notion of true and false.
• 29:01
  Speaker 2: They don't they confused. It's all probabilistic, so they will
• 29:04
  Speaker 2: make an error. Sometimes after twenty thirty forty legal statements,
• 29:09
  Speaker 2: they will just change one true into false. So we're
• 29:13
  Speaker 2: still going to need to have lawyers for it. This
• 29:15
  Speaker 2: is one of those professions where I don't think it's
• 29:17
  Speaker 2: going to be automated fully by AI. But as I said,
• 29:20
  Speaker 2: there are other professions that have already been automated. For example,
• 29:23
  Speaker 2: content creators. So you can go online, go to YouTube.
• 29:25
  Speaker 2: You're going to see a lot of videos summarizing AI,
• 29:28
  Speaker 2: AI bubble summarizing conflict in Ukraine or Iran, all generated
• 29:33
  Speaker 2: using generative AI. So some jobs have already gone away,
• 29:37
  Speaker 2: the jobs that require to be you to be one
• 29:39
  Speaker 2: hundred percent accurate, those jobs are not going to go
• 29:43
  Speaker 2: away using the current generation of AI systems. Remember, there
• 29:47
  Speaker 2: are new generations on the horizons. My startup is working
• 29:51
  Speaker 2: on it, other startups are working on it. But with
• 29:53
  Speaker 2: the current generation of tools, they will not displace those
• 29:56
  Speaker 2: jobs yet.
• 29:57
  Speaker 1: Yeah, it sounds to me like we shouldn't necessarily be
• 30:00
  Speaker 1: frightened the current generation of tools, but we should hardly
• 30:02
  Speaker 1: be pretty frightened the next generation.
• 30:05
  Speaker 2: I would give a very simple example here. So me
• 30:07
  Speaker 2: and my colleagues we've built Alpha go at Deepmine in
• 30:11
  Speaker 2: twenty fifteen, which is the system that won in computer
• 30:14
  Speaker 2: Goal with the World champion, and so back then it
• 30:17
  Speaker 2: was a state of the art system. These days, you
• 30:22
  Speaker 2: yourself can win against that system. I can do it
• 30:25
  Speaker 2: as well. Why because those systems have stayed the same,
• 30:30
  Speaker 2: and humans have identified flaws in that systems. They have
• 30:34
  Speaker 2: identified hallucinations errors, And now when you play against that system,
• 30:39
  Speaker 2: you just exploit this system, exploit its weaknesses. The system
• 30:43
  Speaker 2: is not adapting itself, not rewiring itself. So to answer
• 30:46
  Speaker 2: your question quickly, I'm personally not worried of existing AI
• 30:50
  Speaker 2: systems because they are adapting themselves. However, I do have
• 30:53
  Speaker 2: to mention I do worry that people are going to
• 30:58
  Speaker 2: be using existing systems in domains where they should not
• 31:03
  Speaker 2: be used, for example, for identifying targets to bombing around.
• 31:07
  Speaker 2: It's just don't do those things. Those systems make errors
• 31:09
  Speaker 2: and hallucinations. So I worry about misuse of existing AI tools.
• 31:15
  Speaker 2: I don't worry about these tools themselves being malicious. No,
• 31:19
  Speaker 2: I worry about inadvertent misuse of those tools without understanding
• 31:23
  Speaker 2: what they are not good for.
• 31:26
  Speaker 1: Okay, interesting, always something to worry about. Can I ask
• 31:29
  Speaker 1: you one last thing. One of the things that you
• 31:31
  Speaker 1: mentioned earlier was the extraordinary energy inefficiency of the current systems.
• 31:36
  Speaker 1: And it is true, isn't it. The human brain is
• 31:39
  Speaker 1: remarkably energy efficient. And when you look at these models,
• 31:43
  Speaker 1: the amount of energy that they will use to simply
• 31:45
  Speaker 1: have the same thought that the human can have on
• 31:47
  Speaker 1: a couple of words, it's that's an extraordinary problem that
• 31:50
  Speaker 1: in your new generation that we talked about earlier will
• 31:54
  Speaker 1: be diminished.
• 31:55
  Speaker 2: Absolutely. The systems that I know that fractal brain is
• 31:59
  Speaker 2: coming up with, innate AI is coming up with. See
• 32:03
  Speaker 2: those systems, they don't use back propagation to produce the
• 32:08
  Speaker 2: next word. They don't have to load from the memory
• 32:13
  Speaker 2: all of those one point five trillion parameters just to
• 32:17
  Speaker 2: produce the next word. They don't do that. They load
• 32:20
  Speaker 2: maybe one hundred, maybe one thousand parameters three four orders
• 32:24
  Speaker 2: of magnitude better power efficiency. And we know this thing,
• 32:27
  Speaker 2: and we know that we've trained what first version of
• 32:29
  Speaker 2: our fractal language model using I think zero point one
• 32:33
  Speaker 2: percent electricity that open AI use for GPT one and two,
• 32:36
  Speaker 2: so we know it's possible to do that. It finds
• 32:38
  Speaker 2: its interesting. Before this podcast, we had a conversation. If
• 32:42
  Speaker 2: you recall me on Easter dinner and cooking potatoes. So
• 32:46
  Speaker 2: as we're talking right now, I can assure you that
• 32:50
  Speaker 2: for me and for you to produce the next word
• 32:53
  Speaker 2: in our conversation, you are not thinking about potatoes for Easter.
• 32:57
  Speaker 2: You're not doing that thing. You don't need those connections too.
• 33:00
  Speaker 2: When we talk about a we don't do those things.
• 33:02
  Speaker 2: And maybe now you think about your Eastern dinner. But
• 33:04
  Speaker 2: the point is that those systems should produce the next
• 33:08
  Speaker 2: word only loading up parameters which are important for the
• 33:12
  Speaker 2: next world. And this is not billions of parameters, nor
• 33:16
  Speaker 2: hundreds of thousands of most maybe a thousand, two thousand parameters.
• 33:19
  Speaker 2: So yes, the next generation systems, in addition to being deterministic,
• 33:23
  Speaker 2: you can trust their output. They have power and the
• 33:27
  Speaker 2: data efficiency which is remarkable, talking three orders of magnitude better,
• 33:31
  Speaker 2: it's coming out. This thing is coming better. And again
• 33:34
  Speaker 2: it's the world may not be ready for it yet.
• 33:36
  Speaker 2: Plus you didn't understand my own motivation for building those
• 33:40
  Speaker 2: systems and the dangers. We have systems which are adapting themselves,
• 33:45
  Speaker 2: rewiring themselves, so to some extent you cannot outsmart them permanently.
• 33:50
  Speaker 2: They'll learn from your mistakes. It's like your kid. Try
• 33:53
  Speaker 2: using some technique on your kid. It's going to learn
• 33:55
  Speaker 2: how to adapt itself. Those systems do it as well,
• 33:58
  Speaker 2: So I personally do not no if it's a good
• 34:00
  Speaker 2: time to release those systems to the general public. I don't.
• 34:04
  Speaker 2: That's why those systems are not being published because think
• 34:07
  Speaker 2: about it. If it has a solution for a system
• 34:09
  Speaker 2: which is learning all the time adapting it's flaws, that
• 34:13
  Speaker 2: becomes spooky to release that system. So I worry more
• 34:18
  Speaker 2: about potential misuse of those next generation systems, less worry
• 34:22
  Speaker 2: about their performance because they're already beating existing systems on
• 34:26
  Speaker 2: common benchmarks.
• 34:27
  Speaker 1: Okay, thank you very much. It's one last thing imish
• 34:31
  Speaker 1: before we go. I think our listeners will have been
• 34:33
  Speaker 1: absolutely fascinated by all this and what I often ask people,
• 34:36
  Speaker 1: I'm going to ask you as well, and I hope
• 34:37
  Speaker 1: that we'll be able to understand that.
• 34:40
  Speaker 2: Your suggestion, what are you reading at the moment, Well,
• 34:42
  Speaker 2: I wish I could tell you the IM reading books
• 34:44
  Speaker 2: on AI, but I don't. In fact, four years ago
• 34:46
  Speaker 2: I started learning without any necessity Spanish, zero necessity, nothing.
• 34:51
  Speaker 2: The reason I started doing that is I wanted to
• 34:55
  Speaker 2: again learn a language the way humans do that without
• 35:00
  Speaker 2: billions of words, not with a small number of words.
• 35:02
  Speaker 2: So as an expers a mental experiment to learn whether
• 35:05
  Speaker 2: our fractal language model learns language in the same way
• 35:08
  Speaker 2: as I'm learning Spanish, I started learning Spanish, and because
• 35:11
  Speaker 2: of that, I'll I'll disappoint you. I'm reading kids' books,
• 35:15
  Speaker 2: reading Diary of the wimpik It in Spanish. Of course,
• 35:17
  Speaker 2: I'm reading Harry Potter as well in Spanish. I'm reading
• 35:19
  Speaker 2: lots and lots of books in Spanish, but these are
• 35:23
  Speaker 2: elementary school books. I'm sorry to disappointed.
• 35:25
  Speaker 1: You're reading the same things in Spanish that my son is.
• 35:28
  Speaker 1: So there we go in common.
• 35:29
  Speaker 2: Yes, absolutely, this is actually good because if you want
• 35:32
  Speaker 2: to understand, for example, how language works, at least try
• 35:35
  Speaker 2: to learn yet another language. Now that you can do
• 35:37
  Speaker 2: an introspection, you can see how you're learning a language,
• 35:40
  Speaker 2: and so it's remarkable you can learn a language after
• 35:43
  Speaker 2: in my case, about seven or eight thousand hours. I
• 35:46
  Speaker 2: guess my Spanish is better than my English right now,
• 35:48
  Speaker 2: So eight thousand hours, how many tokens we're talking about?
• 35:51
  Speaker 2: Couple million tokens, couple million tokens of a training set
• 35:56
  Speaker 2: rather than couple trillion tokens. So that's what fascinates me.
• 36:01
  Speaker 2: And yeah, maybe next time we can speak in the
• 36:03
  Speaker 2: Spanish on the podcast.
• 36:04
  Speaker 1: No, I'll have to get one of my kids in
• 36:06
  Speaker 1: for that.
• 36:07
  Speaker 2: Okay, sounds good.
• 36:09
  Speaker 1: Yeahhish, thank you so much for joining us today.
• 36:11
  Speaker 2: Its pleasure. Thank you so much for having me.
• 36:18
  Speaker 1: Thanks for listening to this week's Marin Talks Money. If
• 36:20
  Speaker 1: you like our show, rate to review and subscribe wherever
• 36:22
  Speaker 1: you listen to podcasts, and keep sending your questions or
• 36:24
  Speaker 1: comments the Merrin Money at Bloomberg dot net. You can
• 36:27
  Speaker 1: also follow me and John on Twitter or x I
• 36:30
  Speaker 1: met Marinas w and John is Underscore Stepic. This episode
• 36:34
  Speaker 1: was hosted by Me Marenzumaset Web was produced by Sammersadi
• 36:37
  Speaker 1: and Moses and sound designed by Blake Maples and Aaron
• 36:40
  Speaker 1: Kasper and special thanks of course to Yannish Mareski.