Latest / Beyond Vibe Coding / #S2.08 AI Sovereignty - the sipgate experiment with 80 developers - Stefan Lange-Hegermann
Transcript
- André Neubauer: And then you decided to buy that stuff. So is there a specific reason why you went that path? Stefan Lange-Hegermann: That's a good question. Well, actually, yes, André Neubauer: I give my best, I give my best. Sebastian: Welcome to another episode of our new season of Beyond Vibe Coding, partnering with Impala Search. André Neubauer: the go-to tech and executive search agency in Germany. Sebastian: In this podcast, we explore the transformational change in software engineering and knowledge work in general. I'm Sebastian Heidemeyer zu Erpen, CTO at NorthAo. André Neubauer: and I'm Andre CDPO at Trusted Shops. Great to have you back. This time we are talking again to someone from Zipgate who are very actively utilizing AI. Sebastian: Yes. And this is actually the first episode about local AI that we're running. We have planned more episodes to dive deeper into this topic. But for now, we hope you enjoy this first one. Welcome Stefan. We're super happy to have you on the show. You're the second person from Zipgate actually, we're having on the show. And maybe we touch upon that in a minute or so, but before that, please as always introduce yourself to our listeners. Stefan Lange-Hegermann: Sure. Yeah, my name is Stefan. I have been at Zipgate for 18 years now. Yeah, I entered the industry right from university during the dot-com years because it was so exciting. I dropped out of university and started to work in the field. And yeah, then I joined Zipgate a little bit later and have been doing everything at Zipgate. like everything a tech person can do. I started as a front-end developer. We were the second Scrum team at Zipgate. Then I became product owner, product lead, and now I do the lead for AI and for APIs. So yeah, and now I was so excited that I get a second chance to enter an industry that is starting off. So that was AI, and I'm trying to... trying to make sense of AI for a subgate as a company right now. And that's my job. André Neubauer: Can you elaborate what that means? So making sense of AI for Zipgate or being the AI lead at Zipgate? Stefan Lange-Hegermann: Yeah, I can try. Well, making sense of AI is something we all try, I guess. And no one would say we are able to do it. So I try to do it in a way that makes sense for Subgate, which makes it a little bit easier, but just a little bit easier. As a company, Subgate has always moved fast. We've always, we were very, very early in the agile movement. have tried to keep pace with everything. In our industry, I think we were the first ones to experiment with AI and now we are in a good position and we were able to start agents from within. So that is the starting point. And in, I think 2018 maybe was our first venture into AI. From a product standpoint, we were asking ourselves, is there a way to integrate AI into telephony? And the answer back then, after about a year of work, was, well, there are very interesting ways to do it, but the technology is not there yet. We cannot make sense of what people say, and we cannot understand them thoroughly, especially in German. So we always looked into that, and we tried to keep pace. we were looking for a point when it became a technology that worked for us. And then everyone knows the LLM revolution came. Even speech to text became better, and we started to build products around it. And during that time, it became clear that building products around it was not the only thing, because the building itself was becoming a thing driven by AI. engineering changed profoundly in the last two years. And my job right now is to make sense of how we integrate AI into our daily work, how we integrate it into our products, and how our APIs play into that whole field. Because we believe that very shortly in the future, many people are not going to be able to click around and use a product anymore because they will have an agent that does it for them. A colleague of mine just a few weeks ago said, well, I don't know how to click anymore. I need an agent to do it. So if you don't have an agent for it, I can't use it. I thought that was a very interesting point and that opened the eyes of many people at Subgate. So we are now actively pursuing becoming an, well, AI first sounds like a buzzword for managers who don't know what they're doing, André Neubauer: Hmm. Stefan Lange-Hegermann: but we still try to become an AI first company that does engineering with AI and does products that use AI to make the product better and not just for having AI in the list of features, which is quite a task. So yeah, that's what I try to do. I try to integrate it into our strategy. I try to make sure that we don't move away from where the action happens. And I try to steer us in the right direction with AI usage. Yeah. And that pretty much leads us to today's topic because one of the biggest problems we had in the last few weeks, well, actually, it was not a practical problem, but it was something we would think thought about was what happens if we lose access. to the technology. André Neubauer: Yep. Yep. Sebastian: How come? How how did this question come up? Yeah. it all all so we'll we'll get into that in a in a minute, but before that I I wanna ask maybe one more question on on specifically your AI like product landscape or how you view AI within your products, because I could imagine then that in the telephony space, text to speech, I think you mentioned this, right? Speech to text is vital, but not just that. You you probably also need grounding for specific information that that agents want to share or yeah, how how can we imagine that? Stefan Lange-Hegermann: Well, actually, everyone in the industry is doing the same stack. They are starting with speech-to-text, then they have an LLM, and then they have text-to-speech. That's very much simplified, but that is the stack we all have. Interesting is, where do we attach our technology? And we are in the very lucky position that before the big AI hype, we started to rebuild our platform in a way that we are now able to integrate any technology at any point in the whole telephony framework, starting from the phone numbers to the point where here we have interconnects to other networks. So that made it easy for us to integrate those technologies into what we have. And that is a huge asset for us because we are now able to experiment wherever we want. access to data because we asked our customers very early, can we use parts of your data for our work? And we started to anonymize data from our customers who gave permission very early on. So now we have a full stack of, on every layer of the technology where we can attach what we do. And that makes us able to have a very, very quick development cycles and to see where interesting points are. often say it's very hard for someone who doesn't have the insights to have good ideas that exceed the ideas of others. And I believe the full picture gives us a unique point of view where we can see the whole technology stack and we can see points where we can attach new technology that, well, at least not that many people have thought about before. That is pretty unique with the company and we have invested for a very long time. We had an AI team, I think for 10 years now, which is now bigger than it was a few years ago, but still people were paid to think about AI for a very long time. And that helps a lot. having access to all the technology layers and the data is another big asset. Sebastian: And maybe can you share a concrete example of how this like translates into a concrete product that that you're building? Stefan Lange-Hegermann: Let me think about which one I can tell you about. yes. Sebastian: Yes, yeah, yeah, of course. Stefan Lange-Hegermann: Well, actually, my team does experiments, and they don't do that in a lab, although it's called a labs experiment. They do that with real customer data. They ask customers if they want to join the labs program, and if they do, they have to sign that we can use their anonymized data. They did that very early on for a pretty simple feature, which is like we do call analysis. So for this call analysis, we used a huge LLM like GPT for a long, time. And we had all this data, all the data coming in and coming out of the LLM. So we thought, why do we need that huge LLM with a huge prompt that tells it what to do? we can just fine tune the LLM to spit out just the data we need. And so we were able to minimize the compute, the required compute for getting the same, if not even better results. And that was because we had the data that came into the huge LLMs before. And as a side effect, we can host that LLM ourselves, which makes the data even more secure than it was before. That is one thing we Sebastian: Interesting. Stefan Lange-Hegermann: were able to do. Another thing I am thinking about right now is we have a lot of data from real calls between humans and LLMs are usually trained on things like Wikipedia, books and stuff. So LLMs don't know how people talk on the phone. Best case, they know how people are supposed to talk on the phone in movies and books. And I think we can improve André Neubauer: Yeah, very different on phones. Stefan Lange-Hegermann: on how a a computer and LLM talks to people on the phone in a way that makes them feel more natural and more like a human. Sebastian: Interesting. And that does this also n so I I I assume that this is not only about the content but also about the pretty much text to speech. So the way how yeah the tonality changes, pauses and everything. Stefan Lange-Hegermann: We usually work with partners for that because there are lot larger data protection hurdles for voice audio data than there is for text data. So that is something we are thinking about how to do it, but we don't want to do anything wrong and collect data we are not supposed to collect. Sebastian: For sure. Stefan Lange-Hegermann: simple things like anonymizing that data. is actually, given the technology today, we can train a model on the voice of the person, then we can replace personal information in that call with the same voice with other information. Then we could train on that data, but that gives a big margin for errors. So these things are complicated. Sebastian: For sure. André Neubauer: Follow-up question, are you then also really training models or to what degree are you using AI? Is it training, it fine-tuning, is it just inference? Stefan Lange-Hegermann: Well, we are experimenting with training, but right now I don't believe that training will make what we do better. I think fine-tuning André Neubauer: Okay. Stefan Lange-Hegermann: is the way for us to go, because the training can be very expensive, and the base models are actually pretty good, especially the small ones. And you can do a lot with just fine-tuning them. Sebastian: Yeah. Awesome. Thanks a lot. So it was a little bit a longer intro than expected, but anyhow, it was interesting. that's why we asked these follow-up questions. And before we get to the main section, right? The meet the the LLM sovereignty, maybe a quick question regarding your how you are currently working, the first segment in our show, right? So what what is your tech stack? How how you utilize A AI? Stefan Lange-Hegermann: Yeah, I think I use most LLMs every day, but I use Claude the most. And as I come from a development background, I usually use Claude code for everything. also for like writing documents, preparing meetings, stuff like that, I usually use Claude code. We have an internal agent called Aeronux, which is able to... access to a certain extent internal systems to get some data. I use that a lot to get statistics, things like that from our systems. But my main workhorse is just the Claude code command line tool. And I have some projects. Most of them are documented in Markdown. I use Obsidian to have a better view of what's going on. yeah, Cloud Code is doing a lot of the heavy lifting, actually. I try to get transcripts of all the meetings I'm in, and I try to get my stack to be clever about putting these into perspective and putting them into the right folders. But inside the company, we work like we throw over a lot of... Claude artifacts to other people and work together on those things I think the workflow went from from a lot of Text files from a lot of Google Docs to a lot of HTML Most André Neubauer: Hmm. Stefan Lange-Hegermann: things have these Most things most most artifacts we share in the company look like Claude code build artifacts now. I think you know what I mean and André Neubauer: Yes, yeah. Stefan Lange-Hegermann: Yeah, use a lot of, as my projects are usually not like projects in Claude because I have folders for everything. But my biggest thing is having one large folder with all the strategy and so on. Yeah, I mean, know, Markus Andrezak and he once told us strategy is infrastructure now. And I think that is one of the main points we are looking after because Only context makes the results good. And most of the time, the human is there to give context, but we are happy when the LLM can get as much context as possible from structured data. So that's what we try to do. And then I have some automated agents running. They are all running on our Aeronauts system. It's that internal AI that schedules crons. Usually those things are not AI. In the beginning, everyone was like, yeah, we need to build AI systems. And now it's more like build a Python script that fetches the data. Because especially with these daily reports, it's always the same. There's nothing an AI could bring that we can't do with Python or stuff like that. Usually these reports are not very intelligent, but built by AI. I think that's one thing where a lot of this is going. mean, you build a dashboard with AI, but you don't leave the interpretation to AI in many, ways. Although we sometimes try to have AI find new facts, new information in the data, new insights, André Neubauer: insights. Stefan Lange-Hegermann: but... With existing products that have been running for 10 years, finding new insights for an AI is not that easy. Yeah, what we do a lot is actually we have like we André Neubauer: Yeah, I can imagine. Out of curiosity. Yeah. Stefan Lange-Hegermann: have phone companies, so we have a lot of fraud and frauders are getting more André Neubauer: How come? Stefan Lange-Hegermann: clever every day. So they know how they can structure their stuff. They build actually build websites that look like a company website just for us André Neubauer: Yeah. Stefan Lange-Hegermann: when we check if they really exist and stuff like that. works very good with AI. So AI is very good in recognizing frauds and stuff like that. We like that a lot. I think that's many hours per day that we can pass on to an LLM. André Neubauer: You mentioned Aeronux. Is that a custom thing you're using to give bit of harness to the entire organization or how? Stefan Lange-Hegermann: Yes. I would say it's the open claw of Zipgate. It doesn't have that much ways to reach out and fetch stuff because we have to protect the data, but still it's a very useful tool. It has access to some of our insights, and it can assemble them and help me make sense if I have a new question that never came up. If I have an old question, it usually just finds the data in artifacts. Sebastian: Cool. Thanks a lot. Now I think it's time for the meet, right? For the main topic of today. And you already hinted that you are running some stuff yourself, right? Already. So some some smaller models. But I think recently there there was a LinkedIn post from you, right, where you described I think an experiment that you ran. Stefan Lange-Hegermann: Yes. Sebastian: Maybe that's a good starting point. Stefan Lange-Hegermann: OK, that's very exciting, actually. Well, we've been thinking about, is there a reason to buy hardware for a while? it was always like we weren't really sure. We had companies make us offers, but we were not sure if we were going to do it. But then the Claude situation happened, where the newest Claude version was like, kept from the customers, not just in Europe, but officially just from people from outside the US. So we got into thinking, we were pretty quick. We were sure that we needed something to mitigate for a situation where maybe even older models will be locked for European companies. So what we did is we rented. pretty big machine, Blackwell Architecture B300 machines from Finland, and installed at that time, the biggest coding model available as open weights, which was GLM 5.2. And we just reached out to all of our developers and told them for a week, use that. Stop using cloud code. think. About 90 % of our developers are using Cloud Code. Some are using Codecs. Some are using local models for smaller stuff. But most are using Cloud Code. And we told them, try this. they didn't have a harness. Some even tried to use it inside the Cloud Code command line app. Some used PyAgents, and some used OpenCode. And all of a sudden, we had a lot of traffic on that one machine that was running the GLM model. At one point, I saw about 80 sessions at once. So that's actually 80 requests to the LLM running at once. And the machine didn't even blink an eye. The verdict of most developers was, this is so much faster than Claude Code, which was impressive, although the machine was kind of expensive too. 7,000 euros a week is something. It doesn't sound that much when you go weekly, but it becomes much more if you count monthly. So that was actually a very big success. We had this huge machine for a week, and we actually stopped after that week to do that. Our developers, most of them are back to Cloud Code at the moment. They can still use the same access data to get a GLM model, which is then hosted. on the service of one of the companies that sells them by token use. But right André Neubauer: Hmm. Stefan Lange-Hegermann: now, we are working on getting the hardware in-house and installing these models, which will probably not be GLM, but Kimi, which is much larger. We are a little afraid that it doesn't fit on these brand new machines. But I heard it should work. And we are going to put them in our own data center. And then our developers will probably all work on that. We will not tell them never to use anything else, because we believe that experimenting, trying new stuff is very, important. But for us at the moment, especially with coding, it's not just about data protection. It's not even a very important thing, because we have our code on GitHub. That's already there. So the most important thing for us is sovereignty, access to our own hardware, access to models we want to use, and not being cut off of the action when we really need it. André Neubauer: Adjusted the mass, you say it's 7,000 euro on a weekly base, like 50 weeks, like 350,000 euro. It's a lot, right? Don't get me wrong. But if you do also the math on, let's say, 100 Claude-Courts subscriptions, and then some over usage, you get in that direction, right? So it's not completely off. You could maybe also... save some money, you said like it was not fully utilized, that machine. So I think you may end up on par, you could end up on par. Stefan Lange-Hegermann: Let's say for a company like us, it's more like seven figures we would pay for Entropic a year. André Neubauer: Yeah, so then even better, right? So the use case is not on saving money or like spending more actually. Even that case is positive. Yeah, it's a massive cost. absolutely. Sebastian: It's a massive cost saver. Yeah, yeah. Stefan Lange-Hegermann: Well, yes. Everything is like, there is almost nothing that gets worse, actually, which is not something we expected. mean, I was like, my workflow when I write code, when I vibe code, I mean, I'm not a developer anymore, so I still know what's happening there. I usually just take screenshots, give them to Cloud Code, and tell it, yeah, here, that looks bad. Do something about it. But we figured that most of our developers don't do that usually. They are more text-based. Most of them just speak into their microphones and tell Claude what to do. So GLM doesn't have vision. So that was like my biggest fear that they would not be able to follow with their workflows anymore. But most developers don't have that workflow actually, I found out. So most just talk into the microphones and have the LLM do what they do. So there's really not much that got worse. I think most developers were pretty sure that the quality is almost on par, more like Opus André Neubauer: Hmm. Stefan Lange-Hegermann: 4.6 maybe, but still pretty good. And everything we heard about Kimi, and I already tried Kimi on the Moonshot AI service, it gets even better. So we're really excited about it. André Neubauer: Yeah, if you have a good harness in place for software engineering, which allows you to replace models, think that should be then also a win-win-win situation. Other than that, I would question quality. But as you just mentioned, even there, was on par. Stefan Lange-Hegermann: Yeah, I mean, it's hard to measure software quality on a very quantitative level, but as far as we could see, quality was very good. And in some cases, developers said the quality was even better than what they would have expected from Claude. Sebastian: That's interesting. because I would have assumed that the engineers would be very skeptical towards the model, which I'm not sure if this is true in your case, but if they are skeptical, then hearing generally quality is on par or even better. Then then I mean that's th this would be the biggest blocker from my side. Like if you don't run proper evals where you check if in your cases the model works better or worse or whatever, right? Then the the user's perspective is the most important thing. That's pretty impressive, to be very no honest. Yeah. Stefan Lange-Hegermann: I didn't expect that, actually. expect our developers are skeptical towards things like that. They're usually not skeptical towards something new because they always have the freedom to try something new. That was like a big thing because we had developers who had been using Py agents for a while with even local models on their machines. So the tooling, getting the tooling right for everyone was kind of easy because they were able to talk to their peers and they could help them. But there was always an openness towards new technology. in that case, mean, there have been so many promises made about models that are as good as something. I mean, I always say if you go to a trade fair and you talk to people, everyone's going to tell you, we have our own models. I'm pretty sure I have yet to find someone who really has their own models. There's so many lies that I expected this to not work out that good, but it did. It was really awesome. And some people really even come to me and say, when can we use that again? Because I liked it so much. André Neubauer: Is there a specific reason why you stopped that experiment? Because, well, you mentioned that you spent way more, so it's comparable cheap. And then you decided to buy that stuff. So is there a specific reason why you went that path? Stefan Lange-Hegermann: That's a good question. Well, actually, yes, André Neubauer: I give my best, I give my best. Stefan Lange-Hegermann: we limited the experiment for a week, and we just wanted to see what's going to happen in that week. André Neubauer: Yeah, sure. Stefan Lange-Hegermann: I think one of the reasons is we didn't have the best prices we could get for the machines, because with other things we ran from them, we don't do that ad hoc, which we did in that case. André Neubauer: Hmm. Yep. Okay. Stefan Lange-Hegermann: And I think we got lucky because they accidentally blocked the machines for us because they offered us a long-term contract and said, we're going to block them for you. And I think they accidentally did that for a longer time than they wanted to. Yeah, and actually, we want to get the harness right. Having everyone on Claude Code is fine because everyone's going to use it somehow in a similar way. And we want to make sure that we have like a harness for the open models that works for the whole company so that we get consistent results. And I don't want to handle situations where maybe a team or several teams have worse results than others. And we just want to make sure that we have like a consistent harness, a consistent technology in the background. So yeah, that we don't have to like handle edge cases that wouldn't appear otherwise. Sebastian: Yeah, that's what what you mentioned, right? That some folks didn't actually have harnesses before because they were using cloud code, so they needed to switch to something new. But honestly, so I'm I'm a big fan of Pi. I've been saying this on this podcast very often. And I tried different kinds of models, including actually, and we we ran a similar experiment on a smaller machine. We at North have a DGX with like A100 so eight A100 GPUs. That's only I think 40 gigabytes per GPU or something. but we deployed the Quen 3.635 billion parameter model with like which is a mixture of experts, right? Always three billion active and did run a similar experiment. So LMperv, 16 users and the throughput like Was just going up. So for the test duration, we didn't saturate these four that we only used four GPUs, we didn't saturate them. And honestly, also the quality of even the small model was quite decent. So compared to probably anything, a year ago, then this was already quite a lot better, including reasoning, right? And that's given the fact that I'm using a harness that has a lot of customization that. that has like skills that has the the code base, right? There's a lot of grounding for the LLMN. And if it does then have thinking capabilities, then that that is a pretty mighty solution. Right now I'm I'm working with Ornit orn or Ornith, I think that's that's something I'm I'm testing on my local machine, which is the optimized coding model based on on the Alibaba Quinn 3.6 model and I I have to admit that yes, this this is something that in case the frontier laps like are not available, then that is a decent solution that doesn't let me drop back to zero, but rather to, I don't know, ninety percent or ninety-five percent, right? So yeah. Stefan Lange-Hegermann: Yeah, think one of the things we really have to build into the harness we build now for the open models is better decisions about what models to use for what use case. I that is a hot topic right now in the scene. And I guess we are going to have to do that too, because sometimes even a fine-tuned model for a specific case might even be better to solve a problem than than a large model is, even for coding. And so, yeah, we'll have to look into that. That is also one of the things we're definitely going to look at. On my desk at work, I have one of these Studio Max M8 Ultra biggest box they ever made things. And that's a lot of fun to play around with the models on that machine, actually. And it does even run André Neubauer: Yeah, this is- Stefan Lange-Hegermann: GLM. because it has 512 gigabytes of RAM, but it's very slow. André Neubauer: Okay, not that bad, Sebastian: Impressive. André Neubauer: not that Sebastian: Yes, yes. André Neubauer: bad. That would be my follow-up question. the hardware you're just buying, is for coding. First question, and the second is, have you looked into local hardware? So local hardware, mean like works, workplace, workspace, workspace hardware. You just mentioned MacStudio. So I'm wondering... Like why this could be a pass too. Stefan Lange-Hegermann: believe it can be a pass, but right now, think for what we're doing with understanding large code bases, and I think it's good for us to have at least access to a large model too. André Neubauer: Hmm. Stefan Lange-Hegermann: Right now, I'm not sure if the models are getting smaller or larger. When we made the buying decision for the machines, One of our founders said, yeah, I don't know, maybe it's too big. It's all getting smaller. We don't need those big machines. And after Kimi came out, he came to the office and talked to our CTO and told him, well, we need larger boxes because the models are getting larger. So we don't know if the models are getting larger. I mean, there's a lot of research. You can install 38B models on iPhones now with one bit quantization. work. So maybe it's all getting smaller, maybe it's all getting bigger. But one thing that at least the Studio Mac has is it's always kind of slow compared to other enterprise-grade André Neubauer: Yeah. Stefan Lange-Hegermann: hardware. mean, even compared to consumer graphics cards, it's kind of slow, although those are so expensive you can buy them. So I don't know. We didn't play with these 5,000 euros machines, AI machines that everyone's building right now. We didn't try those, but I think for us right now, a centralized solution is better. We didn't specifically buy the hardware for coding actually, or decide to buy it for coding. We thought a lot of our other models can run on that hardware too for our customer facing products. run our own text-to-speech and speech-to-text models and we our own. Okay, I'm also talking about our own models. They are not our own. Other people's models on our own hardware. And also have an LLM that we can use for that. So it's a smaller one though. so there is an aspect of using it for our customer facing products, but we are also thinking about buying just A little bit cheaper hardware for that, so I'm sorry. Sorry. Buying cheaper hardware for that. So yeah, I don't know. mean, thing that makes calculating the cost efficiency the easiest was coding. So making the decision based on André Neubauer: Yep, true. True. Stefan Lange-Hegermann: money was the easiest to just say coding is much cheaper. Sebastian: Absolutely. And and I also think that there are differences, right? So on the one hand, if you wanna replace the infrastructure for eighty or a hundred engineers, then that's a different thing than having something for one person to do some w use cases, right? To to to utilize some use cases. And and this is where I would also differentiate, given specifically also given the fact that you usually for coding you have huge context that that you always need to like transfer, right? So you always have huge context windows that you need to have and then like the model itself plus the the context size, all of that needs to be readily available and also they're quite like these complex coding agent, like there's constant turns between the the agent basically and the the server, the GPU server, right? So they they are quite compute heavy and and extensive. That's a totally different ball game to a chat assistant or a a simple rag or something like that, right? Stefan Lange-Hegermann: Absolutely. Yeah. André Neubauer: But that actually then translate into if you're a larger engineer organization or mid to large engineer organization, you should always go for central infrastructure. True? Stefan Lange-Hegermann: You Sebastian: For now until André Neubauer: The way you described it? Yeah, yeah, for now, for now. Like, only Stefan Lange-Hegermann: just today. André Neubauer: for now. We are not in the section where we do any kind of... Come on. predictions. Stefan Lange-Hegermann: Yeah, exactly. think, well, talking about predictions, I believe that the American market is not going to allow open weight models from China in anymore, like maybe three months. I just hope that the EU doesn't just follow blindly and that we can still use open weight models from China in a few months. So if that is the case and they develop in the pace that they do right now, I believe that we can still say that a few months from now, that it is more cost effective to have your own hardware with your own models. André Neubauer: And you would also say like you go full sovereign, not just hosting your own infrastructure, but also using EU based models. So have you also looked into what Mistral is providing? Will that be a case or what is your strategy in that regard? Stefan Lange-Hegermann: We have actually in the product you can today you can choose between using an American model from OpenAI and a Mistral model. Although I don't know how the sovereignty of Mistral models holds up right now, it's still something that our customers want. I just read that just I think a one figure percentage of companies is just left who are not thinking about going sovereign with their software as a service products they are using. So I think that is a huge topic and it's also a huge, I mean, historically we didn't have like large enterprise customers. just, most of our customers are kind of small, small medium sized companies. Also, they are thinking about it, many of them. We have a lot of lawyers, we have a lot of people from the medicine fields of work, so we have to be sure that our data is safe in a way. So that is something we definitely have to offer. And as a company, as a whole, we want to make sure that no one accidentally falls into that trap. I would say... To say it carefully, many people from professions where they would have a very big incentive to have good data protection are not very careful with their data. So for us, it's very important to give them defaults that protect them. And so I expect European hosted or even hosted on our servers models to be the default by the end of the year for everything. Sebastian: That's a word. Yeah. But I can relate to that. I I sense also like a strong push in this direction or momentum in this direction. And yes, of course, the Fable 5 incident was a huge booster. And giving at the same time these Chinese models being on par or partially even better, looking at Gimmick 3, right, which is currently I think leading the LM arena when it comes to coding. Yeah, so why not? yeah, absolutely. Stefan Lange-Hegermann: I think it's totally exciting right now. Getting the models home and controlling the harness gives us so many really exciting ways to work with LLMs and to offer them inside the company to our customers that I'm really excited about the future regarding that. if we can keep pace. So many software as a service products, which I heard are getting worse at the moment because people are just web coding stuff in without proper harness and without thinking about what's going to happen. think software as a service is getting so much better in the future and so much faster development cycles. And that's really exciting to me as someone who came from the dot com era. Sebastian: That that's a very positive outlook, yeah. one one final word also on the on the model, like bigger versus smaller discussion, right? Something I haven't mentioned managed yet is try out this I think one B mini CPM model that has been fine-tuned using Claude Fable five traces, which like has been at least advertised as your mini Fable Five on your own machine. So I I try to to test models like this, because if this six hundred fifty seven megabyte model would actually be on on a on a similar quality level, then then this would really flip the big versus small model discussion quite a bit. Stefan Lange-Hegermann: Absolutely. I always like what's going in my mind when I think about these is more like this. have stuff at home. We have hamburgers at home and you get this German bread with just a piece of meat in between. right now I was always a little bit disappointed with smaller distilled models. But I think the way, way more interesting thing is like specialized models for for certain things, like a specialized model for unit testing, a specialized model for, I don't know, user experience design. Those things might be very cool for the future, I think. And they can distill. mean, you can distill Fable skills in unit testing into a model. And I believe a smaller model is going to be much better than a smaller model where you try to cram all of Fable 5 into. Sebastian: For sure, for sure. Yeah, for me it's a little bit the the the question or nah what what happens when you distill a model, right? You you take part of the knowledge of the model and transfer it to different model. It's also alluding to what you just said, take the user experience testing part or the the unit testing part, whatever, right? And just put only the knowledge that that is needed for this, right? the question is really how dense. is the the knowledge in these huge models in reality so or like if you reverse it, isn't there a big chunk of knowledge that is actually never needed, like 90%, 95%, maybe 99%, even because it's essentially all the knowledge of of humanity, if you wish, right? That that you have in in these models. And and then wouldn't it actually be possible to distill 1%, 5%? 10% size model that contains most relevant knowledge for like 99% of all use cases that you can think of, right? This this is the the other discussion. But anyhow, this is a very theoretical discussion. And then you would have to have a very concrete idea of the different knowledge areas, right? To pretty much generate the learning path for the distilled model so that you are building up on on the like very basic knowledge that is there in the beginning after a basic training, right? In order to then have it learn specialized yeah knowledge areas if you wish. But but I'm I'm pretty sure that this is an area where there was there there will also definitely happen a lot over the course of the next six to twelve months and we'll see many smaller models perform at at great quality levels. Like for me the eye opener was really the Quinn 3.6 thirty five B model, which in my like tests I didn't see a big difference to to the huge I don't know how big actually GBT five point five is but but anyhow, right? So I didn't see a huge difference. Stefan Lange-Hegermann: If that model was just specialized on coding and not on word knowledge, that would probably even become better. So although for being a good developer, you always have to like know a lot of stuff around that is not development. And that is... Sebastian: Yes. Yes, that's key. Exactly. That I I just wanted to point that out, yeah. Stefan Lange-Hegermann: And back in the day when we had visitors, especially press in the office, we still have a lot of art in the office. And our CEO always said, how is someone who doesn't know good art supposed to make good user interfaces? I mean, I don't understand LLMs well enough to know if that has an influence or doesn't. But I still believe it should have. Sebastian: Yeah. Well, thank you. I think we're like closing in on the end of this episode, right? So we've discussed extensively the topic of sovereign LMs and really interesting insights from your side. We have two smaller segments towards the end of the episode always. And what is the what one is the reality check? So what are wow moments or even maybe what the fuck moments where you still believe so wow, we we are doing all of this stuff, but This is not working. Why is that? So do you have these kinds of moments? Stefan Lange-Hegermann: Yeah, I'm going to go back, like back to when LLMs were very, very new and we were just finding out what they were doing. And at that point I was thinking about some experiment I could do. And I was like, what if the LLM could tell people something about that and they would believe the LLM knows it without knowing it. So I researched how do these... Well, I don't even know the English word, but on the carnival, the people who tell you your future, how do they do it? And there's some techniques to do it. So I put this technique in the prompt and I used, back then models were not, had no vision. There was no vision model at all. So I took one of those models that texts photos with what's in them. That was a little bit more detailed. It could tell you there's a person with a beard and glasses. So I used that and then I took a picture of people. and put that into the LLM with the technique the people are using to tell people their fortune. fortune tellers, no, there we are. Sebastian: Fortunately, yeah André Neubauer: Fortune teller. Stefan Lange-Hegermann: Fortune tellers tell their fortune. And then I had that output to the people. So it was like a kiosk. You could go up to it and then it would detect there's a person and it would take a picture and put that into the LLM. And some people were really sure that the model knew something about them that it didn't know. Sebastian: Awesome. Stefan Lange-Hegermann: It is always the last thing it said was your future is going to be bright or something like that. So it made a prediction. And there was one person in the company where it said, your future is not good looking good. And I was like, yeah, that is just this random and it doesn't do that. Often you're the, the first person where that happened. And the same person was standing in front of the computer for a second time and it said it again. André Neubauer: Wow. Sebastian: I hope it didn't wa it it it it w wasn't really a fortune teller at that moment. Stefan Lange-Hegermann: So no, I don't think it was, but it was like, well, he was a little concerned after that. And yeah, that was like the first huge wow moment where I said, wow, this technology is like crazy and things are going to come out of it. I think it's also a metaphor for how we view LLMs right now and what we... think they can do what they maybe can't do. Because in that case, it was very sure that it didn't know things about people. Sebastian: Yes. Yes. But but really that's a very good one. Very good one. Love it. Stefan Lange-Hegermann: Yeah, that was like the biggest wow moment. And actually I don't have like moments where I would say, I think I have a very good grasp on what they can do and what they can't do. Sebastian: Mm. Mm. Stefan Lange-Hegermann: So I usually don't get disappointed when something doesn't work. Sebastian: It's more like, yeah, of course. Right. So you you you try it out thinking maybe it could work, but assuming it won't work, right? And then it doesn't. Yeah. Stefan Lange-Hegermann: Yeah. And it doesn't work. that's okay. mean, one thing when you're in a real time voice, that is sometimes a little bit of a lackluster is like when, when models are not able to be fast enough to do what you want to do. I mean, voice has to be quick and we're working on like technologies to make it even quicker. But then like you have a hundred trials and everything works and then you present it to someone. And then there's like a network. problem or anything, and it takes like five seconds to answer. And then the founder of the company comes to you and tells you, so when are you going to build the real product that we're going to ship to customers? And you're like, well, that is exactly the product we're shipping to customers. That may be that. That may be things that are unexpected that don't depend on the LLM per se are the ones that are the ones where I sometimes think, I'm disappointed. I was expecting much more. Sebastian: Yes, can relate to that. Thanks a lot. And then final section, do you have a prediction for our listeners? Stefan Lange-Hegermann: Well, I could like take a picture and let the LLM do it, but... Sebastian: Yeah. André Neubauer: Ha Stefan Lange-Hegermann: Well, believe, well, one thing today, I talked to our CEO and we were both like, I don't think we know what we don't. I think we don't know how we're going to work next year at all. And we have no idea. although we are trying to find out how we're going to work next year, we still have no idea. And so I believe that we don't have an idea. But I believe that it's going to be much more collaborative work with AI than it is today. I think that we're going to find out what makes us human. So what are the things we can do better than AI and how we can do them better. And I assume that we are going to even get better at these things because we know that is what distinguishes us. I believe that knowing ourselves better and knowing how to use AI to amplify those things that we can do, we are going to have very cool products and we are going to have, in some cases, things that help us make our lives easier and also our lives so much harder. I'm always coming back to the dot com bubble. Back then we were all enthusiastic. There was nothing negative about all of this. along came social media. so, yeah, we don't know what's going to happen, but I believe we have a chance to find something that really helps us get better. And I really believe that. And I hope it's going to be that way and not the other way around. and I do not believe that we have AGI in a year. Sebastian: That that yeah, that's a that's an easy one, I would say. But yeah, no, no, no. Stefan Lange-Hegermann: Hahaha Sebastian: thanks a lot for that. Yeah, awesome. thanks for really for sharing your insights and really enjoyed the the episode. yeah. Thank you. Stefan Lange-Hegermann: Thank you. Thank you for having me. André Neubauer: Thanks. Sebastian: yes. André Neubauer: Good. Now to the recap. What stuck with us, Sebastian? Sebastian: Yeah, for me, the impressive thing was number one, I looked it up actually that you actually need eight B three hundreds to run GLM five point two, which I yeah, had just didn't know. But then what was even more impressive that with this cluster they were able to serve eighty engineers or probably more faster than clawed code that was Really impressive. I wouldn't have expected this. André Neubauer: Yeah, maybe what could be a reason for that, that it's the Claude harness, So Claude quote harness, which is maybe holding back here a bit, but not sure about that. Not sure, but it's fascinating. As you said, I think they probably have also planned that infrastructure. maybe also for more people, even though you're right, that is actually the entry barrier, right? This is crazy. And actually my first learning takeaway is in that same field, practice always beats theory. So GLM 5.2 is on par with US models for most use cases, which is, I think, a nice takeaway. because I think we all tend to always grab in the highest box. And Tropic is releasing Fable. Everyone is going to use Fable. I think for most of the use cases, well, it's not even like you can also take a lower box, but you can actually also use other models in this regard, China models. And my second takeaway is, well, it's actually going back to the hardware topic. So central hardware versus workspace hardware. A lot of people are talking about the next Max Studio and M5 or M5 Pro Ultra. don't know. So the idea of running modeds locally. not on your own hardware, but really on your own desk, so to say. Not sure after this talk today. I think if you're at a certain scale, at least budget-wise, you could also afford central hardware and then go for really frontier models rather than having to deal with well, this still very compressed models. just to make it work on limited hardware. Sebastian: Yes. Yeah, that that's really the tricky thing in there, right? I think a DGX Spark with 128 gigabytes of RAM is great for like running a to Gwen 3.6 with 35 billion parameters or even bigger models that that should totally work. Also quite okay-ish, but if you want to run these huge models, then you'd need something bigger. And yeah, what what I didn't mention, obviously, you need the eight B three hundreds for running GLM five point two in full precision. So if you have like them quantized, then then you you don't need this big size of of machines. That's why he could also run it on his desktop, but then slower. but the the general question that also pretty much or the the interesting topic is really like yeah, the R models getting actually bigger or smaller? So do do you always need the biggest models? I mean Kimi K3 now is a huge model also way bigger than GLM five point two I think and it beats Kimi sorry it beats GLM five point two and it beats even Fable five in benchmarks. The question is really compared to to some other smaller models where's the where's the sp sweet spot where's the trade-off right and that that is also super interesting and makes it so so exciting. yeah it's another it's another dimension of change that just happens. And we we don't know. I I don't know if so I'm I'm pretty sure that there will be advancements in model distillation that make s models work on phones, on laptops that are reasonably fast. and also quite capable for a lot of use cases. And the question is just like what's the the use case at hand, right? there are a lot of coding workflows actually that are established that that you can use for for different harnesses that make use of different model types already. So you util usually you utilize a very big model of the frontier model for planning and then you utilize smaller models for implementation or for like code scouting or code reviews or something like that, right? And this seems to be working quite well. And that's a hint that yeah, differentiation is probably the key here. And one one thing, sorry, I'm I'm now really getting into a talkative mode. I I just yesterday saw André Neubauer: Shoot. Sebastian: a presentation of LM Studio Bionic, I think is the name. That's a great product for conscious people who Want run their own LLMs, but not for every case. So it it it is a tool that you can run on your computer. It's pretty much like a cloud code or a codex. So you can upload documents, you can use text-to-spee with all locally on your own model served by LM Studio and can run, use it similarly. You can build agents that do stuff, but you can also say For this specific use case, I want to use a cloud-based model. Obviously, then served by the LM Studio Cloud because okay, they need to make money somehow, right? But it's a great like fully fledged solution for people who are conscious running local AI. So maybe that's a hint for everyone who's interested in in local AI. LM Studio Bionics seems to be a pretty interesting tool. I haven't tested it yet, but I definitely want to test it. André Neubauer: Yeah, it's maybe a good activity for the weekend if weather is bad. Maybe one thing I just realized and we also haven't covered that is there's a huge advantage if you run your own models. doesn't matter whether it's a central or like workspace hardware. Like no question on the data you're putting in, right? I think that's a huge advantage. So Sebastian: Yes. Yes. André Neubauer: definitely something where it also makes sense to look into LLM Studio. Sebastian: Yes. And with that, I think that's it for today, right? Thank you and bye bye.