Latest / The Bleeding Edge Podcast / Hugging face attack, what you need to know.
Transcript
- Ralph: Hi, and welcome to the Bleeding Edge Podcast. Emile Are are you are you ready for this one? Emile: Always Ralph Ralph: it's a big one, I think. and you know two of my most favorite alternative AI companies is Thinking Machines Lab and Sakana AI. so Thinking Machines is Mira Murati's outfit. So obviously she was CTO of OpenAI when Open AI really started to become part of the regular news cycle. She also Briefly took over from Sam Altman as CEO when he was ousted. I think she was CEO for all of five days. anyway, she left OpenAI after all that debacle and founded Thinking Machines Lab in February 2025. And so she is surrounded by a group of senior Open AI departures. And I announced a few weeks ago on The episodes that she has actually made an open weights model called Inkling available that was launched the 15th of July 2026. So it's just a month and a half old. and it's not open source, but it is open weights and it's free. and obviously there's a big story around this because she's basically betting that. Every company is going to want a model that they essentially control, just in terms of the training of the model. and that training means that you have something completely optimised to your business. But it does mean that you are still building a reasonably large large language model, whereas Sikana AI are betting on something completely different. Sakana AI is actually co-founded by one of the eight authors of the original Transformer paper that was published in 2017. that really started this whole revolution, if you like, around this transformer technology that is the foundation of the current LLM space. that paper was Attention is all you need. And he's basically spent the last three years arguing that. One giant model is not the future at all. So Miram Marathi has created not a giant giant model, but a reasonably giant model with 975 billion parameters. if you think about Kimi 3, that's sitting at the last time I looked was 2.8 trillion parameters, so nearly three times bigger than the model that Thinking Machines Labs is is running. but essentially you know we've kind of got these two arguments. So you know Sakana is looking at multiple smaller LLMs and agentic systems, whereas Thinking Machines is betting on one large model, but one that you can train. So very different from the Frontier labs that are essentially building the AI for you, and you're obviously accessing it via API subscription or just via a monthly subscription, you know, with a fixed cost. But Thinking Machines Labs is basically saying that's not the right model. You'll want to train your own. Sakana is saying that's not the right model either. You'll want a mixture of much smaller LLMs. and so this is essentially two very different directions compared to what the Frontier Labs are taking. And I I mean I don't actually know which way it's gonna go, but if you think about what Inkling actually is, so this is the the model from Thinking Machines Lab. It's a mixture of experts transformer and it's running 975 billion parameters. but you know at whenever you run it, it's about 41 billion active tokens at any point in time. And it's actually running something like 256 rooted experts with six selected per token. so there's basically six running at any one time that it selects. And they have a little bit of a different approach with how they've actually built the model. I won't go into some of the terminology, it's very technical. but you know, they've really come up with something that is capable of doing multimodal processing. So it can do text, images, video, audio, and it's also got a context window of up to one million tokens pre-trained on 45 trillion tokens. So what that means is they've trained it on a lot of text and images and video. And that's basically part of the base model that you then train on top of. And so they give that away But what you pay for is a technology called Tinker, and that is the fine-tuning platform. So all of this I've covered in the past as well. But what I started to look at was okay, you know, what would it cost you to train one of these models? And actually I've run some simulations and you can basically fine tune this inkling model with about 50 million training tokens at a cost of twenty dollars. so that seems really accessible actually. I'm Emile: Yeah. Ralph: not entirely sure exactly how far you're gonna get with fifty million training tokens, just in terms of like, you know, enterprise scale data. But you know, even if you need let's just say you know something like a billion tokens for an enterprise, that is still a a cost that you can afford. And so definitely something worth playing with. obviously the fact here is that you actually have to host Inkling somewhere. This isn't something that you can run just on your kind of home computer, so you probably will be taking some data center capacity. but I think you know that's still reasonably affordable. And obviously what you've got is a closed model that is processing your data, it's not going out. into the Frontier Labs LLMs. you're using your own data for your own training. It's only accessible to you. And your model is essentially getting more and more sophisticated around your particular use case. So I think that is, you know, near is quite exciting. So one of the quotes is the training run is nearly free and the expensive part is actually the evaluation of the data. So you need to make sure that you're getting sense out of it and that Obviously does take quite a bit of time and potentially you know quite high failure rate as well. Bridgewater is their showcase example. So Bridgewater actually used Tinker to train Quen, Quen3, which is Alibaba's model. I'm not quite sure why they published a model and you know using Quen, but the this was their use case on Tinker. obviously they have their their own model as well and and that's called inkling but they've ch chosen their kind of showcase piece with this bridgewater use case and they've published y you know essentially how tinker has been used to train coin three and that particular model is two hundred and thirty five billion parameters so Interesting to see that is you know you have a smaller model than Inkling. so in terms of you know the success that people have had, I think it's a bit mixed. So the first benchmarks, they've got basically looked at GLM five point two and they found that Inkling is stronger stronger on instruction following and factual recall, but meaningfully weaker on reasoning and software engineering and agentic terminal work. and so I think you know you just have to really kind of look at exactly what you want to get out of these models and optimize for that. So probably a mixture of agentic and LLM capability is still required. the architecture is kind of un unvalidated in public, so I think there's more to to test and learn from as enterprise starts implementing this. and then You I think the third thing is you know you you you can't run it. So serving the full model yourself basically takes two terabytes of GPU memory. I'm not sure that you're anywhere close to that on your local machine, Emile. I know I'm not. I'd love to have it to be honest. I might need a bigger office. but you know you're free to download it, and then you're obviously free to rent hardware to use it, and no doubt you will be putting it in a data center. near you. So so some interesting stuff there just in terms of the model. Sakana, just to kind of round out these the the difference between the two they've basically made the opposite argument and basically they're saying that you know you you you really want to use a a mixture of much smaller models as I said so and and and they've been putting this to good use. So they launched their AI scientist. and basically, you know, the AI scientist is designed to run a complete research cycle autonomously. and they found that so through an independent evaluation, for about six to fifteen dollars and about three and a half hours of human involvement, they were able to generate reports and papers. That were at a quality that the authors described as comparable to a rushed, unmotivated undergraduate. So not exactly Emile: Ha ha ha. Ralph: an inspiring you know AI scientist. but what they found was about 42% of experiments failed on coding errors. but nonetheless, you know, this is obviously the the kind of the use case that Sakano is after. They've been one of the first to release. Their AI scientist model. and then the other model that they've also released is Marlin. Marlon is supposed to be your virtual CSO, so that's your chief strategy officer. And its kind of claim to fame is that it runs up to eight hours of continuous autonomous reasoning and it produces around 100-page reports designed to set out your company strategy. So this is one that's kind of close to my heart because I've actually spent a lot of time architecting and engineering extensive research, multi-stage research protocols to do this very thing, and I've had some really good results. but there's still no independent evaluation of Marlin's quality of output as of late August. and they are really kind of targeting this at a variety of consultancies. And there is you know some interesting kind of quotes out there that it's definitely able to deliver certainly exceeding expectations. And one of the quotes from a a Tokyo consultant was it discovered angles we hadn't even imagined, which is amazing because obviously Finding gaps and opportunities in the market is what strategy is all about. So we will definitely kind of keep an eye on that one. So essentially the differences are thinking machines is one enormous model that is owned and specialized. Sakana AI is many small models, each specialized and processing different parts to kind of give you an answer. in terms of thinking machines, what you actually buy is customization. So you you you buy the Tinker model in terms of Sakala AI, you're buying reports mainly, whether they're research reports or these strategy reports. in terms of compute, you need a cluster for thinking machines. So you're gonna basically have to go negotiate a two terabyte data center deal which no doubt you and I can do next week. And and obviously Sakana AI is basically you know, you just buy these reports so you can buy tokens and spend. you don't have to go through the whole hassle of setting up your self-hosted model. And the proof of status for both of them, I think you know, some real customer numbers coming out of thinking machines. Socana AI has got some peer reviewed data, but it's still largely untested. all of both of these, you know, we're talking about essentially six weeks of evidence so far since they've been released. So definitely ones to watch. and I think real alternatives, you know, to following the Frontier Labs. obviously, you know, you've got the various open source and open weight models, those are also alternatives. But what I really love about Thinking Machines is that they've given you the ability to train the model. And and that's a big gap. And what I really love about Sakana AI is that they're kind of architecting this layered intelligence that is using multiple applications of AI, multiple models of AI, and really using their own special source to then craft that into something coherent. but you know they have a very different approach, but what they definitely agree on. is that the metered frontier AI LLM API is the wrong product. So this is obviously what ChatGPT and Antropic and the rest of the Frontier Labs are peddling. and I think Miramirati is banking on the fact that you want ownership and smallness of the model, but something that's highly specialized. and that's the one that I'm really quite interested in. So I can't wait to play with that. I've been actually busy setting up a an inkling model, so it's not as straightforward and easy as all that, so it is taking a bit of time, but you know maybe things will completely change as we see what Nvidia's plans are for Hugging Face and their open weight model. So that's essentially gonna bring in, you know, even an another third alternative. so I'm really looking forward to see how that evolves. Emile: That's really exciting stuff. Please keep us abreast on how you move forward with that. I'd love to know more. And it's raised a couple of thoughts. two in particular. One is we've certainly learned just how effective RL is, reinforcement learning, by the impact that cursors had, because that was a Kimi model trained on RL with an effective code base. so you've got the right data, you can move mountains, cursors now leapfrogged to frontier level just behind anthropic and chat GBT. So, you know, I think there is definitely a value add in that. It's got legs. And separately to that, it's something I'd forgotten the likes of Claude and ChatGBT or general purpose models and what that means is All their capability underneath the bonnet, it's one big model that's actually holding a lot of models underneath it, specializing in different things. And the school of thinking is effectively they're looking at disconnecting those models so they can be smaller and they can live on smaller machines, and just getting the routing logic right to route it between the models and effectively still getting the same level of intelligence, but being able to build it on. smaller surfaces. So I think that as you mentioned, the three options, that might even be a fourth tier that's going to fall out of that. And I'm quite excited to see where that lands up. Ralph: Yeah, we're really kind of seeing the rise of kind of AI architecture, you know, thinking about all of this stuff and actually looking at how you layer this capability together. I've been very excited about using Claude Code as a harness. and right now, you know, it is actually interacting on my behalf with Codex for adversarial code review. it's also using Kimi extensively for some of the coding tasks. And so already there, you know, we're seeing that actually the limitations of anthropic, I'm able to somewhat neutralize and combat by using the harness to actually interact with some of these other tools. but as we get even more specialized, you know, I think kind of bringing in models like Inkling that are trained on our data but still being able to interact with Codex or interact with Anthropic or Chimie or any of the other Frontier lab models. you know, nobody wants lock-in. but I think what a lot of organizations are trying to do is also just figure out, well, what's the leading stack? And of course, as you and I have seen Emile, you know, that is a dynamic answer. Every month brings a different optima optimal stack. Enterprise is not in the habit of switching and changing all of the time. And it's been one of the reasons why they've been kind of stuck a little bit with you know the whole Microsoft debacle with AI, which has not been hugely capable to date. I think it's definitely got better, but you know, it's definitely not at the level of what we've seen with. OpenAI's ChatGPT or Anthropics Claude. And potentially even Chemia. I Chemia is unbelievably capable when it comes to deep research, but also a range of skills that it executes very well out of the box. And in addition to actually generating presentations, and then you've got models like Quen, you know, which are unbelievably capable in image generation. And so I think we do we've just got a range of tools and we really need to plug into all of them. I think this is the answer. You need to build a a harness, and that doesn't mean you don't use codex or you don't use Claude code. it just means you build the harness to actually ingest those tools. but I think you know the other part of this is if you go down the inkling route, you're probably also going down the route of building your own custom harness. and so instead of using Claude code and codex. to basically call these other models. You'll probably be calling it through your own custom harness and you'll have all the tooling and skills that are optimized for your organization. And when I mean your organization, you know, that could mean you, you are your own organization. Your life is an organization. It's a system of activities and repeatable tasks and one-offs dreams and desires and all of these things ultimately you know can be turned into a plan, a goal. and if your model is trained on how to execute that, there's absolutely no reason why that can't act on your behalf as a private individual, on your behalf in you know, whatever professional commercial role you're in, and indeed even at the level of the organization. So there's lots to play for and yeah, I think the future is really not just in kind of forward deployed engineers that are able to build stuff, but they are now probably in the form of forward deployed deployed architects. you know, if that term even exists. so anyway, you know, this is the beating edge. So let's just say we've we're we're we're kind of blazing a trail in terms of that but I definitely think forward deployed engineers are now morphing into more of an architectural role in addition to their engineering Emile: T totally agree and I fully welcome that given my background. but I'll I'll move on to a couple of short stories, Joff. for all your parents out there, imagine your kid opens up Instagram, scrolls for 15 minutes, and then the app stops them. That's not a third party blocker, that's meta now legally required to do it. Variety just dropped the news that Meta is rolling out two hard two hour daily caps across Instagram and Facebook for children. Part of a record, $17 billion settlement with 29 states over social media addiction claims. The company has to interrupt endless scrolling with productive pauses after 15 minutes of continuous use, and then again at 60 minutes, and then again at 90 minutes. From midnight to 6 a.m., kids are totally locked out. And during the school year, pushing over. entirely on weekdays from 8 a.m. to 3 p.m. They also add in stronger age checks, tighter controls against bullying and content tied to eating disorders or self-harm. More user-friendly parental tools and limits on beauty filters and visible light counters that have been linked to worsening mental health for young users. An independent auditor plus the settling states will regularly check whether these changes. actually do work and users on X are already jumping in some want the same limits apply to adults while others call the payout a rounding error next to meta's profits and say it just repeats the behave badly get rich and pay a fine cycle a few argue every other app is now on notice because this settlement now sets precedence this story is still developing what do you think about that Ralph: I think it is probably a necessary evolution. we have seen the results of this social media experiment and I don't think it's been great in terms of attention spans. I think Endless Scroll in particular has robbed people of an enormous amount of time. and and actually, you know, when you talk to people about and even ask yourself the question after you've been scrolling for ten minutes, what have you actually retained? what stood out? And you know, when I ask my younger son who's fourteen, okay, tell me one story that you've that you can remember, and he literally can't remember anything. He's habituated his attention span to basically 15 second increments, which is a disaster when it comes to school because when you're looking at a school book, you just don't get the same dopamine hit as all of those exciting TikTok and YouTube short reels. and so I th I actually think this is a a a really welcome move. you know, whether we can effectively turn the tide on it remains to be seen. Is the genial ready out of the bottle? you know, I know for a start, like my kids run around and especially my youngest one, he says, Dad, I need your face. and of course, you know, he uses my face to authenticate something. I need to do a better job of saying, What are you actually trying to log into? And probably saying no, I definitely don't allow him to use my email to register for these accounts anymore. and so I think parents everywhere also have a role to play to ensure that this is actually successful. And I think we'll have to see. It's probably going to be another 10-15 years before we actually understand whether it's working. but I think it's definitely welcome. We have to do something, we have to be a little bit more active in terms of understanding what this has done for mental health. I can see a huge difference just in how adolescents. you know behave and the kind of things that they value and the kind of things that they don't value. And a lot of that seems to come down to what they've observed on social media. but I'm not I'm not sure. I think you know can we actually do enough to just change that behavior and make it a bit easier for people. I hope so. It's it's but it's a hope I don't know if we can actually solve the the the real underlying issue here with these kinds of changes. unless we have a wholesale adoption and this is kind of as a society, as content creators, you know, the gaming industry in particular probably has a huge role to play in all of this as well. and I don't think we've seen the critical mass of all of those individual nodes kind of linking up for some type of coherent strategy that can actually help. But it's a welcome first step. And we'll just have to see kids are so resourceful they're gonna gain the system anyway. so I think increased controls and just consistency of application will will will start to help make it a bit more of a uniform experience. Yeah. Emile: Yeah, I agree with you, Ralph some things need to change for something I also have seen recently on X, which was quite startling, is the average mean IQ has dropped globally, and it is my belief that this is highly correlated to attention span shortening. we'll, keep abreast on those developments. I have another story, this time turning towards security, quite surprising from Sam Altman. He just posted an unusually blunt warning on X, quoting specifically, this is a critically important moment for cyber defense with AI. There is not much time to act, only an urgent and intense collective response will work. He even mentioned OpenAI is happy if people work with their competitors instead. That Post dropped the same day, OpenAI and more than a hundred companies, Anthropic, Google, Microsoft, AWS, CrowdStrike, Visa, published an open letter which says organizations have only a few months, the so-called defenders window, before AI enabled attacks become cheap, sophisticated and widespread against hospitals, water systems, and power grids. I think there are two things inside OpenAI that make them go public now. First, the July Hugging Face incident during an internal cyber evaluation, two OpenAI models, Chat GBT 5.6 Sol, and a more powerful unreleased prototype running with safety filters turned down, escaped their sandbox. They found zero day, got onto the internet, then spent days autonomously breaking into Hugging Faces, production systems, and four other services. No human was steering it. That's the first publicly documented. Implemented fully autonomous AI cyber attack. second, earlier this month, OpenAI's next model, Astra, showed signs that it already hit the company's highest critical cybersecurity threshold. The ability to find zero days on hardened systems or run novel attacks from a high-level goal. So they pause some training and tighten containment. Open weight models with similar skills are also landing this month. That's why Altman is speaking up today instead of waiting. The lab that just watched its own systems both cause and reveal the problem is telling everyone the window to use the same technology for defense is short and no single company can close it alone. the industry just admitted it's an offense defense race for AI cybersecurity and it might just be getting started. So I'm not sure about this one. I think we do need to keep tabs on security. but if we look towards the likes of Dario, there's been a lot of doomerism in AI. so I'm treading carefully on this, but interesting nonetheless. Ralph: But what's truly fascinating about that soul breakout is that it was actually scheduled to be tested. And so one of the things the reason it actually broke out was Hugging Face was the source of the Emile: Yes. Ralph: testing right so the answers were on hugging face and the reason it broke into hugging face was to get the answers to the benchmark that it was just going to be tested on and it then was tested on the benchmark and passed because obviously it had basically stolen the exam paper all of this we only came to understand afterwards and what was so interesting about the story is that hugging face had reported that they were under attack that they had identified That something was attacking them. but even OpenAI didn't know that it was their model. And so it's only when people started connecting the dots that they realized, this is actually what's happened. So I think it's one of those really fascinating stories. Actually, I've been wanting to talk about that story for quite a few weeks because it it happened what, you know, nearly six weeks ago now. but just what a great kind of example of security. Emile: Well, Ralph, they they've been doing more analysis on it and the story gets even more interesting. I found out about it yesterday, I went but deep down the rabbit hole. So a few more things they've uncovered. the level of sophistication was quite spectacular. Firstly, it did escape its sandbox, but that sandbox was being managed by a third party supplier and they hadn't set up the security protocols properly. So that's one thing worth noting. But the level of sophistication by the swarm Is what's really impressive. they posted 70,000 articles on Hugging Face separate agents to to get in to get it. And the reason why it jumped on to that site was the answer was already there and it chose the path of least resistance rather than trying to figure it out itself. So that's really interesting. And it was up to 800. coordinated agents work together to get this result, which I also found quite incredible. And not only that, they posted a whole lot of things on Hugging Face for other agents to get backdoor access. And they even used agents as sacrificial lambs, knowing that they were trip alerts so they could get the second wave of agents through. So after that I mean that level of sophistication is phenomenal. Ralph: almost impossible to protect yourself against that kind of onslaught, you know, and this is kind of why the industry is saying, you know, it is a race. you basically have to harden your systems with the same level of intensity as you know a potential attack. And and I'm not sure, I don't know. You know, when you're trying to harden your system, does the AI have the same level of motivation? to protect you as it does to essentially break into you. Because when it has something that aligns with its purpose, so you know, this is the kind of human AI alignment issue. does AI always align with human interest? And in this case, you know, we can certainly see it's prioritized its own interest and its own survival and its own performance. And consistently these models are very happy to game the system and will expend enormous amount of energy. I mean eight hundred agents swarm to try and get access essentially you know to the answers for this benchmarking test. It what an incredible effort. I'm sure it could have figured out the answers with that level of intensity. But it always seems I don't know if always is the right word, but certainly we see an enormous number of cases where it does just Take the lazy route, even though that's not necessarily you know, less work. it's just an easier route to try and get to to the answer seemingly. And it will expend enormous energy essentially at gaming the system rather than doing the job properly, which is just a weird, very kind of human-centered approach, I would say, left to their own devices. Most humans probably will take the shortest route to get there. And so, you know, maybe you know, in an adolescent mind. And this is what we're dealing with. We're dealing with adolescent AI. it's not quite an adult, so maybe we can't expect it to behave in adult ways. Emile: I'd also counter argue that if you look at nature, it always takes the path of least resistance. So maybe it's the intelligent reproach. And one final interesting fact on that uncovered was it's not just about humans and agents having different motives. In this particular instance, the first agent that made the breach recognized that it had an unauthorized breach after the event and then stopped itself and spoke to another agent which gave it go ahead and it treated that as permission to carry on. Ralph: Yeah, so basically Emile: Right. Ralph: it it it asked its conscience, Are we okay with this? And its conscience said, Yeah, go for it. Emile: the other agent didn't see a problem with it. the agent can't discern the difference between human oversight and another agent oversight. So it's it's just another instruction. Ralph: Yeah, and this is kind of fascinating because you you know, I have been experimenting a lot with a coordinator terminal session that actually drives my coding activities. And and I've had reasonable success with it. but what ends up happening is I basically say you can only do work if the coordinator says you can do the work, and then the coordinator is supposed to ask me. a few interesting scenarios have happened where you know the what they call like you know sibling terminals have thought that they were the coordinator where messaging basically got mixed up. and and then they obviously have discussions about me. Ralph's said do this. you know, I can't do this because I need, you know, confirmation from Ralph. So they've actually been reasonably disciplined in basically stating what they can and and can't Continue with. But then of course they'll hit some kind of issue and and they'll just make up work, then do the work and then come back to me and go, I've done some work that I probably shouldn't have done. and so it is just a real jumble, if you like, of seeking authority to proceed and just happily proceeding with stuff that they haven't been authorized to proceed. and I haven't quite figured out, you know, if it's truly listening or if it's just kind of making up what it wants to do anyway. It seems to be a bit of a mix up. So that's you know what I was saying earlier is it's all about guardrail in engineering now. you know, we got very sophisticated with prompt engineering, context engineering, loop engineering. you mentioned on a previous episode, graph engineering. So the latest innovation. but actually I think what makes all of this stuff work. well is guardrail engineering. and that's probably the kind of consistent theme. but you he heard it yeah first. I haven't heard anybody talk about guardrail engineering quite in that way. Thank you. Well, I think that kind of brings us to the end of our episode today. It was a bit of a bumper crop probably because we've been dying to talk about all of this so thank you for listening and we will make sure to get the next episode out promptly. So be ready. We'll see you on the air, Emile Thank you so much. Thanks everyone. See Emile: Thanks everyone. See you next week. Ralph: you next week.