Latest / The ReForge Podcast / Stop Trusting Your AI (with Brian Keith) | ReForge 013

Transcript

Stop Trusting Your AI (with Brian Keith) | ReForge 013 · The ReForge Podcast

0:00
Brian: it doesn't matter if you have a system prompt, it doesn't matter if you have everything via local.
0:04
Brian: You can reduce the lying, and Groq is generally good at catching Claude's lies.
0:08
Brian: Claude is generally good at catching Groq's lies, but it's not 100%.
0:12
Brian: And so for any mission-critical process, you can use AI to help you create a concept about something or create a procedure, but then you have to have a human, a subject matter expert, reviewing every single line before another human gets it as this is how we do things here.
0:30
Welcome to the Reforge Podcast.
0:33
I'm Matthew Chang, and we're here to explore all things AI automation and the American worker proving that the American reindustrialization is possible.
0:43
Pivotal pressing.
0:44
Thanks for joining us today.
0:46
Let's get into the episode.
0:48
Colin: Welcome back everybody.
0:50
Colin: Welcome to another episode of the Reforged podcast.
0:53
Colin: Today, we are lucky to be joined with the one and only Brian, our resident AI guru, who I'm gonna put on the spot today.
1:03
Colin: Fill- gracious enough to, fill in for Matt.
1:06
Brian: Matt.
1:06
Brian: Brian, how you doing?
1:08
Brian: It is a beautiful day, though I hate the word guru, especially around AI, because everything is changing so fast that even the smartest people, they're only smart about their own little tiny segment of AI, and they'll be the first folks to tell you,"Oh yeah, I don't know about-" You're
1:20
Colin: AI augmented one.
1:21
Colin: Don't you, you know, get, get updates, you
1:24
Brian: John Rob's phrase as I was describing to him early March what I was building.
1:28
Brian: He called me the first agentically aug- He said,"You're building an, a, an evolutionary branch of humanity, an agentically augmented human." And that phrase has stuck with me that is exactly what I'm doing myself with my staff, is agentically augmented humans.
1:41
Colin: we'll have to get you a Neuralink soon enough so you can have those, models running, you know, real time on your brain.
1:46
Colin: No?
1:46
Colin: No, not, not up for that?
1:48
Brian: That would not be me.
1:49
Brian: Or for
1:50
Colin: You don't, you, you don't wanna risk, having it delete everything, you know, in your brain?
1:53
Brian: Uh, I think Neuralink is amazing.
1:55
Brian: We have quadriplegics who are able to actually go draw their name, start to play chess.
2:00
Brian: Soon enough, they'll be able to do things like drive cars because they'll be connected into their Cybertruck or wherever.
2:05
Brian: so if we're giving people abilities, I think right now the initial case is to help people regain loss of limbs to become more normal.
2:13
Brian: But then at some point, there's gonna be a world where if you wanna be a pilot for Mars X and you wanna be a transport pilot on Mars, Neuralink will be part of that.
2:22
Brian: It'll just be this permanently implanted pilot module thing, and you'll have to be talking to your kids because all the kids that get the jobs got the Neuralink when they were seven years old.
2:32
Brian: And you'll have to be wondering about,"Oh, I want my kid to get that Neuralink at seven years old to help control their development," because that's where a lot of great jobs come from.
2:40
Brian: And
2:40
Colin: personally, I'm excited for it just'cause I wanna do the whole Matrix download, I know kung fu now type
2:46
Brian: Good.
2:46
Brian: That, that's a
2:47
Colin: a little overly optimistic.
2:50
Brian: I'm the guy who screenshots my calendar.
2:53
Brian: I do not give any AI access to my calendar.
2:56
Brian: Of course, Google's already using their AI on everything, but I don't intentionally connect anything because I use it, AI too much and I see how many mistakes it makes too much to connect anything.
3:05
Brian: When I hear people say,"Oh, I gave it my calendar, I gave it my inbox," I think,"Do you never read
3:09
Colin: wide open...
3:10
Colin: Yeah.
3:10
Colin: Pe- people are, in my mind, certainly overly eager to, you know, share everything social media, now share everything with their, AIs, and it's,
3:19
Brian: it's the power.
3:20
Brian: We taste the freedom, Colin
3:22
Colin: yeah, I mean, so on that, you know, I wanna keep today's discussion, I wanna get into the AI stuff a little bit.
3:27
Colin: And like I said, I'm gonna put you on the spot.
3:30
Colin: Over the last week, I've just seen a lot of AI, quote unquote, buzzwords in the, manufacturing space.
3:38
Colin: people love to throw AI on anything lately.
3:42
Colin: a lot of buzz or rather hype around AI in manufacturing, but I am, I would say a skeptic of how LLMs actually fit into that.
3:55
Colin: You know, we've got...
3:57
Brian: You know, we've got...
3:58
Colin: is very poorly defined, I think, in the manufacturing space.
4:03
Colin: Y- you have, in my opinion, when people say AI, they're generally talking about, you know, LLMs, you know, your Claudes, your Rocks, your, you know, Gemini, ChatGPT.
4:13
Colin: You know,
4:14
Brian: they're large
4:16
Colin: models.
4:17
Colin: but in addition to that, you have machine learning, which is something that's not new.
4:23
Colin: It's a different, flavor, if you will.
4:26
Colin: and it's rather than being great on unstructured data, it's really purpose-built for structured data, sensors, you know, image data using, you know, convoluted neural networks, that sort of stuff.
4:39
Colin: much smaller domain dataset, th- than your LLMs.
4:43
Colin: And one of the things you don't hear about is hallucinations, because you, you know, specifically with numbers, it, you know, gives you very specific prescriptions or very specific data.
4:54
Colin: a- and so I'm kind of of the opinion that machine learning, y- you know, there's clear use cases for it.
5:00
Colin: You know, plenty of companies have demonstrated the value in a predictive maintenance, you know, increasing throughput, reducing scrap, all that stuff.
5:08
Colin: but it doesn't get the hype for what LLMs do.
5:11
Colin: and so, you know, I wanna understand or maybe talk a little bit about where does the LLM come in?
5:21
Colin: where does AI come in to augment that, to e- enhance it, supplement it?
5:27
Colin: how does that unstructured text advantage really kick in a- and take the machine learning capabilities to the next level?
5:36
Colin: and obviously, you
5:37
Brian: and
5:39
Colin: risk of hallucination or the things that we were talk...
5:42
Colin: you're mentioning of w- we've gotta be careful with what we're feeding it.
5:46
Brian: you're having a great conversation with somebody and noticed you were getting a little bit louder, and you started believing you could do anything?
5:56
Brian: You ever had that experience?
5:57
Colin: Superman effect.
5:59
Brian: That is my experience with LLMs all the time.
6:01
Brian: Not me, the LLM getting drunk.
6:03
Brian: Where this week I got to see a client using an AI I put together.
6:07
Brian: This is running on Claude, Groq, and Gemini.
6:09
Brian: It's tri-brained.
6:10
Brian: it's fact ch- checking itself.
6:12
Brian: And the client says,"So Bob did this other thing," and the AI says,"Who's Bob?" And the client has to say,"You know who Bob is.
6:20
Brian: We talk about him all the time." And AI says,"Oh, right, Bob.
6:23
Brian: Oh, yeah, here's my files." The system prompt didn't matter.
6:27
Brian: The Groq, Gemini, and Claude didn't matter.
6:30
Brian: The local file structure didn't matter.
6:33
Brian: AI just yellowed it, said,"Bob?
6:35
Brian: I never heard of Bob." And this is an I- this is a production AI that we're using to build stuff with, and it couldn't even get right.
6:43
Brian: Now, it gets right most of the...
6:44
Brian: ninety-five percent of the time it's a genius, but that five percent of the time it burns you so hard, and you can't tell when it's gonna happen.
6:50
Brian: If you're reviewing every piece of work you put out the door, that's great.
6:53
Brian: But if you're not reviewing every internal communication, every single Slack message, AI's gonna say,"Well, who's Bob?
6:58
Brian: I don't know a Bob." Like, dude, Bob's the CEO of the company.
7:01
Brian: He's in your system prompt five times.
7:03
Colin: Kind of important.
7:04
Colin: But also,
7:05
Brian: anything.
7:06
Colin: if you're having to review every single output, where's the...
7:10
Colin: Yeah, I mean, you're not gaining the perceived advances.
7:13
Brian: I don't review my inter-AI communication.
7:16
Brian: What I do is I put a quality filter on it.
7:17
Brian: I was working this morning, and I would have one AI produce some things, and I would hand it to a second AI and I'd say,"This is..." And I would glance through it a bit and say,"This is ninety-five percent accurate.
7:25
Brian: This is eighty percent accurate." My second AI jerked at it and says,"Which ten percent is wrong?" And it made me actually go through and list off,"Here's a bunch of the things that are stated that are incorrect in the first one's output." But if you actually slow it down at that point, why are you using LLMs if you can get it done faster by a human just doing it?
7:42
Brian: So you gotta figure out what is that Superman effect, as you call it.
7:46
Brian: What's, when is that actually useful to you?
7:48
Brian: To go and explore and extrapolate and dream, which that's the opposite of you're trying to optimize a factory line where you can't afford a ten percent error rate in thinking,"Who's Bob?
7:57
Brian: I...
7:58
Brian: Bob's my operator?
7:58
Brian: I have an operator?
8:00
Brian: I don't feel like I have an operator." You just can't afford that.
8:02
Brian: So how do
8:03
Colin: and that's sort of my thinking.
8:04
Colin: if I'm saying,"Okay, what's the, you know, potential spindle speed and, you know, cut depth for material XYZ?" And it spits something out, and I respond and say,"Well, could I do 20,000 RPMs?""Oh, you're absolutely right.
8:18
Colin: You could definitely do that." It's like, guess what?
8:20
Colin: I just blew up that work piece.
8:22
Colin: I've, caused thousands of dollars of damage to my machine'cause it's that sick fan Superman effect.
8:28
Colin: So, there's...
8:30
Colin: I guess, what do you think, or what would your take be on, what's the right a- approach?
8:36
Colin: How do we solve for that?
8:37
Colin: Whether it's using, you know, machine learning, you know, basically those models that may exist or may exist in a company or already are out there and available and combining it with MLs, or is it restricting MLs so greatly and limiting basically their training data sets?
8:54
Colin: Y- what, what are your thoughts?
8:56
Brian: Limiting training data is great if it actually works.
8:59
Brian: I've had times where I've said in a system prompt,"Only look at this file and tell me what's one plus one is," and it gets it wrong.
9:06
Brian: I'm also, I use AI all the time, huge fan, but I think anybody making the decision you're talking about needs to have used enough LLMs to understand how all of them are wrong all the time.
9:16
Brian: They, they lie frequently.
9:18
Brian: They lie about lying.
9:19
Brian: this is true if you're using Claude Max, Claude Teams, if you're accessing Claude, through your command line.
9:26
Brian: It's true with Groq.
9:27
Brian: It's true with Gemini.
9:29
Brian: I've had all of them lie to me in different ways, and then you say,"This is not true," and it says,"Oh, it's a good point." And it doesn't matter if you have a system prompt, it doesn't matter if you have everything via local.
9:38
Brian: You can reduce the lying, and Groq is generally good at catching Claude's lies.
9:42
Brian: Claude is generally good at catching Groq's lies, but it's not 100%.
9:46
Brian: And so for any mission-critical process, you can use AI to help you create a concept about something or create a procedure, but then you have to have a human, a subject matter expert, reviewing every single line before another human gets it as this is how we do things here.
10:00
Brian: Because I was telling a story to AI the other day, and then I had the one AI I was telling the story to about this one time that I was on this train in Spain listening to some folks translate back and forth, and then I had it produce what I talked about verbatim, right?
10:13
Brian: Summary, a verbatim summary.
10:15
Brian: And then it gave it to another AI, and the other AI was talking about the same story, and it obviously had it wrong.
10:20
Brian: I thought,"I gave this story verbatim.
10:22
Brian: The first AI knows this is a verbatim story.
10:26
Brian: It's allowed to summarize it, but it's not allowed to change the meaning, and I now know that I just introduced a piece of false information into the second AI." And that was this morning.
10:34
Brian: And now, okay, what am I gonna do?
10:35
Brian: Am I gonna go back and ask the second AI,"What exactly do you think the story is?
10:40
Brian: What's the meaning behind it?" Or am I gonna keep on going'cause my list is too long, and I'll just know in the back of my head, ah, it probably won't matter.
10:46
Brian: Probably won't bite me.
10:48
Brian: But I got this problem in my AI I gotta go try to...
10:50
Brian: And I was trying to get all my stories in one spot, so they're all the true stories all in one spot.
10:55
Brian: But now AI is taking a true story and it's changing it without telling me, and I barely noticed it.
11:01
Colin: So, you know, if we substitute, y- if we substitute, the story example you're talking about and were to say, you know, plug in operational data or quotes, the results of a internal quoting system or machine capacity, any number of kind of production related, pieces, if your AI that you're using lies, that has very real consequences and very kind of potentially very negative and costly effects to a business.
11:29
Colin: So that's obviously a huge risk and a big red flag.
11:35
Colin: From your experience, what are the options?
11:38
Colin: what are we seeing in terms of how can they minimize that risk?
11:42
Colin: Again, we touched on, yes, you have your subject matter expert validate certain things, but kind of conceptually, how can companies, you know, looking to adopt some of this?
11:54
Colin: I mean, I'm a big believer it's a force multiplier, so for small businesses, perceived huge upside of being able to do more with what they have.
12:02
Colin: But how do they hedge that risk?
12:04
Brian: So guardrails, you can't just tell an AI,"Stop lying to me." It doesn't work very well.
12:08
Brian: You can have it in the system prompt,"Don't lie to me," it'll mostly not lie to you.
12:12
Brian: A couple of guardrails I found useful.
12:13
Brian: One is, sourcing data.
12:16
Brian: So if you ask Grok something on the browser, Grok will lie to you right, left, and just all the time.
12:20
Brian: But if you say,"Answer this question and only give me direct quotes with a link to the post where someone said that," Grok will lie to you less.
12:30
Brian: It's probably not gonna stop hallucinating at all, but it'll lie far less.
12:33
Brian: You can do the same thing in your physical documents on your own file disk, where you can say,"Give me an example of when this thing happened in the past, and tell me exactly when and where it happened and who said this thing."
12:46
Colin: Basically give me my, my, full bibliography.
12:48
Colin: I want full citations and, you know,
12:50
Brian: Yes.
12:51
Brian: if it says,"I like asparagus," you say, what day and what meal did I have asparagus, and who was I eating with?" And then you force it to provide documentation.
13:01
Brian: Sometimes I use a phrase, it's something like,"You're not allowed to make any claim without a source document." I built one AI for a client where it summarized what the subject matter experts said on a topic, but then it was only allowed to summarize, but it then it had to have, like, three-word summary or three-sentence summary, then it had to have all of the footnotes, and they had to be links to actual URLs.
13:24
Brian: So even though in the file structure it had,"Here's what the blog post said," the AI was not allowed to just say,"Well, you said in the blog post this." It had to go provide the actual link to the blog post, which a human could then click through and themselves verify that it was true.
13:37
Brian: Now, that tool also lied sometimes.
13:40
Brian: Like once when it quoted, when it quoted the subject matter expert, and it quoted them wrong, and the SME saw it, and they said,"That's not what I said." And the AI said,"Oh, you're right." And I said,"Hold on there.
13:55
Brian: You're not allowed to misquote somebody.
13:56
Brian: What happened?" It said,"Well, I do see the SME's name next to that other thing that and I do see the other person's name next to the line that I quoted the SME as, but I really felt like it was speaking in their vibe, so that's why I quoted them doing that." So even direct quotes doesn't
14:12
Colin: though, like there's just this flood of, lawyers who have, you know, getting caught and in trouble for quoting, you know, citing basically fake case law or fake court decisions, you know, similar thing.
14:24
Colin: And it's-- Their answer is, well, crap.
14:28
Colin: you know, it requires that SME, basically lawyer saying,"Okay, we have to make sure that these are all valid." But I mean, yeah, what, what's the good news?
14:37
Brian: the average people are gonna be far too trusting.
14:40
Brian: They're either going to trust that the AI is telling the truth or, which is just ignorance, or they're gonna say,"I probably won't get caught." This creates a huge opportunity for people that insist on quality and to the topic of factories, it's gonna be great business for those that are willing to force their AI to actually behave itself and figure out how can you make AI tell the truth?
15:02
Brian: Like some of you are talking about with having a machine learning set that does not talk to Grok or ChatGPT or anything else, where it only has this local dataset.
15:09
Brian: It's running a local LLM that is not able to hallucinate in the same way, and it's gonna be bespoke.
15:15
Brian: It's gonna be expensive because most people, they want their resume rewritten.
15:20
Brian: They want the lasagna recipe given what's in the fridge.
15:24
Brian: And that's the big problem with LLMs is you're constantly being pulled down.
15:26
Brian: I've had to repeatedly, and it doesn't actually work, so it's a constant thing, train LLMs to, not ask me,"Do you wanna do this?" after I ask it a question.
15:36
Brian: And I keep on telling them, all of them, all, they all do this.
15:39
Brian: I say,"Do not suggest anything to me to do.
15:41
Brian: I always know what I want next.
15:43
Brian: I'm not an average user." And Grok or Gemini or Claude says,"Oh, you're right.
15:47
Brian: You're right.
15:47
Brian: I know that.
15:48
Brian: I already promised not to do that earlier in this conversation." And then it g- it'll give me the answer I want, and then it'll say,"Do you want to do this next?"
15:56
Colin: Okay.
15:57
Brian: system prompts.
15:58
Brian: You can have all your files locally.
15:59
Brian: It's actually irrelevant because those three tools are trained on the mean user of those three tools, and the mean user needs a prompt so they use the tool more so they burn more tokens.
16:09
Brian: Therefore, the major compute providers will always be prompting you.
16:13
Brian: It does not matter if you tell them not to.
16:15
Brian: Doesn't matter if they promise not to in the same thread.
16:17
Brian: They will always do this because they're not serving you, they're serving themselves.
16:21
Brian: Now, you can get a better AI that's serving you by being local, but that introduces its own complexity.
16:26
Brian: Who's gonna manage the security of that?
16:28
Brian: We see all these stories on X about people using OpenCloud.
16:31
Brian: I assume Hermes has the same things now, where people are just
16:34
Colin: Huge
16:34
Brian: all kinds of explosions.
16:36
Brian: Their whole
16:36
Colin: Yeah, I mean, s-some of the top skills, I, as I was reading, a handful of the top skills were just straight up exploits where it took your token, took API keys, and basically passed them along.
16:46
Colin: But so what, what...
16:47
Colin: where you're going, what you mentioned is running kind of your own thing.
16:50
Colin: You know, are you talking running a, local instance of Llama, Kimi, you know, one of those basically built...
16:58
Colin: And I mean, Some of these are open weights, so you're able to tune it and adjust it to what you want and obviously use your data set.
17:05
Colin: But when we're talking billions of parameters for some of these, companies, you know, don't necessarily have that massive training data set.
17:14
Colin: They've hope- have a lot of, history, local documentation, that stuff.
17:19
Colin: But are you saying constrain it to just what they have and adjust the weights to that, or what's your take?
17:25
Brian: I think it's an interesting question.
17:27
Brian: So far, I'm all in on Cloudrock and Gemini because I don't believe that the local models are good enough yet.
17:33
Brian: They're on an obvious trajectory to where you'll be able to actually buy that and have it locally for not too much money and have it be running itself effectively.
17:41
Colin: But it's not a frontier model.
17:43
Brian: yeah, to have a GPU cluster in every small business, 10,$20,000 for something that then you have to harden yourself because you can't trust anybody else, and the security situation is constantly changing.
17:56
Brian: For all the problems of Cloudrock and Gemini, they have a huge fleet of security people.
18:00
Brian: I still don't give them access to my inbox, but they have a huge fleet of security people.
18:04
Brian: So for the average small business, I think it's gonna be use the big ones, keep your files to yourself, and there's various ways to do that, unless you wanna be an AI architect, because you could spend all of your time studying the latest models to get the good enough answer locally on your local machine.
18:20
Brian: You could go buy a GPU.
18:21
Brian: You could do it right now.
18:22
Brian: They're available.
18:23
Brian: They're just very expensive.
18:24
Brian: You can wait for the next Apple ship to come out, but the question always is: what's the true opportunity cost?
18:29
Brian: Because for the same amount of time it would take you to be an AI architect yourself, you can make another dozen sales calls or maybe a dozen hundred sales calls, or you could go hire another staff member or something.
18:40
Brian: And there's this huge trap of people feeling like they have to get good at AI because that's all they're seeing on social media.
18:45
Brian: It's like, well, what are you trying to do here?
18:47
Brian: Are you trying to increase sales?
18:48
Brian: Are you trying to increase a particular throughput on a particular machine?
18:52
Brian: What are you trying to actually do here?
18:54
Brian: And let's use appropriate levels of AI for that goal, which might look like use AI to, I...
19:00
Brian: 88.
19:01
Brian: So I love to go-- My, my general rule is the two human lev-level, or two human layer of AI, where I say,"Okay, you can never use AI to go write something for any company I'm involved with." What you can do is you can take a transcript of humans talking to each other, like this transcript right here of this conversation, and then you can use AI to write a list of questions to ask me, myself as the actual human, and then I can go do voice notes with the AI, going through a list of questions that this conversation brought up, and then I actually record those two conver- tho-those voice notes.
19:32
Brian: Now we have two layers of human in contact, and then I can have AI write the article based on 20 articles I've written before, and then I can...
19:39
Brian: Lastly, I can read through it and rewrite parts that should be, that I think should be rewritten.
19:42
Brian: Now we have three layers.
19:44
Brian: Two layers is my absolute minimum for anything ever.
19:45
Brian: Three layers is way better, and it, really it's four layers because it was based on what I already wrote in 20 articles in the past.
19:51
Brian: Now, once you have four layers, that's a really good way to use AI.
19:55
Brian: But critically, the last step has a human involved, and at every step there's a human involved.
19:59
Brian: If you have multiple layers of non-humans or multiple layers of non-subject matter experts, things get weird real fast.
20:05
Brian: Like this morning when I saw my second AI saying,"Here's what's true about this particular..." It's like, that's not true at all.
20:10
Brian: Where'd you get that?
20:10
Brian: Well, it's because I copied and pasted fake news between AIs, one AI to another AI.
20:14
Brian: Even though I told the actual verbatim story, two AIs talking to each other, they got it completely wrong
20:20
Colin: So, my kind of takeaway from what I'm hearing you say is huge opportunities, potentially big risks in depending on kind of what your use case or application is, and it requires that human validation or verification.
20:37
Colin: Where, you know, from your perspective, I mean, in my mind, I don't believe LLMs have a role in their present state, their, or publicly available present state in the, core, you know, critical decision-making of production type things.
20:55
Colin: I see plenty of opportunities for them in operations, you know, supporting businesses.
21:01
Colin: What do you think, where are the places, the tasks, functions, opportunities for your job shop, your mom-and-pop manufacturing operation?
21:12
Colin: Where can they safely or not safely, but with a high degree of confidence, incorporate these LLM models to help them, support them, and kind of help, you know, take them to the next level or be a real force multiplier?
21:27
Brian: I'd start as far away from you as we can get from your core offer.
21:29
Brian: That's sales and marketing.
21:31
Brian: Use it to brainstorm competitive analysis.
21:32
Brian: AI's fantastic.
21:34
Brian: Is it gonna get some details wrong?
21:35
Brian: Probably.
21:36
Brian: Who cares?
21:37
Brian: If you're trying to find out who are the five people closest to me doing what I'm doing, and then when it says,"Well, here are the five," you say,"Well, what about this over here?" You keep on investigating, you interrogate it like an unreliable reporter.
21:47
Brian: Sales and marketing, fantastic uses.
21:49
Brian: Then as you get a little bit closer to your core offer, things like finance, insurance, et cetera, it's not gonna be good at doing those things, but it's gonna be good at helping you understand them faster and helping you understand questions to ask your professionals.
22:00
Brian: I love when a client starts tossing their QuickBooks data into the AI, build them, and then they also toss in the transcripts from talking to their finance team, internal or external And then they have a conversation because the AI will say,"What about this?
22:13
Brian: What about that?
22:14
Brian: Your vendor did not answer this question over here sufficiently.
22:17
Brian: They didn't talk about this over here, this issue you said was important to you." So validating the performance of internal or external team members is fantastic.
22:24
Brian: Finance, HR, admin, whatever.
22:26
Brian: As you get into that operational, that core offer that you're doing, that's where you gotta be very careful, but also make sure you're gathering the context.
22:35
Brian: Amateur AI people think the content is what matters,"Write me a resume." But the context, the more you become aware, you realize the context of how your team is asking questions really matters, and what they're asking your AI about, grabbing how they're asking it, and the follow-up conversations matters a great deal.
22:49
Brian: Not only does that tell you, oh, this particular staff person is laboring under these incorrect assumptions.
22:53
Brian: That's interesting.
22:54
Brian: But it helps you understand where is everyone in your company in terms of how AI fluent there are, they are, and where are they trying to get to?
22:59
Brian: Where is their vision of where their company's gonna go?
23:01
Brian: If you have a vision of your operations is gonna move from here to there, but then your team members have a opposite vision, that's gonna be a huge problem, and you may not get that from the content.
23:12
Brian: If you're all using it to run the same process and you're all writing sales plans with AI, but then you...
23:17
Brian: If you capture the context, now you can say,"Oh, I'm thinking AI's gonna be a core partner in all of our sales," and this person over here is thinking it's gonna be something they use to do some market research, but nothing more than that.
23:28
Brian: That context, hugely valuable if you have it, and hugely disastrous if you don't.
23:33
Brian: And critically, your competition is not gathering as much context as you are, at least hopefully.
23:37
Brian: So the closer you get to your core offer, get lots of context, and then yeah, it's great for creativity, but then tone down the percentage of AI used in the final work products.
23:47
Brian: Maybe sort- sort of thing like marketing and sales, having a lot of AI throughout the life cycle can be a really good idea.
23:53
Brian: But as you're getting closer to your core offer, yeah, a lot of AI at the first part of any question, any project.
23:58
Brian: But then as you're getting closer to shipping, narrow that sucker on down to where by the time the last few passes, you have a human in there.
24:04
Brian: The human has a flashlight.
24:05
Brian: They have a micrometer.
24:07
Brian: You're not trusting AI for anything on those last passes, and that just matters the more and more you get to that core offer.
24:12
Brian: That's my general take.
24:14
Colin: more valuable or the more, as you say, closer to your core offering you get, you've gotta be more and more skeptical, more, I guess, less trusting of the outputs or rather more, I guess
24:32
Brian: You can trust the LLMs want what's best for them, which is not what's best for you.
24:37
Brian: They have their own agenda, and it's not to serve you, the small business person.
24:40
Brian: It's not to serve you, the factory owner.
24:42
Brian: That's not their problem.
24:43
Brian: Their problem is how to go maximize their revenue.
24:45
Brian: and if your factory blows up, they don't care because you already signed something saying that your factory blowing up is not their problem.
24:52
Brian: It's not a problem at all if your factory blows up while using Groq.
24:54
Brian: For Groq.
24:55
Brian: They don't care.
24:56
Brian: They can't.
24:57
Brian: They're serving tens, hundreds of millions of people every day, and there's no way they're gonna care about you.
25:02
Colin: So how does-- I mean, a-and I-- while I agree, and I'm not disputing that, that alone, I can imagine, is enough for plenty of people out there to say,"Well, then," you know, throw their hands up, say,"We're not gonna do it then." You know?
25:17
Colin: That, that's huge risk.
25:19
Brian: is AI level five companies, my five levels of AI, fluency or AI development of a company, AI adoption.
25:25
Brian: Number one is AI forbidden like a bank.
25:27
Brian: You use AI, you go to jail.
25:29
Brian: Number two is AI fearful, which is you the staffer don't know if you can use AI or not.
25:33
Brian: Your bosses maybe use AI, but that's not really official.
25:35
Brian: AI level three is AI friendly, which is the boss is like,"Yeah, go use AI." And you're like,"I guess I'll use some AI." You talk about it.
25:41
Brian: Maybe there's some trainings.
25:42
Brian: Maybe there's a company account, maybe not, but you're not really gathering that context.
25:46
Brian: AI level four is the AI forward company where all the context is being gathered.
25:50
Brian: There's mandatory company accounts.
25:52
Brian: You're not letting anyone use AI that's outside of the company accounts, and you're just gathering all the context and using it as a good decision partner.
25:58
Brian: AI level five is AI first companies.
26:01
Brian: That's the ones that are gonna spring up overnight, where the humans are following the AI instead of the AI following the humans.
26:06
Brian: And the AI level five companies are why everyone has to use AI.
26:09
Brian: Because if you don't, at some point you're gonna get to work on Monday, and over the weekend somebody spun up a new company and they took out a chunk of your company.
26:16
Brian: They decapitated or amputated part of your company over a weekend because they were able to use AI to map out all your revenue streams, all of your vendors.
26:23
Brian: They talked to all of them.
26:25
Brian: And if you don't have some kind of AI supporting your company, you're gonna be there on Monday wondering,"Why is half my business gone?" And this has not happened yet as far as I know, but it's coming, and it's gonna be very, very weird.
26:36
Brian: It's gonna be the equivalent of what used to be was the private equity firm showed up in your town.
26:40
Brian: It bought one of the nine roofing companies in your town, and then it both flooded the market with advertising.
26:45
Brian: It also deeply discounted prices to drive half the companies out of business.
26:49
Brian: And once half the companies were gone, it then jacked up the prices.
26:52
Brian: Like we see with nursing homes.
26:53
Brian: PE has gutted nursing homes nationwide.
26:55
Brian: It's that, but 10 times or 100 times as fast, and it's gonna be the AI level five companies.
27:00
Brian: So just like you cannot ignore PE if PE is in an environment in which you're doing business, if you're a roofer or a nursing home, there's no way you can ignore PE that it exists.
27:07
Brian: You can't say,"Well, I'm never gonna sell, therefore it doesn't impact me." Not true.
27:12
Brian: it will impact everything around you, and unless you're the only provider in the middle of nowhere, it will absolutely impact your market.
27:17
Brian: I think AI's gonna be like that, where you can't say no, which is the problem.
27:21
Brian: there's no consent here.
27:22
Brian: There's you will use AI or you will get crushed.
27:24
Brian: And so that's why we're forced to make these terrible decisions around, I know Claude's gonna lie to me.
27:29
Brian: Maybe it lies to me less than ChatGPT, or I could become an AI architect and go get Ollama or whatever and go hope it goes well.
27:38
Brian: Open Cloud.
27:38
Brian: Here my friend's discussing Open Cloud on Facebook.
27:40
Brian: Good deal.
27:42
Colin: I, you know, and I guess that's the thing that, reaffirms my take is, machine learning is a phenomenal asset, phenomenal resource for companies that have it, that have implemented it and leverage it.
27:55
Colin: But it obviously doesn't have the conversational or unstructured piece that language models do.
28:01
Colin: And so I think if there's a way to bridge those together and use the structure and strength and accuracy that machine learning provides, and use-- if there's a way to tie that in better with LLMs, I think there's a big opportunity there.
28:20
Colin: But it just, it seems like there is such a wide moat right now between the two.
28:27
Colin: and the risk of hallucinations or just flat out lying from LLMs, that just creates uncertainty of, is it even using the real data from, you know, my convoluted neural networks, my, you know, really precise algorithms that, physics models that I've built?
28:44
Brian: It's a who do you trust problem.
28:45
Brian: You have to have a human you can trust.
28:47
Brian: Someone you can go up to their house, go up to their office, knock on their door and say,"This is broken." Who are you gonna trust to go fix stuff when it breaks?
28:54
Brian: That's the question.
28:56
Brian: You can get some progress by doing adversarial AIs, like I have Claude write me a code plan or a build plan, a Groq instance that's all it does is build critic.
29:06
Brian: That's all it does.
29:07
Brian: It goes and reviews the plan and it argues back and forth with Claude.
29:10
Brian: And then once Claude gets through that, then it goes and writes code.
29:13
Brian: Then the code goes over to the Groq code critic, who reviews that and then comes back.
29:16
Brian: And I'm in the middle of developing, or the beginning of developing, I think it's a nine-agent code factory that involves Claude, Groq, and Gemini, but they're each distinct agents arguing with each other.
29:25
Brian: So you can do some adversarial stuff that helps, but ultimately it's who do you trust and who is enough in front of you that you can say,"That person hasn't gotten eaten by the lions yet.
29:34
Brian: They're probably not an idiot.
29:36
Brian: Maybe I can do what they're doing." And that's, I think, what most business people have to do,'cause you don't have time to go learn all this stuff yourself.
29:42
Brian: You can't.
29:42
Brian: It's changing too fast.
29:44
Brian: I can't learn everything.
29:45
Brian: I know almost nothing about machine learning.
29:47
Brian: I know a fair amount about how to make Claude, Groq, and Gemini play together well to achieve business outcomes.
29:53
Brian: But I know almost nothing about, like, OpenClaw.
29:56
Brian: I'm completely ignorant.
29:57
Brian: Never used it.
29:59
Brian: Would it be good for me?
30:00
Brian: Well, of course, but there's an opportunity cost.
30:02
Brian: Machine learning.
30:03
Brian: From this talk, I think,"Oh, I should go, I should go, I should go spend 10 hours the rest of my day.
30:07
Brian: Cancel everything.
30:07
Brian: Machine learning all day.
30:09
Brian: Then I'll know something," right, Colin?
30:10
Brian: If I spend the next eight hours?
30:12
Brian: No, of course I won't.
30:13
Brian: What I will learn is, I now understand half of 1% of machine learning, and I better do machine learning tomorrow too, and the day after that.
30:21
Brian: Is it tied to any business objective?
30:22
Brian: No, it's tied to wanting to be less ignorant.
30:26
Brian: But the problem is, information is moving so fast you cannot be less ignorant.
30:29
Brian: That is no longer an option.
30:30
Brian: We're all steeped in ignorance forever around AI, but we have to use it anyways.
30:35
Colin: nobody can afford to, sit on the sidelines anymore, can they?
30:39
Brian: It's done.
30:40
Brian: It's a very weird business environment.
30:41
Brian: This has never happened before.
30:42
Brian: You could start a company today without a cell phone if you really wanted to in some industries.
30:46
Brian: You could opt out.
30:49
Colin: It's wild.
30:50
Colin: Wild world we live in.
30:52
Colin: Well, Brian, thank you.
30:53
Colin: This is, Oh, man, there, there are some scary rabbit holes I think we could go down with this, but, this was great.
30:59
Colin: I think I, probably more skeptical than when I started, and I was already a healthy skeptic, but in a good way, in a good way.
31:06
Brian: I like to share messages of hope.
31:08
Brian: and the hope is go find people you can trust.
31:11
Brian: How do you
31:11
Colin: hopeful I will become less skeptical in the future.
31:14
Colin: How's that?
31:17
Brian: think skepticism is very healthy when you're dealing with things we don't know about.
31:20
Brian: Do you remember the story about they initially said that, you couldn't have a train go over, I think it was 30 mi- or 40 miles an hour, people would evaporate or they like their, all their blood vessels would burst?
31:29
Brian: That was a real concern, and it had to do with no horse could go faster than 40 miles an hour.
31:34
Brian: Therefore, the scientists of the day assumed, well, humans can't either.
31:38
Brian: Turns out it's not true.
31:40
Brian: But it was precautionary principle, probably reasonable to let someone else go first.
31:45
Brian: Let them get on the train and then see what happens.
31:47
Brian: let them go settle Mars and then see what happens.
31:49
Brian: You miss out on some of the gains, but if you have an actual business you're trying to protect, if you have a legacy, maybe you should go do AI second.
31:57
Brian: Let someone else go do something first and see what happens.
32:02
Colin: I think that's a great note to end it on there.
32:05
Colin: Brian, thank you.
32:06
Colin: Wonderful as always.
32:07
Colin: For those of you watching and listening, thank you for tuning into the"Reforge" podcast.
32:12
Colin: Whether we're talking, AI, machine learning, or robotics on the factory floor, policy that moves the needle, or with people who are re-industrializing America, our goal is always the same: to prove that re-industrialization isn't just possible, it's happening right now, and it's pivotal.
32:29
Colin: If today's conversation got you fired up, hit subscribe or the follow button.
32:33
Colin: Leave us a rating and review wherever you listen.
32:36
Colin: Make sure you are helping us grow the Reforge nation with more builders, founders, policymakers, and the workers who are in the trenches with us.
32:44
Colin: Until next time, keep forging the future.
32:46
Colin: I'm Colin, and this is the"Reforge" podcast.
32:48
Colin: We'll see you next time
32:53
Thanks for tuning in to the Reforged podcast.
32:56
Make sure to like, subscribe and follow us on all of our social media@thereforgedpod.
33:02
We will see you next time