Latest / The Pragmatic Engineer / How AI is changing software engineering at Shopify with Farhan Thawar

Transcript

How AI is changing software engineering at Shopify with Farhan Thawar · The Pragmatic Engineer

0:00
SPEAKER_02: We've been using AI tools for a long time in engineering.
0:02
I believe we were the first company outside of GitHub to use
0:05
GitHub Copilot.
0:06
And the reason I know that is because when Thomas Domke became
0:09
the CEO of GitHub,
0:10
SPEAKER_02: the same day I emailed him saying I would like GitHub Copilot.
0:12
SPEAKER_02: This is 2021, a year before ChatGPT.
0:14
SPEAKER_02: He messaged me back saying it's not available for commercial use.
0:17
SPEAKER_02: And I said, that's not what I asked.
0:18
SPEAKER_02: I don't care what you have for commercial use.
0:21
I would like this deployed for all Shopify engineers as soon as
0:24
humanly possible.
0:24
I think it took them about a month and we were not charged for
0:27
two years because
0:27
SPEAKER_02: there was not a SKU to charge us.
0:29
SPEAKER_02: And we said in exchange, we'll give you lots and lots of feedback.
0:31
SPEAKER_02: And so we were using Copilot for a long time.
0:33
SPEAKER_02: And then we worked with them as we started looking at their roadmap.
0:36
SPEAKER_02: We then deployed Cursor internally as well.
0:38
SPEAKER_02: Actually, we were all very new to Cursor, about a year into Cursor.
0:41
SPEAKER_02: So we were kind of trying all these things.
0:42
SPEAKER_02: And as we try them,
0:43
SPEAKER_02: we want to see what's working,
0:44
SPEAKER_02: what's not working,
0:45
SPEAKER_02: and then we let more engineers use it.
0:46
The most interesting thing about Cursor is that the growth in
0:49
Cursor at Shopify is
0:50
SPEAKER_02: happening a lot outside of engineering and outside of R&D.
0:54
SPEAKER_02: Finance, sales, support, those are the teams using Cursor.
0:58
SPEAKER_00: What happens when a company goes all in on AI?
1:01
SPEAKER_00: Shopify did exactly this a few years ago,
1:03
SPEAKER_00: and Shopify's head of engineering,
1:05
SPEAKER_00: Farhan Tavar,
1:06
SPEAKER_00: told me exactly how it's going so far.
1:08
SPEAKER_00: In this episode,
1:09
SPEAKER_00: we discuss how Shopify works closely with AI Labs,
1:12
and why Farhan paired for an hour with an engineer at Antrofic
1:15
working on the Cloud
1:16
SPEAKER_00: Code team.
1:17
Why Shopify is planning to hire 1,000 interns in a year and how
1:20
they incorporated
1:20
SPEAKER_00: AI into their interview process.
1:22
Why Shopify has no cost limit on how much an engineer or team
1:25
can spend on AI
1:26
SPEAKER_00: tokens and many more.
1:28
SPEAKER_00: If you're interested to know what an AI-first tech company operates like,
1:31
SPEAKER_00: this episode is for you.
1:33
SPEAKER_00: This podcast was recorded as a live podcast at LDX3 in London.
1:37
If you enjoyed the podcast,
1:38
please do subscribe on your favorite podcast player and on
1:40
YouTube.
1:41
So welcome,
1:42
everyone,
1:42
to this very special live podcast with Farhan Tawar.
1:47
SPEAKER_00: And we're going to talk about AI at Shopify.
1:50
But before we start about AI,
1:52
I just wanted to talk a little bit about you,
1:54
Farhan.
1:54
SPEAKER_00: So I did a little bit of research.
1:56
SPEAKER_00: I talked with a couple of engineers at Shopify.
1:59
SPEAKER_00: And they told me, you know, your role is the head of engineering.
2:03
SPEAKER_00: And engineers told me that they see you in everything.
2:06
SPEAKER_00: You have reworked Shopify's intern hiring program.
2:09
SPEAKER_00: You've defined what Shopify does.
2:11
SPEAKER_00: In fact,
2:11
you even were in the weeds to deploying an internal hackfest to
2:16
more than getting
2:18
SPEAKER_00: the Wi-Fi right for a hackfest.
2:20
SPEAKER_00: So I'd like to know, what does a head of engineering do at Shopify?
2:23
SPEAKER_00: And specifically, what do you do?
2:24
SPEAKER_00: Or maybe, what do you not do?
2:25
SPEAKER_02: Yeah,
2:26
SPEAKER_02: so I think what's interesting about Shopify is that we use this line,
2:29
SPEAKER_02: we're not a swim lane company,
2:31
which means that we don't try to put people into these roles
2:34
where you are like,
2:35
SPEAKER_02: you're in product,
2:36
SPEAKER_02: so only think about product.
2:37
Or you're in engineering,
2:38
only think about how the code is written or architecture.
2:41
SPEAKER_02: We're very much just curious problem solvers.
2:43
SPEAKER_02: And so if something's broken,
2:45
SPEAKER_02: we expect curious people to go and look at the problem,
2:48
SPEAKER_02: even doesn't matter what their role is,
2:49
SPEAKER_02: and try to solve it.
2:51
SPEAKER_02: And so last year at our Shopify Summit,
2:53
SPEAKER_02: which is our employee event where 7,000 people come to one space,
2:57
SPEAKER_02: the Wi-Fi did not work super well.
2:59
And so I spent all my time in that summit for three days trying
3:02
to fix the Wi-Fi.
3:03
SPEAKER_02: I became known as the chief Wi-Fi officer.
3:05
SPEAKER_02: And so this year,
3:06
SPEAKER_02: which was two weeks ago,
3:08
SPEAKER_02: I deployed 300 Ubiquity APs,
3:12
SPEAKER_02: 800 switches in order to have a much,
3:16
SPEAKER_02: much smoother Wi-Fi experience,
3:18
which is how that all got started in terms of the team built
3:22
lots and lots of memes
3:23
SPEAKER_02: because they thought the Wi-Fi was not going to work,
3:24
SPEAKER_02: but it did work.
3:25
SPEAKER_02: And so I did publish all those memes even though
3:28
SPEAKER_00: I need to ask you because a lot of leaders would say,
3:31
SPEAKER_00: look,
3:31
SPEAKER_00: like,
3:32
SPEAKER_00: you know,
3:32
SPEAKER_00: your time is super valuable.
3:33
SPEAKER_00: You're kind of ahead of an organization.
3:35
SPEAKER_00: How many people are?
3:36
SPEAKER_00: 3,000.
3:37
SPEAKER_00: 3,000 people.
3:37
SPEAKER_00: You're kind of up there and your time is now,
3:39
SPEAKER_00: you know,
3:39
SPEAKER_00: if you measure it in dollars,
3:40
SPEAKER_00: it'll be expensive.
3:41
SPEAKER_00: Right.
3:41
SPEAKER_00: It's a lot better to just get a specialist to do these things.
3:45
SPEAKER_00: Why do you still do these things?
3:46
And do you disagree with that advice of like focus on the high
3:50
leverage things?
3:51
SPEAKER_00: Yeah.
3:51
SPEAKER_02: Yeah,
3:52
SPEAKER_02: so we don't love the,
3:53
SPEAKER_02: there's that notion of like hire smart people and get out of their way.
3:56
Instead,
3:56
what we say is like hire smart people and pair with them on
3:59
problems.
4:00
So the difference for us is we really like to bring in
4:02
obviously lots of smart
4:03
SPEAKER_02: people,
4:03
SPEAKER_02: hands-on people,
4:04
SPEAKER_02: and instead of just saying,
4:05
SPEAKER_02: hey,
4:05
you take this area and then go away and just solve it for me
4:08
and come back,
4:09
SPEAKER_02: we like to pair with them and say,
4:10
SPEAKER_02: you're smart,
4:10
SPEAKER_02: I'm smart,
4:11
SPEAKER_02: can we work on this problem together?
4:12
SPEAKER_00: This episode is brought to you by WorkOS.
4:15
SPEAKER_00: If you're building a SaaS app,
4:16
at some point your customers will start asking for enterprise
4:18
features like SAML
4:19
SPEAKER_00: authentication,
4:20
SPEAKER_00: SKIM provisioning,
4:21
SPEAKER_00: and fine-grained authorization.
4:23
SPEAKER_00: That's where WorkOS comes in,
4:25
SPEAKER_00: making it fast and painless to add enterprise features to your app.
4:28
SPEAKER_00: Their APIs are easy to understand,
4:30
SPEAKER_00: and you can ship quickly and get back to building other features.
4:33
WorkOS also provides a free user management solution called
4:36
AuthKit for up to 1
4:37
SPEAKER_00: million monthly active users.
4:39
It's a drop-in replacement for Auth0 and comes standard with
4:42
useful features like
4:42
SPEAKER_00: domain verification,
4:44
SPEAKER_00: role-based access control,
4:45
SPEAKER_00: bot protection,
4:46
SPEAKER_00: and MFA.
4:47
SPEAKER_00: It's powered by Radix components, which means zero compromises in design.
4:51
You get limited customizations as well as modular templates
4:54
designed for quick integrations.
4:56
SPEAKER_00: Today,
4:56
SPEAKER_00: hundreds of fast-growing startups are powered by WorkOS,
4:59
SPEAKER_00: including ones you probably know,
5:00
SPEAKER_00: like Cursor,
5:01
SPEAKER_00: Vercel,
5:01
SPEAKER_00: and Perplexity.
5:03
SPEAKER_00: Check it out at WorkOS.com to learn more.
5:06
SPEAKER_00: That is WorkOS.com.
5:08
SPEAKER_00: If you want to build a great product, you have to ship quickly.
5:12
SPEAKER_00: But how do you know what works?
5:14
SPEAKER_00: More importantly, how do you avoid shipping things that don't work?
5:18
SPEAKER_00: The answer, Statsik.
5:20
SPEAKER_00: Statsik is a unified platform for flags,
5:22
SPEAKER_00: analytics,
5:23
SPEAKER_00: experiments,
5:24
SPEAKER_00: and more,
5:24
combining five plus products into a single platform with a
5:27
unified set of data.
5:29
SPEAKER_00: Here's how it works.
5:31
SPEAKER_00: First, StatSick helps you ship a feature via FeatureFly or Config.
5:35
SPEAKER_00: Then it measures how it's working,
5:37
SPEAKER_00: from alerts and errors,
5:39
SPEAKER_00: to replays of people using that feature,
5:41
SPEAKER_00: to measurement of top-line impact.
5:43
SPEAKER_00: Then you get your analytics,
5:44
SPEAKER_00: user account metrics,
5:45
SPEAKER_00: and dashboards to track your progress over time,
5:48
SPEAKER_00: all linked to the stuff you ship.
5:50
SPEAKER_00: Even better,
5:50
SPEAKER_00: Statsik is incredibly affordable with a super generous free tier,
5:54
SPEAKER_00: a starter program with $50,000 of free credits,
5:57
SPEAKER_00: and custom plans to help you consolidate your existing spend on flags,
6:00
SPEAKER_00: analytics,
6:00
SPEAKER_00: or A-B testing tools.
6:02
SPEAKER_00: To get started, go to statsik.com slash pragmatic.
6:06
SPEAKER_00: That is S-T-A-T-S-I-G dot com slash pragmatic.
6:09
SPEAKER_00: Happy building.
6:11
SPEAKER_00: So this leads me really nice into the next thing I heard about you,
6:14
SPEAKER_00: which is at Shopify,
6:16
SPEAKER_00: you have gone kind of AI first.
6:18
SPEAKER_00: And we're going to talk a little bit about that.
6:19
SPEAKER_00: But one thing that you do is you have access to some of the top AI labs.
6:24
SPEAKER_00: And what you told me just before, that you pair with them.
6:27
You just told me that the other day,
6:28
you were pairing with Antropic engineers.
6:31
Can you tell me about this whole pairing and how you work with
6:33
these AI labs?
6:34
SPEAKER_02: Yeah, because again, we're like super curious problem solvers.
6:37
And so when there's something interesting happening in the
6:39
industry,
6:39
we want to be close to it.
6:41
SPEAKER_02: So obviously,
6:41
SPEAKER_02: ChatGPT comes out,
6:42
SPEAKER_02: and then Anthropics builds their model,
6:44
SPEAKER_02: Gemini has their model,
6:45
SPEAKER_02: Cohere has their model.
6:46
SPEAKER_02: We want to be as close to the leading edge as possible,
6:48
SPEAKER_02: and so we try to be close to those people and what they're working on.
6:52
SPEAKER_02: And one of the things we do is when something comes out,
6:54
SPEAKER_02: so in this example it was Cloud Code,
6:56
SPEAKER_02: we deployed it inside of Shopify,
6:58
SPEAKER_02: we saw people using it.
6:59
SPEAKER_00: This was in May, right?
7:00
SPEAKER_00: Three or four weeks ago.
7:00
SPEAKER_02: Exactly.
7:01
SPEAKER_02: Well, so Cloud Code came out before that, like let's say six months ago.
7:04
SPEAKER_02: And then we started spending time with Anthropic.
7:07
SPEAKER_02: I wanted to see how Anthropic was using Cloud Code internally.
7:10
SPEAKER_02: Similar to like OpenAI is now releasing codecs.
7:12
SPEAKER_02: I want to figure out how they're using it internally.
7:14
SPEAKER_02: And so what I did was I reached out to Anthropic and said,
7:16
SPEAKER_02: hey,
7:16
SPEAKER_02: I'd like to pair with one of your applied AI engineers.
7:19
SPEAKER_02: We spent an hour building a bunch of things.
7:21
SPEAKER_02: And I learned how he sees Cloud Code being used internally in Anthropic.
7:26
And I can take that back and say,
7:27
well,
7:27
here's how we're using it at Shopify.
7:28
SPEAKER_02: And we can together figure out where we think this thing can go.
7:31
SPEAKER_00: Is this just a matter of, like, you just ask them?
7:35
SPEAKER_02: Yeah, I mean, because we are Shopify, we're very lucky.
7:37
SPEAKER_02: People want to see how we work.
7:39
SPEAKER_02: We tend to work in an unorthodox way,
7:42
SPEAKER_02: and they also are interested in how we use it,
7:44
SPEAKER_02: but they also use it.
7:45
SPEAKER_02: And so it was more of a, again, a pairing.
7:47
SPEAKER_02: It was a true pairing.
7:47
SPEAKER_02: It wasn't like me as a customer trying to push them and say,
7:50
SPEAKER_02: I really want,
7:51
SPEAKER_02: it was,
7:51
SPEAKER_02: we have a shared Slack channel.
7:52
SPEAKER_02: I pinged in there saying,
7:53
SPEAKER_02: hey,
7:53
who'd want to pair with me for an hour on something I'm
7:55
building inside Shopify?
7:56
SPEAKER_02: Somebody put their hand up on applied AI engineer and said,
7:59
we'll show you how we work and we show you how we work and we
8:02
can together figure
8:03
SPEAKER_02: out where we think this product could go.
8:05
SPEAKER_02: So I do that with a lot of the providers.
8:07
And as you're pairing,
8:07
is it just you pairing and then you later tell people?
8:10
SPEAKER_00: Yeah,
8:11
SPEAKER_02: so I might record the session and then share it internally to say,
8:14
SPEAKER_02: hey,
8:14
SPEAKER_02: I just did a pairing session with Anthropic.
8:16
SPEAKER_02: They might use it for a case study that they're building.
8:19
SPEAKER_02: But the idea is,
8:20
SPEAKER_02: again,
8:20
SPEAKER_02: because I'm just curious,
8:21
SPEAKER_02: when somebody says something,
8:22
SPEAKER_02: I'm like,
8:22
SPEAKER_02: oh,
8:22
SPEAKER_02: how does that work?
8:24
SPEAKER_02: How can we use this?
8:25
SPEAKER_02: And so I think it's a way for us to just learn.
8:28
And one recent thing that I just wanted to go into before we
8:32
get into some more of
8:32
SPEAKER_00: the AI things,
8:33
SPEAKER_00: it's a really fun fact that most people don't know.
8:35
SPEAKER_00: You just came out of a code red for seven months.
8:39
SPEAKER_00: I think this is the first time we're looking, what was this code red?
8:42
SPEAKER_00: What was it about?
8:43
SPEAKER_00: And what did people outside and inside of Shopify see of it?
8:46
SPEAKER_02: Yeah,
8:46
so we were internally seeing a lot of things that weren't
8:50
bubbling to the surface,
8:51
SPEAKER_02: but that can actually affect the operation of a system.
8:54
SPEAKER_02: So you would typically call it tech debt,
8:56
SPEAKER_02: but we had a lot of signals showing that the tech debt was growing.
8:59
SPEAKER_02: Not just the general ones,
9:00
SPEAKER_02: like taking longer to update a piece of software,
9:04
SPEAKER_02: or build a version 2 of a product,
9:06
SPEAKER_02: or people having a hard time maintaining some of the software stack.
9:10
We saw things even,
9:11
not externally,
9:11
but internally that were shooting out signals.
9:13
SPEAKER_02: Lots of exceptions growing.
9:16
Even in some cases,
9:16
because we build things at the very low level of the stack.
9:18
SPEAKER_02: So for example, we're core contributors to Ruby.
9:21
SPEAKER_02: We patch MySQL.
9:23
I believe we have the second largest MySQL fleet in the world
9:25
outside of meta.
9:26
And we were seeing things like seg faults and things like that
9:29
that I was not
9:30
SPEAKER_02: willing to feel like we should move on and just build features as normal.
9:33
And so we actually got together,
9:36
spent,
9:36
we didn't think it was going to be seven months.
9:37
SPEAKER_02: We thought it was going to maybe be three months.
9:39
SPEAKER_02: And we took something between like 30 to 50% of engineering.
9:43
SPEAKER_02: We didn't have an actual number.
9:44
SPEAKER_02: We just said, these things have to not grow anymore, like exceptions.
9:48
SPEAKER_02: Unique exception counts have to shrink.
9:50
SPEAKER_02: There should be zero seg faults.
9:52
SPEAKER_02: We should understand everything that's happening in the system.
9:53
And so we spent almost seven months doing that over the last,
9:57
you know,
9:57
from November till now.
9:59
SPEAKER_00: How did you make sure that you fixed the right thing?
10:00
SPEAKER_00: Did you talk to the team saying, hey, bring up the things?
10:03
SPEAKER_00: And also, how did you decide when you will stop?
10:05
SPEAKER_00: Because whenever I hear these code reds,
10:07
SPEAKER_00: it's always a lot of companies go to code yellows,
10:09
SPEAKER_00: code reds,
10:10
SPEAKER_00: but sometimes they can drag on forever.
10:12
SPEAKER_02: Yeah, so we had multiple metrics that we used.
10:15
SPEAKER_02: One was literally what I said,
10:16
SPEAKER_02: exception counts,
10:17
SPEAKER_02: unique exception counts,
10:18
SPEAKER_02: and segfaults,
10:19
SPEAKER_02: which literally would say segfaults are at zero now,
10:21
SPEAKER_02: and exceptions should be going down.
10:23
SPEAKER_02: But we also looked at other metrics that we have.
10:26
SPEAKER_02: We want to have four nines of reliability across our different surfaces,
10:30
SPEAKER_02: storefronts,
10:31
SPEAKER_02: merchant admin,
10:32
SPEAKER_02: point of sale.
10:33
SPEAKER_02: We looked at all these things.
10:34
SPEAKER_02: And we also used the 28-day rolling average of those things.
10:37
SPEAKER_02: And once those were all green and we saw that,
10:40
SPEAKER_02: again,
10:40
SPEAKER_02: segfaults was zero and unique exceptions stopped and was shrinking,
10:44
we then felt like we were in a place where we can start
10:45
building features again.
10:47
SPEAKER_00: And now going on to AI.
10:49
SPEAKER_00: So you're a huge early adopter of AI coding tools.
10:53
Can you tell me on what tools you started to use and what
10:56
you're using right now
10:58
SPEAKER_00: and what you're liking,
11:00
SPEAKER_00: what engineers are liking specifically,
11:02
SPEAKER_00: to focus more on software engineering,
11:03
SPEAKER_00: but maybe even outside of software?
11:05
SPEAKER_02: Yeah, so we've been using AI tools for a long time in engineering.
11:08
So I believe we were the first company outside of GitHub to use
11:12
GitHub Copilot.
11:13
And the reason I know that is because when Thomas Domke became
11:17
the CEO of GitHub,
11:18
SPEAKER_02: the same day I emailed him saying I would like GitHub Copilot.
11:20
SPEAKER_02: This is 2021, a year before ChatGPT.
11:23
SPEAKER_02: He messaged me back saying, it's not available for commercial use.
11:26
SPEAKER_02: And I said, that's not what I asked.
11:27
SPEAKER_02: I don't care what you have for commercial use.
11:30
I would like this deployed for all Shopify engineers as soon as
11:33
humanly possible.
11:34
SPEAKER_02: I think it took them about a month,
11:35
SPEAKER_02: and we were not charged,
11:36
SPEAKER_02: I think,
11:37
SPEAKER_02: for two years because there was not a SKU to charge us.
11:40
SPEAKER_02: And we said, in exchange, we'll give you lots and lots of feedback.
11:43
SPEAKER_02: And so we were using Copilot for a long time.
11:45
SPEAKER_02: And then we worked with them as we started looking at their roadmap.
11:48
SPEAKER_02: We then deployed Cursor internally as well.
11:50
SPEAKER_02: Actually, we were very new to Cursor, about a year.
11:51
SPEAKER_00: Really?
11:52
SPEAKER_00: Yeah.
11:53
SPEAKER_00: They were a tiny startup back then.
11:54
SPEAKER_02: Yeah.
11:55
SPEAKER_02: I think that's super late.
11:56
SPEAKER_02: You're saying it super early.
11:57
SPEAKER_02: I feel like it's very late to Cursor.
11:59
SPEAKER_02: We knew about Cursor for a long time.
12:00
SPEAKER_02: We like to have one tool at Jobify for the things that we're doing.
12:03
SPEAKER_02: And so we don't like to have multiple tools.
12:05
SPEAKER_02: So we have Figma.
12:07
SPEAKER_02: We don't try to have lots of other visualization tools.
12:10
SPEAKER_02: We have MySQL.
12:11
SPEAKER_02: We don't have other databases.
12:12
SPEAKER_02: So we try to focus on one tool.
12:14
SPEAKER_02: And so our bet was VS Code.
12:15
But because of AI's proliferation and we don't know what's
12:18
going to happen,
12:19
We then started making a change in our stance around trying
12:22
more tools than just one.
12:24
SPEAKER_02: Maybe we consolidate, maybe we don't.
12:26
SPEAKER_02: But right now, we have Cursor and VS Code as our two AI tools.
12:30
And then Cloud Code came as the agentic workflow,
12:32
and we started using Cloud Code.
12:34
SPEAKER_02: We tried Devon last year.
12:35
SPEAKER_02: So we are kind of trying all these things.
12:37
SPEAKER_02: And as we try them,
12:38
SPEAKER_02: we want to see what's working,
12:39
SPEAKER_02: what's not working,
12:39
SPEAKER_02: and then we let more engineers use it.
12:41
The most interesting thing about Cursor is that the growth in
12:44
Cursor at Shopify is
12:45
SPEAKER_02: happening a lot outside of engineering.
12:47
And outside of R&D,
12:49
finance,
12:50
sales,
12:51
support,
12:52
those are the teams using Cursor.
12:54
SPEAKER_00: What do they use it for?
12:55
So what's happening is because Cursor is actually so ubiquitous
12:58
on building things,
12:59
they're using it to build MCP servers outside of regular
13:04
engineering domains.
13:05
SPEAKER_02: So for example, Salesforce.
13:08
SPEAKER_02: Google Calendar, Gmail, Slack.
13:11
They're building MCP servers to access those services and then
13:16
building homepages
13:17
SPEAKER_02: for themselves.
13:17
SPEAKER_02: If you're a salesperson,
13:18
SPEAKER_02: connect to Salesforce,
13:20
SPEAKER_02: connect to Google Calendar,
13:21
SPEAKER_02: connect to my email,
13:22
SPEAKER_02: and tell me what opportunity I should be working on.
13:24
SPEAKER_00: So just going back,
13:26
SPEAKER_00: with an MCP server,
13:27
SPEAKER_00: when you have a service,
13:28
SPEAKER_00: you can put an MCP server,
13:29
SPEAKER_00: and then you can activate it,
13:30
SPEAKER_00: for example,
13:31
SPEAKER_00: with Cursor or with an agent.
13:33
SPEAKER_02: Exactly.
13:33
SPEAKER_00: Do engineers help build that MCP server?
13:36
SPEAKER_02: Sometimes, yes, but a lot of times, no.
13:37
Now that these services are coming out with their own MCP
13:40
agents so that you can
13:40
SPEAKER_02: literally just download GitHub's MCP server,
13:43
SPEAKER_02: right?
13:43
SPEAKER_02: You don't have to actually build it yourself.
13:44
SPEAKER_02: But a lot of times,
13:45
SPEAKER_02: they're actually just building it themselves,
13:47
SPEAKER_02: and they're watching videos,
13:49
SPEAKER_02: and they're actually just deploying it on their own without any engineer
13:51
SPEAKER_02: intervention at all.
13:52
SPEAKER_02: Now, it's end of one, right?
13:53
SPEAKER_02: They're building software for each individual person,
13:55
SPEAKER_02: not like infrastructure for all sorts of people.
13:57
SPEAKER_00: Yeah.
13:57
SPEAKER_00: Does this remind you of anything?
13:59
SPEAKER_00: You've been around in the tech industry for a long time,
14:01
SPEAKER_00: but the fact that the non-technical people are now kind of building,
14:05
SPEAKER_00: do you see any parallels from the past?
14:07
SPEAKER_00: Did this happen before, or is this new?
14:10
SPEAKER_02: This coding part is new,
14:11
although if you remember like Wissy Wig editors and like people
14:15
starting to like
14:15
SPEAKER_02: self-serve.
14:17
I think it reeks of that,
14:18
but they are,
14:19
I mean,
14:20
they're not reading the code,
14:21
right?
14:21
SPEAKER_02: They're really much vibe coding.
14:23
SPEAKER_02: And if it doesn't work, they just delete and start over.
14:25
SPEAKER_02: But even if you read Anthropix documentation on Cloud Code,
14:28
like they say one third of the time you can kind of get
14:30
something working with one
14:31
SPEAKER_02: shot.
14:32
SPEAKER_02: And so we're starting to see that.
14:33
SPEAKER_02: And people are getting used to the fact that if it doesn't work,
14:35
SPEAKER_02: they just delete and start over and try to get it working.
14:37
SPEAKER_00: I find this really interesting on how this is happening.
14:41
SPEAKER_00: Is it changing the dynamics with engineering?
14:42
SPEAKER_00: Because one thing I noticed when working at places like Uber,
14:46
SPEAKER_00: everyone was a bit jealous of engineering.
14:48
SPEAKER_00: They wanted engineering resources.
14:49
SPEAKER_00: They couldn't get it.
14:50
SPEAKER_00: They really wanted these things.
14:52
SPEAKER_00: Do you see any of this change?
14:54
SPEAKER_02: I'm going to say yes and no.
14:55
SPEAKER_02: So there's a weird thing happening whereby,
14:57
SPEAKER_02: so let's say you're a PM,
14:58
SPEAKER_02: and let's say you're a technical PM.
15:00
You now probably have enough knowledge to vibe code some new
15:05
feature you've been
15:06
SPEAKER_02: waiting for engineering to build for you yourself.
15:08
The question is,
15:10
can you submit a PR,
15:12
and should the PR be accepted by engineering?
15:15
SPEAKER_02: I'm going to ask you first, and I'll tell you what I think.
15:17
SPEAKER_00: Well,
15:18
I think you can always submit a PR and engineering will tell
15:22
you why it can either
15:24
SPEAKER_00: be accepted,
15:25
SPEAKER_00: if it's good,
15:26
SPEAKER_00: or what is missing.
15:28
SPEAKER_00: May that be conceptual, may it be coding standards and all those things.
15:31
But at least,
15:33
you know,
15:33
and this feedback for them,
15:35
they're seeing what this person wants to do.
15:36
SPEAKER_00: I think it should be a great thing.
15:38
SPEAKER_02: Yes, I'll say yes with one caveat.
15:40
And the caveat is the problem with VibeCoding today is you
15:44
might generate 10,000
15:45
lines of code for a very simple application feature that now
15:50
the burden on
15:51
SPEAKER_02: engineering to read the 10,000 lines is there.
15:54
SPEAKER_02: So what we say is,
15:56
SPEAKER_02: yes,
15:57
SPEAKER_02: you can submit a PR,
15:58
SPEAKER_02: but you have to understand the code you are writing.
16:03
SPEAKER_02: Exactly.
16:04
SPEAKER_02: Before you submit it.
16:04
SPEAKER_02: Because I can do the same thing and I can say,
16:06
SPEAKER_02: hey,
16:07
SPEAKER_02: I'm going to write a blog post,
16:09
SPEAKER_02: generate 20 pages,
16:11
SPEAKER_02: and then send it to you and be like,
16:12
SPEAKER_02: hey,
16:12
SPEAKER_02: edit this and post it on your blog.
16:13
SPEAKER_02: And now you have the burden of having to.
16:15
SPEAKER_02: Now,
16:15
SPEAKER_02: if I said I read it all and it is my voice and I agree with it,
16:18
SPEAKER_02: then you might be more likely to read it.
16:20
The interesting thing that we're probably going to see this
16:21
from with open source
16:22
projects because it's so easy to use AI tools and they're going
16:25
to get a lot of
16:26
SPEAKER_00: these things.
16:26
SPEAKER_00: I think policies like this are interesting.
16:28
SPEAKER_00: Exactly.
16:28
Now,
16:28
I wanted to get your take on one thing,
16:30
which is going around in social media.
16:32
There's a lot of people predicting it's the end of SaaS because
16:35
people can create
16:36
SPEAKER_00: their own SaaS.
16:36
Now,
16:37
what you told me is some of your non-technical people are
16:40
creating small solutions.
16:42
SPEAKER_02: For themselves.
16:42
SPEAKER_00: For themselves.
16:44
SPEAKER_00: How do you see this potentially changing SaaS vendors?
16:48
SPEAKER_00: And you're also someone who actually buys a lot of SaaS vendors.
16:53
SPEAKER_00: You understand them.
16:54
SPEAKER_00: Do you think this changes anything or not really?
16:56
SPEAKER_00: Is it just more personal software?
16:57
SPEAKER_02: Yeah,
16:58
I'd like to think we buy less SaaS than most companies only
17:00
because we try to
17:01
SPEAKER_02: consolidate down.
17:02
That being said,
17:03
there is this notion,
17:04
we're in this middle zone right now.
17:05
The middle zone is basically this idea that you can vibe code
17:09
something maybe for yourself.
17:11
It's unclear whether you can vibe code as a platform or vibe
17:14
code something that
17:15
SPEAKER_02: gets into the infrastructure layer of what you're building because
17:18
SPEAKER_02: you do really need to understand,
17:20
SPEAKER_02: again,
17:20
SPEAKER_02: like you mentioned it earlier,
17:21
SPEAKER_02: like you might be putting yourself in a very precarious situation.
17:24
SPEAKER_02: If you're building on top of something and building on top of something,
17:25
SPEAKER_02: you don't understand what you're building,
17:27
SPEAKER_02: right?
17:27
SPEAKER_02: Maybe it's a prototype, that's fine, or a proof of concept.
17:29
SPEAKER_02: But if you're building infrastructure for the internet,
17:32
SPEAKER_02: you likely want to understand how that's being built.
17:34
SPEAKER_02: So that is not, we're not there yet.
17:36
SPEAKER_02: Is it coming?
17:37
SPEAKER_02: Yes.
17:37
SPEAKER_02: It's for sure coming where someone can vibe code something in the,
17:40
SPEAKER_02: you know,
17:40
SPEAKER_02: Anthropic or OpenAI or Gemini or who cares,
17:44
SPEAKER_02: who model.
17:45
build something that's actually architecturally elegant and the
17:47
right architecture
17:48
SPEAKER_02: for what you need.
17:49
SPEAKER_02: We're not there today.
17:50
SPEAKER_02: So where we are is this notion of still human in the loop,
17:55
SPEAKER_02: and still start over,
17:56
SPEAKER_02: and this is not what I meant,
17:58
SPEAKER_02: and prompt engineering,
17:59
SPEAKER_02: English as the programming language.
18:01
And so I do think you can get yourself very far,
18:04
but I'm not worried yet about SaaS.
18:06
I don't know if we should be worried,
18:07
by the way,
18:08
because let's take another view.
18:10
SPEAKER_02: How much software do we think should there be in the world?
18:13
SPEAKER_02: Probably
18:15
SPEAKER_02: 10,000 times as much as there is now?
18:16
SPEAKER_02: 100,000?
18:17
SPEAKER_02: There's a lot of software that should be in the world,
18:19
SPEAKER_02: and we are satisfying .0001% of the demand.
18:23
So now that everybody can generate software,
18:26
we should welcome them into the software world.
18:29
We believe everyone should be writing software just like my
18:31
sales team is,
18:32
SPEAKER_02: and everybody becomes more productive.
18:34
SPEAKER_02: I'm not yet worried about the software industry.
18:35
SPEAKER_02: I still believe this is Jevon's paradox.
18:37
SPEAKER_02: The more we get, the more we want.
18:40
SPEAKER_00: Yeah,
18:40
a really good analogy that has stuck with me is Simon Willison
18:43
was saying,
18:44
SPEAKER_00: how has the filming industry changed with this?
18:47
Everyone has,
18:48
this used to cost like $5,000 or $10,000 a few years ago,
18:51
this camera.
18:52
SPEAKER_00: Everyone has one of their pockets.
18:54
SPEAKER_00: And there's still a professional movies industry,
18:56
SPEAKER_00: but now there's a bunch of,
18:58
SPEAKER_00: there's YouTube,
18:59
SPEAKER_00: there's TikTok,
18:59
SPEAKER_00: there's all these things.
19:00
So it has become bigger,
19:02
but the barrier for the pros has not really changed.
19:05
SPEAKER_00: Dune and some of these amazing movies are so hard to make.
19:08
SPEAKER_02: All right, Jevons Paradox.
19:10
SPEAKER_02: The more you have, the more you want.
19:11
SPEAKER_02: We all want to be filmmakers now.
19:13
We can be filmmakers and then there's still going to be people
19:15
at the high end and the low end.
19:16
SPEAKER_02: I would say the controversial,
19:19
SPEAKER_02: maybe controversial,
19:19
SPEAKER_02: spicy take is that these phones actually benefited the experts more.
19:26
SPEAKER_02: Meaning the camera like base,
19:27
SPEAKER_02: you know,
19:27
like the being able to pull out your iPhone and take a take a
19:31
video Actually
19:32
SPEAKER_02: benefited those who are really really good at it already.
19:34
Yeah And the same thing is maybe potentially true of vibe
19:37
coding It's like the AI
19:39
SPEAKER_02: agents are gonna help the best engineers more than the mediocre engineer
19:44
SPEAKER_00: So there was a memo that went around in the media about a month ago.
19:47
SPEAKER_00: Toby sent a memo.
19:49
SPEAKER_00: It was meant for internal consumption.
19:50
SPEAKER_00: It was about reflexive AI usage,
19:53
SPEAKER_00: and I guess it got leaked,
19:55
SPEAKER_00: and then Toby posted the whole thing.
19:56
And this is about saying that,
19:58
hey,
19:58
everyone is expected to use AI at Shopify.
20:02
SPEAKER_00: Can you tell me why you felt that memo was sent out?
20:05
SPEAKER_00: What was their response internally?
20:07
And has it changed anything,
20:09
or was it just kind of stating what was there already?
20:11
SPEAKER_02: Yeah,
20:11
it was a fundamental statement that this is something new and
20:15
it was meant to get
20:16
SPEAKER_02: everyone to move from passive to active,
20:19
SPEAKER_02: right?
20:19
SPEAKER_02: So,
20:19
SPEAKER_02: yes,
20:20
SPEAKER_02: there's AI in the world and ChatGPT and all these things existed,
20:23
but we started building infrastructure internally to make it
20:25
easier to use and we
20:27
SPEAKER_02: basically said the following.
20:28
SPEAKER_02: You don't have to use AI in your workflow,
20:30
but we're going to expect that you have this plethora of tools
20:33
available to you,
20:34
such that the expectation is your impact is going to be
20:37
evaluated as if you had the
20:38
SPEAKER_02: tool.
20:39
SPEAKER_02: So for example,
20:41
SPEAKER_02: imagine not having Excel or Google Sheets,
20:43
SPEAKER_02: and you had to do a complicated analysis.
20:46
SPEAKER_02: You don't have to use those tools.
20:47
SPEAKER_02: You could use a pen and paper,
20:48
SPEAKER_02: but we're going to expect that you have these tools available,
20:50
and we expect the level of analysis to be treated as if you had
20:53
these tools.
20:54
SPEAKER_02: And so by us doing it,
20:55
what changed internally was actually probably more of a change
20:59
outside of R&D like
21:00
SPEAKER_02: I mentioned.
21:00
SPEAKER_02: R&D was already leaning in.
21:02
SPEAKER_02: Now, of course, there were some folks who were waiting for, I don't know.
21:05
SPEAKER_00: R&D is engineering.
21:06
SPEAKER_02: R&D is engineering product data and design.
21:08
SPEAKER_02: They were more likely to use these tools because they tend to be more
21:11
SPEAKER_02: forward-looking and wanting to use the latest and greatest.
21:15
SPEAKER_02: However, some people were waiting.
21:16
It changed their shape because we said,
21:19
hey,
21:19
look,
21:19
we're going to expect that you use these.
21:21
SPEAKER_02: But outside of R&D, we saw it even more.
21:23
Again,
21:23
sales,
21:24
finance,
21:24
customer success,
21:25
help centers,
21:26
all those roles started using AI as well.
21:28
SPEAKER_02: And we, again, gave them the infrastructure to use that.
21:30
SPEAKER_02: So we have an LLM proxy.
21:32
SPEAKER_02: It has all of the APIs and all the models available.
21:35
You don't have to be worried that your personal information is
21:38
being leaked to a model.
21:39
SPEAKER_02: You can kind of use those tools.
21:40
SPEAKER_00: Let's talk about these things.
21:41
SPEAKER_00: Let's talk about the tools that you either created or adopted early.
21:45
SPEAKER_00: I heard, can you talk a bit more about this, LLMproxy?
21:48
SPEAKER_00: And also, I heard something about MCP.
21:50
SPEAKER_02: Yeah.
21:51
SPEAKER_02: So we are, when I talk to people, so let me start with LLMproxy.
21:54
The problem at the beginning was people would go to ChatGPT or
21:58
Gemini or Claude,
21:59
SPEAKER_02: and they would like,
21:59
SPEAKER_02: you know what?
21:59
SPEAKER_00: I don't want to put customer data there.
22:00
SPEAKER_02: You don't want to put customer data there, or even employee data.
22:02
SPEAKER_02: You want to use it to write your employee reviews.
22:04
SPEAKER_02: You don't want to put,
22:04
SPEAKER_02: like,
22:05
SPEAKER_02: Simon is having problems with this,
22:07
SPEAKER_02: and put their first name and last name.
22:08
SPEAKER_02: And here's the project they worked on, and the code name's leaked.
22:11
So we wanted to quickly have an internal LLM proxy,
22:14
plus we built on top of LibreChat.
22:17
which is the open source chat product,
22:20
and we're core contributors to that as well.
22:22
SPEAKER_02: And with the LLM proxy, you're sure to be using the enterprise APIs.
22:25
SPEAKER_02: We can check to see which,
22:27
SPEAKER_02: you get a token,
22:28
SPEAKER_02: you ask for a token,
22:28
SPEAKER_02: we can see which teams are using how much,
22:31
SPEAKER_02: like what cost is being incurred by that team or by person.
22:34
We have a leaderboard where we actively celebrate the people
22:37
who use the most tokens.
22:39
SPEAKER_02: Oh, really?
22:39
SPEAKER_02: Yeah,
22:39
SPEAKER_02: because we want to make sure that they are,
22:41
SPEAKER_02: if they're doing great work,
22:42
SPEAKER_02: of course,
22:43
SPEAKER_02: if they script,
22:44
SPEAKER_02: like that's not what I mean.
22:45
SPEAKER_02: Yeah.
22:45
SPEAKER_02: I mean, they're doing great work with AI.
22:47
SPEAKER_02: I want to see how did they spend $1,000 a month in credits by a cursor.
22:51
SPEAKER_02: Maybe that's something they're building something great,
22:53
SPEAKER_02: and they have an agent workforce underneath them.
22:55
SPEAKER_02: So we have this proxy,
22:56
SPEAKER_02: and again,
22:56
SPEAKER_02: it's used to build anything that you want to build,
22:58
SPEAKER_02: and you can ask for a token and use it.
23:00
SPEAKER_02: And then we are very early on MCP.
23:03
SPEAKER_02: We're on the MCP steering committee.
23:05
SPEAKER_02: We are fans of MCPing all the things.
23:08
And so when I talk to people about how to get access to some
23:11
piece of data inside the company,
23:13
we quickly will spin up an MCP endpoint for them so that they
23:16
can just use it.
23:17
SPEAKER_02: So, for example, our internal wiki is called The Vault.
23:21
SPEAKER_02: It has information about all of our projects that are happening.
23:24
SPEAKER_02: It has information on every internal talk that has been done.
23:27
SPEAKER_02: It takes the transcripts of all those.
23:28
SPEAKER_02: You can say something like, when did the point of sale at Shopify launch?
23:32
SPEAKER_02: And it will come back and it will say,
23:33
SPEAKER_02: oh,
23:34
SPEAKER_02: in 2012,
23:35
SPEAKER_02: there was a hackathon where the point of sale was started.
23:38
SPEAKER_02: Based on wiki docs and docs?
23:40
SPEAKER_02: And in the 2013 Q1 board letter,
23:42
SPEAKER_02: it was mentioned,
23:43
SPEAKER_02: and it would tell you all of the history of what happened in Shopify.
23:46
And so we do that by just having all you do is put up an MCP,
23:48
and now the LibreChat has access.
23:50
SPEAKER_00: It's like an internal perplexity or something like that.
23:52
SPEAKER_02: Exactly.
23:52
SPEAKER_02: And so we basically,
23:53
SPEAKER_02: any internal document is crawled by this thing,
23:57
SPEAKER_02: and now you can have access to it by MCP.
23:58
SPEAKER_02: And so any new system comes online,
24:00
SPEAKER_02: we put an MCP in front of it,
24:01
SPEAKER_02: and now everybody has access to it.
24:03
SPEAKER_00: This episode is brought to you by Sonar,
24:04
SPEAKER_00: the creators of SonarQ,
24:06
the industry standard for integrated code quality and code
24:09
security that is trusted
24:10
SPEAKER_00: by over 7 million developers at 400,000 organizations around the world.
24:15
SPEAKER_00: Human-written or AI-generated,
24:17
SonarQ reviews all code and provides actionable code
24:20
intelligence that developers
24:21
SPEAKER_00: can use to improve quality and security early in the development process.
24:26
SPEAKER_00: Sonar recently introduced SonarCube Advanced Security,
24:28
enhancing Sonar's core security capabilities with software
24:31
composition analysis and
24:32
SPEAKER_00: advanced static application security testing.
24:36
SPEAKER_00: With advanced security,
24:37
development teams can identify vulnerabilities in third-party
24:39
dependencies,
24:40
SPEAKER_00: ensure license compliance,
24:41
SPEAKER_00: and generate software bills of materials.
24:44
It's no wonder that SonarCube has been rated the best static
24:46
code analysis tool for
24:47
SPEAKER_00: five years running.
24:49
SPEAKER_00: So join millions of developers from organizations like Microsoft,
24:52
SPEAKER_00: NVIDIA,
24:53
SPEAKER_00: Mercedes-Benz,
24:53
SPEAKER_00: Johnson & Johnson,
24:54
SPEAKER_00: and eBay,
24:55
SPEAKER_00: and supercharge your developers to build better,
24:57
SPEAKER_00: faster with Sonar.
24:59
SPEAKER_00: Visit sonarsource.com slash pragmatic security to learn more.
25:03
SPEAKER_00: And how many MCP servers do you have?
25:05
SPEAKER_00: Do you have them?
25:06
SPEAKER_00: Probably two dozen.
25:08
SPEAKER_00: Two dozen, yeah.
25:08
SPEAKER_00: And they're growing quickly.
25:09
SPEAKER_02: They're growing because people are building a new thing.
25:11
SPEAKER_02: Again, there's one for Salesforce.
25:12
SPEAKER_02: There's ones for Figma.
25:15
SPEAKER_02: There's an MCP in front of anything.
25:16
SPEAKER_02: But I tell anybody who's looking at building AI systems internally,
25:20
SPEAKER_02: the way to make this easy is start putting MCP in front of everything.
25:23
SPEAKER_02: And now your data is accessible.
25:25
SPEAKER_02: We had to happen to use LibreChat.
25:26
SPEAKER_02: You might use something else.
25:27
SPEAKER_02: But you don't have to be in even in cursor, right?
25:29
SPEAKER_02: You just use the chat interface.
25:30
SPEAKER_00: Yeah, you can do chat, you can do the LLMs.
25:33
SPEAKER_00: And I like how you spun up basically a platform team to build the infra,
25:37
SPEAKER_00: to do monitoring,
25:38
SPEAKER_00: to worry about things like a security,
25:41
SPEAKER_00: decide on models,
25:42
SPEAKER_00: all those things.
25:43
SPEAKER_00: Does this team have a name?
25:44
Is it an AI platform or is it just a platform team,
25:47
infra team just took it over?
25:48
SPEAKER_02: Yeah, data platform.
25:49
SPEAKER_02: I don't know.
25:50
SPEAKER_02: I'm trying to figure out because they're inside the data platform team.
25:52
SPEAKER_02: I don't know if they have a name outside.
25:53
SPEAKER_02: I call them LLM proxy, but...
25:55
And was this more like you or someone that said,
25:57
like,
25:57
hey,
25:58
we need to do this?
25:58
SPEAKER_00: Or are they just like, hey, this is a good idea.
26:00
SPEAKER_00: Let's kind of bottom up or top down?
26:02
It was more top down because we felt like this was going to
26:04
grow internally.
26:05
SPEAKER_02: We wanted to make sure that we could.
26:06
SPEAKER_02: We didn't want people signing up for their own personal tokens.
26:09
SPEAKER_02: And it's also a good way for us to see, hey, this team is growing.
26:12
SPEAKER_02: Is the usage growing?
26:13
SPEAKER_02: What's happening there?
26:14
SPEAKER_02: This person's usage?
26:15
SPEAKER_00: Yeah,
26:16
because number one pushback I get from people or even
26:18
engineering directors working
26:19
SPEAKER_00: at companies is the data security issue.
26:22
And obviously when you host it yourself,
26:24
it's good,
26:24
but then you need the infrastructure.
26:25
SPEAKER_00: So it sounds like, at what point did you decide?
26:28
SPEAKER_00: Was this a month ago?
26:30
SPEAKER_00: Was this more than a year ago to actually invest?
26:32
SPEAKER_02: Probably, I mean, I would say every week it's getting better, right?
26:34
But I would say probably a year ago we started thinking about
26:37
it this way.
26:37
SPEAKER_02: We had the LLM proxy.
26:38
SPEAKER_02: When ChatGPT came out,
26:40
SPEAKER_02: We actively have a warning if you go to ChatGBT and it says,
26:43
SPEAKER_02: hey,
26:44
SPEAKER_02: by the way,
26:45
SPEAKER_02: you can go to the internal version.
26:47
SPEAKER_02: That was for a long time.
26:48
SPEAKER_00: Now,
26:49
SPEAKER_00: you mentioned cost,
26:49
SPEAKER_00: and I am hearing stories of CTOs,
26:53
SPEAKER_00: engineering leaders,
26:54
SPEAKER_00: saying,
26:54
SPEAKER_00: OK,
26:55
SPEAKER_00: we are paying for GitHub Copilot.
26:56
SPEAKER_00: I don't know.
26:57
SPEAKER_00: It's $10 or $20 per month for engineer,
26:59
SPEAKER_00: but we have all these seats,
27:00
SPEAKER_00: or maybe we have for everyone.
27:01
And they're telling me Cursor feels a bit too expensive because
27:04
they're now
27:04
SPEAKER_00: assuring $30 or $35,
27:06
SPEAKER_00: and we're not going to go there.
27:07
SPEAKER_00: And generally, there's a big pushback on cost.
27:10
SPEAKER_00: People are saying,
27:10
SPEAKER_00: look,
27:11
SPEAKER_00: AI should make engineers more productive,
27:13
SPEAKER_00: but it's kind of ridiculous to spend $1,000 with an engineer.
27:17
SPEAKER_00: You think about cost differently.
27:18
SPEAKER_00: Tell me about it.
27:18
Yeah,
27:19
I did a tweet storm on this because I think that people are
27:22
looking at it differently.
27:24
SPEAKER_02: So think about it this way.
27:25
If I could give you a tool that could make your engineering
27:28
team more productive by
27:30
SPEAKER_02: even 10%,
27:31
SPEAKER_02: would you pay for it?
27:32
SPEAKER_02: The answer is yes.
27:33
SPEAKER_02: Would you pay $1,000 a month for it is the question.
27:35
SPEAKER_02: And my hypothesis here is $1,000 a month is too cheap.
27:39
like it's if i can get 10 more it is way too cheap like we
27:43
should be five thousand
27:44
dollars a month like it's insane that people are looking at
27:46
this like a hundred
27:47
dollars a month whatever now i did meet somebody who said their
27:50
engineers were
27:50
spending ten thousand dollars a month per engineer and i said i
27:53
want to talk to
27:53
them because i want to see either they're doing something very
27:55
very smart or very
27:56
very stupid but i want to meet them and so i literally said
27:59
please introduce me
28:00
because i i'm i'm unclear how you can use that much like either
28:03
they've got fields
28:03
SPEAKER_02: of agents building amazing things
28:05
SPEAKER_02: or something is going very, very wrong.
28:07
But I would like to understand,
28:08
because there's something interesting happening.
28:10
SPEAKER_02: And you should not be penny-pinching on AI tools,
28:14
SPEAKER_02: because actually the productivity gains are there.
28:17
We don't know what they are,
28:18
because we actually don't know how to gauge developer
28:20
productivity.
28:20
SPEAKER_02: But it is clear that there's something happening there,
28:24
SPEAKER_02: and you should be very open with your pocketbook there.
28:27
SPEAKER_02: And so, again, we celebrate those who spend more.
28:29
SPEAKER_02: And, of course, we talk to them, and what did they do?
28:31
SPEAKER_02: So they were getting the product.
28:32
Do I understand that even though you cannot measure it right
28:34
now or maybe not as
28:35
SPEAKER_00: accurately,
28:36
you believe that there's productivity gains and you believe
28:38
that people need to use
28:39
SPEAKER_00: it to get these?
28:41
SPEAKER_00: So you're kind of treating it as an investment right now for a year,
28:44
SPEAKER_00: for two years,
28:45
SPEAKER_00: whatever that may be.
28:46
SPEAKER_00: And then you're going to figure it out.
28:48
SPEAKER_02: Probably forever.
28:48
I mean,
28:49
again,
28:49
you wouldn't turn off Spellcheck and GrammarCheck and G Suite.
28:53
SPEAKER_02: You wouldn't turn off these tools, right?
28:55
SPEAKER_02: I mean,
28:55
SPEAKER_02: there's probably a debate on when you turn off Slack,
28:57
SPEAKER_02: because that could be a productivity failure.
29:00
But you're seeing the gains,
29:03
and I don't think that we should be penny-pinching.
29:05
SPEAKER_02: Now, over time, we might realize there's an equilibrium.
29:08
SPEAKER_02: I'm like, oh, this is the amount of tokens people should be using.
29:11
SPEAKER_02: Actually,
29:11
SPEAKER_02: if anybody's using Cursor here,
29:12
SPEAKER_02: the number one thing I'll tell you is,
29:13
SPEAKER_02: please move away from the default model.
29:15
Because the default model is nowhere close to actually using a
29:17
sophisticated model.
29:18
SPEAKER_02: And by default, cursor puts you there.
29:21
SPEAKER_02: And so we tell people, please move off the small models.
29:24
SPEAKER_02: Move into something more bigger.
29:26
SPEAKER_02: We'll even say things like, please use O1 Pro for this project.
29:29
SPEAKER_02: Or please use O3 Pro because
29:32
SPEAKER_00: They're more expensive.
29:33
SPEAKER_00: They're more expensive.
29:34
SPEAKER_02: Actually, I'll go even further.
29:35
SPEAKER_02: Mikhail, the CTO, will say something like the following.
29:38
SPEAKER_02: If you don't pay personally for O1 Pro or Gemini Ultra or Claude,
29:45
SPEAKER_02: whatever it's called,
29:46
SPEAKER_02: Advanced,
29:47
SPEAKER_02: the $200 a month one,
29:49
SPEAKER_02: you are crazy because you can afford it.
29:51
SPEAKER_02: And actually,
29:52
you are missing out on actually all of the progress happening
29:54
in LMs because you
29:55
SPEAKER_02: are just stuck to the $10 a month or $20 a month chat GPT.
29:59
SPEAKER_02: That's how far he would go.
30:00
SPEAKER_00: One interesting thing that I've heard when I talk with engineers,
30:03
SPEAKER_00: they're saying leadership is leading by example.
30:05
SPEAKER_00: Toby is hacking,
30:07
SPEAKER_00: and if you go on his Twitter or social media,
30:09
SPEAKER_00: you're going to see all the stuff that he's doing even a year ago.
30:12
SPEAKER_00: I want to ask you,
30:14
SPEAKER_00: what are you doing on the side or experimenting or playing with it,
30:17
SPEAKER_00: and what tools are you excited about?
30:19
SPEAKER_02: Yeah,
30:19
SPEAKER_02: so when I was pairing with that engineer on Anthropic,
30:24
SPEAKER_02: I was building something for myself.
30:25
SPEAKER_02: Yeah, I was building a couple of things.
30:27
SPEAKER_00: It's a great way to use company time, I guess.
30:28
SPEAKER_00: Well,
30:29
SPEAKER_02: I was like,
30:29
SPEAKER_02: let me see what they can build,
30:30
SPEAKER_02: because we didn't want to go super deep into the Shopify API.
30:33
But we do build a lot of things that,
30:35
you know,
30:35
one was actually a commerce flow.
30:37
I was trying to build a commerce flow via,
30:38
like,
30:39
operator,
30:39
where you can actually record
30:41
SPEAKER_02: the browser session,
30:43
and I wanted to see how the session was storing credit card
30:47
information or secure
30:50
SPEAKER_02: tokens as we were getting through a commerce flow.
30:52
SPEAKER_02: And I was trying to figure out,
30:53
I was trying to build almost a subscription product without
30:56
using our subscription
30:56
SPEAKER_02: product from Shopify.
30:58
And so it was an interesting experiment where we were trying to
31:00
see how do we store
31:01
SPEAKER_02: those credentials,
31:02
SPEAKER_02: especially when you want to store like credit card.
31:04
SPEAKER_02: Is it stored in the cloud already?
31:05
SPEAKER_02: Is it stored by the vendor?
31:06
SPEAKER_02: Are we storing it in the browser session?
31:08
SPEAKER_02: That's how I kind of stay as close to it as possible.
31:10
SPEAKER_02: I use cursor pretty often to build my own workflows.
31:13
SPEAKER_02: In this case, it was cloud code.
31:14
SPEAKER_02: We actually even use, I don't know if you've heard of Gumloop.
31:17
Gumloop is an automation platform that is very interesting at
31:20
its own layers.
31:21
SPEAKER_02: Here's one example.
31:22
SPEAKER_02: We tried to build.
31:23
SPEAKER_02: I did something super dumb.
31:25
I was like,
31:26
let's figure out when the next SanDisk Extreme Pro 8-terabyte
31:30
drive comes out.
31:32
SPEAKER_02: We tried to build it in Cloud Code.
31:33
SPEAKER_02: It took us an hour.
31:33
We did not get there because it actually tried to do something
31:37
at the wrong layer.
31:38
SPEAKER_02: It was trying to read the JSON,
31:40
and it was trying to get the image from the SanDisk side and
31:42
figure out if the
31:43
SPEAKER_02: 8-terabyte drive is out.
31:44
SPEAKER_02: It was actually quite complicated.
31:45
SPEAKER_02: We built it in two minutes in Gumloop.
31:47
SPEAKER_02: Not because they're different.
31:47
SPEAKER_02: What did Gumloop do?
31:48
SPEAKER_02: So Gumloop is basically, like, it was browser-based.
31:51
SPEAKER_02: We said,
31:52
SPEAKER_02: go to this website,
31:53
SPEAKER_02: look at the,
31:54
SPEAKER_02: like,
31:54
SPEAKER_02: go through the search box,
31:55
SPEAKER_02: like,
31:55
SPEAKER_02: using English commands.
31:57
And they came back right away and said,
31:58
there's no 8 terabyte drive available.
31:59
SPEAKER_02: And I said, OK, cool.
32:00
SPEAKER_02: Run it every week and send me an email.
32:02
SPEAKER_02: It was just, they're different tools for different jobs.
32:05
SPEAKER_02: And an hour with cloud.
32:06
SPEAKER_02: Again, nothing wrong with Anthropic Cloud Code.
32:07
SPEAKER_02: I use it all the time.
32:08
SPEAKER_02: It was not meant to do this kind of, like, web scraping.
32:11
SPEAKER_02: Because what does it do?
32:12
SPEAKER_02: It tries to build a web scraper.
32:13
SPEAKER_02: Versus Gumloop, it already knows how to scrape websites.
32:16
SPEAKER_02: So it's actually just a different layer of the stack.
32:18
And what I would encourage people to do is figure out what
32:20
you're trying to do.
32:21
And we have a lot of automation happening in Gumloop now
32:23
because people have to do something.
32:25
SPEAKER_02: I want to scrape a LinkedIn profile.
32:27
SPEAKER_02: I want to find out what platform this company is using behind the scenes.
32:31
SPEAKER_02: I want to scrape a website.
32:32
SPEAKER_02: All of these things, it's built for the right thing at the right level.
32:34
SPEAKER_00: You have a lot bigger tool set.
32:36
SPEAKER_02: Exactly.
32:36
SPEAKER_02: So you should figure out where you want to work.
32:39
SPEAKER_02: If you're writing code, plot code.
32:40
SPEAKER_02: If you're doing WebSkyscraper, use Gumloop.
32:43
SPEAKER_00: So far, it was AI, AI, AI.
32:45
SPEAKER_00: And when this letter came out,
32:47
SPEAKER_00: externally,
32:48
SPEAKER_00: there has been a lot of some speculation saying,
32:50
SPEAKER_00: hmm,
32:50
is this a sneaky way for Shopify to freeze headcount because it
32:55
says you shouldn't
32:55
SPEAKER_00: hire unless you can check if you can do it with AI.
32:58
SPEAKER_00: But then you told me something interesting.
33:00
I'm not sure how big Shopify is,
33:02
but you told me that you're applying to hire 1,000 interns.
33:05
Can you put that in context with the size of Shopify and tell
33:08
me why are you hiring
33:10
SPEAKER_00: interns where a lot of companies are saying,
33:12
SPEAKER_00: let's not hire,
33:14
SPEAKER_00: entry-level people because we have AI.
33:16
SPEAKER_00: What are you seeing that others are not seeing?
33:18
SPEAKER_02: Yeah, so a couple of things.
33:19
One is there's this real notion of generations as they come up
33:23
in different,
33:25
that graduate from different programs out of university or even
33:28
coming from high
33:28
SPEAKER_02: school.
33:29
SPEAKER_02: And you really want to be close to the next generation of people.
33:32
SPEAKER_02: One, they know things that you don't know.
33:35
SPEAKER_02: And so we restarted our intern program in full thrust this year.
33:38
SPEAKER_02: So last year we had like 25 interns a term.
33:41
SPEAKER_02: I convinced Toby to hire 1,000 interns this term,
33:43
mostly because I felt that they would be more AI reflexive than
33:49
everybody else.
33:49
SPEAKER_02: That was our hypothesis.
33:51
SPEAKER_02: So my tweet actually said,
33:52
SPEAKER_02: I want you to come to work with an LLM and a brain,
33:55
SPEAKER_02: not one or the other.
33:56
SPEAKER_02: And the idea was that bringing these like centaurs,
34:00
SPEAKER_02: AI centaurs,
34:00
SPEAKER_02: they know how to work with an LLM to solve their whatever workflow it is,
34:04
SPEAKER_02: whether it's a finance person or an engineering intern.
34:07
And we really focused on them coming in and changing our
34:10
internal culture.
34:11
SPEAKER_02: And so 1,000 engineers over the year meant like 350 a term.
34:14
SPEAKER_02: And what we saw there was, one, they're excited, they're hardworking.
34:19
SPEAKER_02: We also have them as a cohort, meaning they come to the office.
34:21
SPEAKER_02: We're a remote company,
34:23
but we have the interns come in because we felt like the
34:25
younger people need to
34:25
have younger people around them to work together versus just
34:28
being remote in their
34:29
SPEAKER_02: condo or whatever.
34:30
SPEAKER_02: We're changing the culture that way as well.
34:32
SPEAKER_02: And we did see this.
34:32
SPEAKER_02: They come, they have their reflexive.
34:34
SPEAKER_02: We saw the same thing in mobile.
34:35
SPEAKER_02: You were in mobile as well, and so was I in my last decade of work.
34:39
SPEAKER_02: And I hired lots of interns because they grew up with mobile phones.
34:42
And so this next generation is growing up with the internet and
34:44
they're growing up
34:45
SPEAKER_02: with phones and they're growing up with LLMs.
34:47
SPEAKER_02: So we wanted them to come in and change our culture around that axis.
34:50
SPEAKER_00: So do you feel like people are learning from
34:52
SPEAKER_02: We always were.
34:54
SPEAKER_02: Interns are the secret weapon.
34:55
SPEAKER_02: You always learn more from the interns.
34:57
SPEAKER_02: People think about internships as community service.
34:59
SPEAKER_02: Oh, that's so great, Shopify.
35:00
SPEAKER_02: You're helping the next generation.
35:01
SPEAKER_02: We don't do it for that.
35:02
SPEAKER_02: We do it to learn from the interns.
35:04
SPEAKER_02: That is always the reason.
35:05
SPEAKER_02: So if you don't have an intern program,
35:07
SPEAKER_02: an early talent program,
35:07
SPEAKER_02: I encourage you,
35:08
you should be bringing those people in because you will learn
35:10
from them more than
35:11
SPEAKER_02: you will get out of them.
35:12
SPEAKER_02: You will take a net positive benefit from them.
35:14
And of course,
35:15
for those who tweet at me and say,
35:16
please don't hire interns.
35:18
SPEAKER_02: You have to pay them.
35:18
SPEAKER_02: I'm like, we pay them.
35:20
SPEAKER_02: I don't know why people think that you don't.
35:21
SPEAKER_02: We pay them very well, actually.
35:23
SPEAKER_02: And we do hire,
35:25
I think I've hired almost 100 in this year from the intern pool
35:29
to come into our
35:30
SPEAKER_02: program.
35:30
SPEAKER_02: And we're very much thinking about the only way to get into Shopify as an
35:34
SPEAKER_02: entry-level engineer is through the intern program.
35:36
SPEAKER_00: I'm fully with you with on the paying.
35:37
I never understood why companies are cheaping out on this
35:40
because sometimes interns
35:41
SPEAKER_00: will reject some of the offers because it doesn't cover the living costs,
35:45
SPEAKER_00: which is wild.
35:45
SPEAKER_00: Yes, of course.
35:46
SPEAKER_00: You have to pay them.
35:47
SPEAKER_00: Actually, one of the best intern, it was another company's loss.
35:50
SPEAKER_00: And in Amsterdam, they offered 500 euros per month, no housing.
35:53
SPEAKER_00: And that intern would have loved to work there.
35:56
SPEAKER_00: What an experience.
35:57
SPEAKER_00: But they applied to Uber because they said it's just too low.
35:59
SPEAKER_00: And then we gave them a new grad salary.
36:01
SPEAKER_00: They could afford housing.
36:02
SPEAKER_00: They could afford to travel.
36:03
SPEAKER_00: Of course.
36:04
SPEAKER_00: And it was...
36:05
SPEAKER_00: that ad intern could have done amazing things at that company,
36:07
SPEAKER_00: and now this person is still senior at Uber,
36:09
SPEAKER_00: you know,
36:10
SPEAKER_00: been there for five years,
36:10
SPEAKER_00: so,
36:11
SPEAKER_00: you know,
36:11
SPEAKER_00: their loss.
36:12
SPEAKER_02: So think about it,
36:13
SPEAKER_02: 350 engineers,
36:14
SPEAKER_02: interns to about 3,000 engineers,
36:16
SPEAKER_02: about 10% at any one point,
36:18
SPEAKER_02: yeah.
36:18
So if I joined Shopify as either a software engineer or an
36:23
engineering manager,
36:24
can you tell me
36:25
SPEAKER_00: how stuff gets done, first as an engineer.
36:29
SPEAKER_00: Like, what tools do people use?
36:32
SPEAKER_00: I'm assuming it's just no longer, you know, VS Code, Code Review.
36:35
SPEAKER_00: There's going to be, like, AR tools here and there.
36:37
SPEAKER_02: Yeah,
36:37
SPEAKER_02: well,
36:37
SPEAKER_02: first on,
36:39
we have a very specific way of writing software to the point of
36:43
we built our own
36:44
SPEAKER_02: project management system.
36:45
SPEAKER_02: Like, we don't... Oh, really?
36:47
SPEAKER_02: We don't use linear.
36:48
SPEAKER_02: We don't use, yeah.
36:49
SPEAKER_02: And the reason is,
36:50
SPEAKER_02: is because we believe that there's that line,
36:53
SPEAKER_02: like,
36:53
SPEAKER_02: first you make the tool,
36:54
SPEAKER_02: and then the tool makes you.
36:56
SPEAKER_02: And so we believe that.
36:58
SPEAKER_02: And so we make the tool so that it can make us.
37:00
SPEAKER_02: Versus, and it's nothing against those amazing tools.
37:02
SPEAKER_02: I'm a big fan.
37:03
SPEAKER_02: Like, linear is beautiful.
37:04
SPEAKER_02: And I've known Jira for a long time.
37:06
SPEAKER_02: I'm a pivotal frackle guy myself, being a pivotal, ex-pivotal person.
37:10
SPEAKER_02: But we do believe that we have a specific way of building software.
37:13
And if we use one of those tools,
37:14
we would be adopting someone else's way.
37:17
And so instead,
37:18
we have a tool called GSD,
37:19
which stands for Get Shit Done.
37:21
SPEAKER_02: And it is a tool that allows us,
37:24
SPEAKER_02: it's a program management tool,
37:25
SPEAKER_02: like a product management tool,
37:26
SPEAKER_02: which allows us to think about what are we building,
37:28
SPEAKER_02: who's on the team,
37:29
SPEAKER_02: what's the latest weekly update.
37:30
SPEAKER_02: It has metrics.
37:31
SPEAKER_02: It actually pulls in PR reviews,
37:32
SPEAKER_02: so you can see it pulls in PRs,
37:34
SPEAKER_02: and you can see there's activity happening on this project.
37:36
SPEAKER_02: Here are the core contributors.
37:37
SPEAKER_02: Here's how long it's been going on for.
37:39
SPEAKER_02: This tool is how you kind of get work done.
37:41
SPEAKER_02: And you're forced to write an update every week.
37:43
SPEAKER_02: And now, actually, we have an AI.
37:44
SPEAKER_02: Here's an interesting one for you.
37:46
SPEAKER_02: We have an AI tool,
37:46
which will pull in the latest PRs and the latest conversations
37:51
from the Slack
37:51
SPEAKER_02: channel.
37:52
SPEAKER_02: And it'll write an update for you.
37:54
SPEAKER_02: And then you can look at it and say, this looks good.
37:56
SPEAKER_02: And you can also tell it, what did I not think about?
37:57
SPEAKER_02: And say, please emphasize this.
38:00
SPEAKER_02: And it'll help you write the project management update every week.
38:02
SPEAKER_00: So this is interesting.
38:03
I want to push you a little bit on this,
38:04
because I think there's two schools of thoughts here.
38:07
SPEAKER_00: One says your weekly update.
38:09
SPEAKER_00: I appreciate that you're doing it.
38:10
The weekly update,
38:11
the point should be for you to look back on what you did.
38:14
SPEAKER_00: Obviously, everyone hates doing this, by the way.
38:17
SPEAKER_00: And by AI doing it, would you not lose that kind of reflection?
38:22
SPEAKER_02: Well, this is why we pair you with the AI.
38:24
SPEAKER_02: So we do auto-publish it if you don't do anything.
38:29
But if you auto-publish it and don't do anything,
38:31
and we look at the stats,
38:33
there is a little bit of loss from you because you are now
38:36
losing out on the
38:36
context that you were supposed to gain as the champion of the
38:39
project by looking
38:39
SPEAKER_02: deeply.
38:40
SPEAKER_02: However,
38:41
SPEAKER_02: we expect everyone,
38:42
SPEAKER_02: just like a PR written by AI,
38:44
SPEAKER_02: to have read the code and read the update.
38:46
SPEAKER_00: Yeah, so you're responsible.
38:47
SPEAKER_02: You're responsible.
38:48
SPEAKER_02: And so we're early days in this.
38:49
SPEAKER_02: We're literally three weeks in on this project,
38:52
SPEAKER_02: and we may revert it and say,
38:53
SPEAKER_02: by the way,
38:53
SPEAKER_02: it turns out everybody stopped
38:55
SPEAKER_02: looking at what was happening week to week.
38:57
SPEAKER_02: But we also,
38:58
SPEAKER_02: every six weeks,
38:59
SPEAKER_02: at a company level,
39:01
SPEAKER_02: with Toby,
39:02
SPEAKER_02: go through every project in the company,
39:04
SPEAKER_02: and we see if we think that it's on track,
39:06
SPEAKER_02: not on track,
39:07
SPEAKER_02: should be kept working on,
39:08
SPEAKER_02: has the right resources,
39:09
SPEAKER_02: is it not too long,
39:10
SPEAKER_02: is it aimed in the right direction?
39:11
You better know what's in there because literally we're going
39:13
to review it at the
39:13
SPEAKER_02: leadership level.
39:14
SPEAKER_00: Yeah.
39:15
SPEAKER_00: I think you have no... I like how... Because I feel like...
39:20
With AI,
39:21
I kind of want it to automate the stuff that I really don't
39:23
want to do.
39:24
SPEAKER_00: And ironically,
39:25
SPEAKER_00: it's really good at automating stuff that we like to do sometimes,
39:27
SPEAKER_00: which is coding.
39:28
SPEAKER_00: But writing an update or writing documentation,
39:31
SPEAKER_00: I don't think as an engineer we loved doing that.
39:33
SPEAKER_02: And so as an engineer... Again, we're trying to reduce toil.
39:36
SPEAKER_02: If we find it reduces actual context, we will revert.
39:41
SPEAKER_00: I love the experimentation and going back and forth.
39:44
SPEAKER_00: As an engineering manager, what additional things might people use?
39:48
SPEAKER_00: Do people build their own tools to stay on top of things?
39:51
SPEAKER_00: Do people go deeper because of...
39:53
SPEAKER_02: Yeah,
39:53
SPEAKER_02: as an engineering manager and a director,
39:55
SPEAKER_02: we try to surface,
39:56
SPEAKER_02: again,
39:56
SPEAKER_02: metrics so that you can see how your team is doing.
39:58
SPEAKER_02: So,
39:58
SPEAKER_02: for example,
39:59
SPEAKER_02: we have tools that allow you to see things like focus time of your team,
40:02
SPEAKER_02: how often are they in meetings,
40:05
SPEAKER_02: AI adoption.
40:06
SPEAKER_02: Are they using AI tools or not?
40:08
SPEAKER_02: We're trying to surface this information.
40:10
SPEAKER_02: How many of your people are on a GSD project versus not?
40:12
Because maybe a project ended and they're freed up to work on
40:15
something else.
40:16
SPEAKER_02: So we try to surface that.
40:17
SPEAKER_02: And engineering managers too,
40:18
SPEAKER_02: we try to get them to be,
40:19
SPEAKER_02: a lot of the best ones come from IC land.
40:21
SPEAKER_02: They started as ICs.
40:22
SPEAKER_02: So we try to get them to do,
40:24
SPEAKER_02: if they're hired outside,
40:26
SPEAKER_02: start as an IC at Jobify before they move into management.
40:29
SPEAKER_00: Oh, here's a fun one that I think we should talk about.
40:32
SPEAKER_00: You mentioned something super interesting to me.
40:33
SPEAKER_00: When you're hiring engineering directors and above,
40:35
SPEAKER_00: in the past,
40:36
SPEAKER_00: it was the usual interviews,
40:37
SPEAKER_00: you know,
40:38
SPEAKER_00: culture,
40:38
SPEAKER_00: strategy,
40:39
SPEAKER_00: all that stuff.
40:40
You added a coding interview for every single engineering
40:43
director and above hire.
40:44
SPEAKER_00: Can you tell me about this?
40:45
SPEAKER_02: Yeah, so it's interesting.
40:46
SPEAKER_02: It's maybe shocking to folks.
40:48
SPEAKER_02: And I know Rodney's sitting there.
40:50
SPEAKER_02: He used to work for me.
40:51
SPEAKER_02: I did the pairing interview with him.
40:52
SPEAKER_02: And so what happens is it is shocking for VPs,
40:55
SPEAKER_02: especially,
40:55
SPEAKER_02: to be like,
40:56
SPEAKER_02: whoa,
40:56
SPEAKER_02: there's a coding interview?
40:57
SPEAKER_02: I'm like,
40:57
SPEAKER_02: yeah,
40:57
because we believe that the best leaders here were ones who
41:01
were not running away
41:01
SPEAKER_02: from coding.
41:02
SPEAKER_02: They just felt like they got better leverage from running a team.
41:05
SPEAKER_02: They still are deeply in love with technology,
41:07
SPEAKER_02: and they still,
41:07
SPEAKER_02: in Rodney's case,
41:08
SPEAKER_02: still codes on the weekend,
41:09
SPEAKER_02: right?
41:09
SPEAKER_02: And so it worked out super well.
41:11
SPEAKER_02: But our whole idea is that you're not running away.
41:14
You're running towards technology,
41:15
and this is just a better way for you to express it.
41:17
SPEAKER_02: So I pair with the candidates,
41:19
SPEAKER_02: and they also see that even though I'm not writing code every day,
41:22
SPEAKER_02: I'm still deep in the weeds of technology.
41:24
SPEAKER_02: I still love technology, and I still want to talk about technical topics.
41:27
SPEAKER_02: And so we pair.
41:27
SPEAKER_02: And some people believe that that's not the best for them.
41:30
And there's lots of great companies out there where that's not
41:32
the requirement.
41:33
But at Shopify,
41:34
we believe people should be as close to the details as
41:35
possible.
41:36
SPEAKER_02: And we are such a tech nerd company.
41:38
SPEAKER_02: Like, again, our salespeople are vibe coding now.
41:40
SPEAKER_02: We want our engineering leaders to be coding as well.
41:43
SPEAKER_02: And so that doesn't mean coding day to day.
41:44
SPEAKER_02: But you should understand code and how code works.
41:46
SPEAKER_02: And a lot of it comes back.
41:47
SPEAKER_02: The muscle memory of coding will come back in these pairing interviews.
41:50
SPEAKER_00: Yeah,
41:51
and I guess especially now we have these coding tools when you
41:54
can generate
41:54
SPEAKER_00: something as long as you can look through it.
41:56
SPEAKER_00: You're in control.
41:57
SPEAKER_00: Well,
41:57
SPEAKER_02: this is the best part about copilots is that a candidate comes,
42:00
SPEAKER_02: the copilot generates tons of code,
42:02
SPEAKER_02: and now I'm like,
42:03
SPEAKER_02: great.
42:03
SPEAKER_02: Is that good code?
42:04
SPEAKER_02: Bad code?
42:05
SPEAKER_00: I love... Oh, hold on.
42:06
SPEAKER_00: So you're using AI during your interview process.
42:09
SPEAKER_00: Yes, yes.
42:10
SPEAKER_00: Oh, you're not running away from it?
42:11
SPEAKER_00: No.
42:11
SPEAKER_00: You know,
42:12
SPEAKER_00: one of the interesting stories that we just learned,
42:14
SPEAKER_00: Cursor has disbanded on the entry process.
42:16
SPEAKER_00: They're the AI tool company.
42:18
SPEAKER_00: Right.
42:18
SPEAKER_00: So you're embracing it.
42:19
SPEAKER_00: We're embracing it.
42:20
SPEAKER_00: Okay, how's it working?
42:21
SPEAKER_00: Tell me.
42:21
I love it,
42:21
because what happens now is the AI will sometimes generate pure
42:24
garbage.
42:25
So you're screen sharing and you say literally use that
42:26
everything you want.
42:27
SPEAKER_00: Yeah, you're screen sharing.
42:27
SPEAKER_00: And they're using, I say, we let them use whatever they want.
42:30
SPEAKER_02: Here's what I'll say.
42:30
If they don't use a co-pilot,
42:32
they usually get creamed by someone who does.
42:34
SPEAKER_02: So they will have no choice but use a co-pilot.
42:36
SPEAKER_02: Sometimes I will shadow an interview and you do the questions myself.
42:39
I've never seen them with a co-pilot and send it to the
42:41
interviewer and say,
42:42
SPEAKER_02: please mark my assignment as well against a candidate.
42:45
SPEAKER_02: I have not lost yet.
42:46
SPEAKER_02: If they don't have a co-pilot, they will lose.
42:49
SPEAKER_02: But when they do have a co-pilot,
42:50
SPEAKER_02: I love seeing the generated code because I want to ask them,
42:53
SPEAKER_02: what do you think?
42:54
SPEAKER_02: Is this good code?
42:55
SPEAKER_02: Is this not good code?
42:55
SPEAKER_02: Are there problems?
42:56
SPEAKER_02: And I've seen engineers,
42:57
SPEAKER_02: for example,
42:58
SPEAKER_02: when there's something very easy to fix,
43:01
SPEAKER_02: they won't fix it.
43:02
SPEAKER_02: They will try to prompt to fix it.
43:04
SPEAKER_02: And I see, are you really an engineer?
43:06
SPEAKER_02: I get the nuance of just prompt and prompt and prompt,
43:10
SPEAKER_02: but sometimes it's right there,
43:11
SPEAKER_02: and they will not prompt.
43:12
SPEAKER_02: I'm like, change the one character, and they won't change it.
43:15
SPEAKER_02: And I'm like, OK, so they're interested.
43:17
SPEAKER_02: I don't want you to be 100% AI coder.
43:18
SPEAKER_02: I want you to be like 90 or 95.
43:20
I want you to be able to go in and look at the code and say,
43:23
oh yeah,
43:23
there's a line.
43:24
SPEAKER_02: That's wrong.
43:25
SPEAKER_00: So I want to ask you,
43:26
what does a standout senior or staff engineer today look at
43:29
Shopify and what has
43:30
SPEAKER_00: changed,
43:31
SPEAKER_00: let's say,
43:31
SPEAKER_00: three years ago when we didn't have any of these tools?
43:34
I'm trying to prod here how you think AI or AI tool usage might
43:39
have changed of
43:39
SPEAKER_00: what we expect an amazing engineer.
43:42
SPEAKER_00: And you think of the person who you think is this fantastic person.
43:46
SPEAKER_02: Well,
43:46
I would say the thing that's changing is I'm seeing some of my
43:48
engineers now
43:49
actually use these AI tools to do the infrastructure that
43:52
they've always wanted to
43:52
SPEAKER_02: do but never felt like they had the time.
43:54
SPEAKER_02: So debt reduction,
43:56
SPEAKER_02: refactoring,
43:58
SPEAKER_02: making things easier to read,
44:00
those are the things I'm seeing the best engineers at Shopify
44:02
use AI tooling to
44:04
SPEAKER_02: actually do,
44:05
SPEAKER_02: and they just never felt like they had the time or the resources.
44:08
The engineer who always felt like,
44:09
I wish I had six months to do a Code Red,
44:11
now can use an AI refactoring tool like a Cloud Code or OpenAI
44:15
Codex to unleash
44:16
SPEAKER_02: these agents in parallel and then review all the PRs,
44:20
SPEAKER_02: but then unleash it and feel like they have a team themselves.
44:22
SPEAKER_02: So that's what I'm seeing.
44:24
SPEAKER_02: And they're not afraid of trying things that may or may not work.
44:28
They might unleash something for 24 hours and like a Devon-like
44:31
tool and it comes
44:31
SPEAKER_02: back and utter garbage.
44:33
SPEAKER_02: They delete it all, but they said it was worth trying.
44:34
SPEAKER_00: Yeah.
44:35
And as a closing question,
44:38
a lot of people are thinking,
44:39
we'd love to be like Shopify.
44:41
SPEAKER_00: I'm either a staff engineer, director of engineering, et cetera.
44:46
SPEAKER_00: I'd love to transform my culture to be a bit more AI first.
44:49
SPEAKER_00: What would your advice be?
44:50
SPEAKER_00: How can people start?
44:52
Not everyone has the luxury of having the CEO saying,
44:54
OK,
44:54
I love these things.
44:56
SPEAKER_00: Do you have tips that you give to your peers?
44:58
SPEAKER_00: I know you're in CTO groups, for example.
44:59
SPEAKER_02: Yeah.
45:00
So I think that the number one thing is role modeling is the
45:03
best example.
45:04
SPEAKER_02: I haven't seen anything work better than role modeling.
45:06
SPEAKER_02: And the way that happens is you have to do it.
45:09
So if you are coding and you are showing people your workflow
45:12
and you are showing,
45:13
SPEAKER_02: you are in those same channels asking like,
45:15
SPEAKER_02: hey,
45:15
SPEAKER_02: I was trying to get this working and it wasn't working.
45:17
SPEAKER_02: Or I was able to get this workflow going via this prompt.
45:20
We have a prompt library internally where you can literally
45:23
grab prompt and say
45:24
SPEAKER_02: like,
45:26
SPEAKER_02: hey,
45:26
SPEAKER_02: this prompt worked for somebody.
45:27
SPEAKER_02: Let me try it and modify it for my use case.
45:28
That's the number one way in which I see people trying to adopt
45:32
AI is because
45:33
SPEAKER_02: they've seen other people do it.
45:34
SPEAKER_02: And so we try to share AI use cases.
45:37
SPEAKER_02: We have, again, prompt libraries.
45:38
We had a hackathon as a part of Shopify Summit a few weeks ago,
45:41
and we leaned heavily into AI.
45:43
SPEAKER_02: And it wasn't just junior people learning AI.
45:45
SPEAKER_02: It was senior people trying to embrace cloud code.
45:47
SPEAKER_02: And one guy said, I haven't coded in six weeks.
45:50
SPEAKER_02: I've only used cloud code.
45:51
Then I pushed him,
45:52
and he said,
45:52
OK,
45:52
I have to make changes here and there.
45:54
SPEAKER_02: I said, cool, you're 95% AI, but that's the point.
45:57
SPEAKER_02: You're not supposed to be 100.
45:57
SPEAKER_02: And role modeling is the number one thing I've seen.
46:00
SPEAKER_02: That and sharing examples.
46:02
SPEAKER_02: So role modeling and then sharing.
46:03
SPEAKER_00: Especially because no one has it figured out.
46:05
SPEAKER_02: No one has it, exactly.
46:06
SPEAKER_02: No one's there yet.
46:08
SPEAKER_00: Well, this was wonderful.
46:09
SPEAKER_00: I love how much you're experimenting.
46:11
SPEAKER_00: I love how you are not afraid to take some turns to learn more.
46:16
SPEAKER_00: And I love how much you're sharing internally.
46:18
SPEAKER_00: I think we got a little window of this,
46:20
SPEAKER_00: but I think it's just something I hope more people,
46:23
SPEAKER_00: more companies will do.
46:24
SPEAKER_00: So this was wonderful.
46:25
SPEAKER_00: Thank you so much.
46:26
SPEAKER_00: Thanks for having me.
46:28
SPEAKER_00: I found it refreshing to see just how practical Shopify is about LLMs.
46:31
It feels to me that they understand that you need to invest and
46:34
experiment to get results.
46:36
One surprising thing for me was understanding how Shopify
46:38
expects to become more
46:39
effective using AI tools by hiring interns who come in with an
46:43
open mind towards AI
46:44
SPEAKER_00: tools and can come up with clever use cases for it.
46:47
SPEAKER_00: For more details on how Shopify operates,
46:49
check out Deep Dice and the Pragmatic Engineer linked in the
46:51
show notes below.
46:52
SPEAKER_00: If you've enjoyed this podcast,
46:53
SPEAKER_00: please do subscribe on your favorite podcast platform and on YouTube.
46:56
This helps more people discover the podcast and a special thank
46:59
you if you leave a rating.
47:00
SPEAKER_00: Thanks and see you in the next one.