Latest / Inside Commerce: Ecommerce Strategy, CX and Technology Podcast / EP249: How To Build An Experimentation Culture In Ecommerce, With Trainline's Experimentation Lead Hannah Tickle
Transcript
- 0:02You're listening to the Insidecommerce podcast, hosted
- 0:05by experienced consultants James Gerd and Paul Rogers.
- 0:08Insidecommerce is your fast track to digital knowledge,
- 0:10featuring interviews with leading e-commerce retailers,
- 0:13industry thoughtleys and the C-Suite key tech companies to
- 0:16explore the latest in digital strategy, customer experience
- 0:18and tech. Discover what the latest trends
- 0:21really mean and get practical advice based on experience to
- 0:24help fine tune your ECOM strategy and tech stack.
- 0:27Follow us on LinkedIn and reach out to James of Paul if you'd
- 0:30like to discuss an e-commerce project.
- 0:32We hope you enjoy the latest episode.
- 0:35Today's episode is sponsored by Scale Commerce Engine.
- 0:38I'll swap more about them later. So let me set the scene for our
- 0:40conversation. We're talking about
- 0:41experimentation in e-commerce, which is rather large but also
- 0:45incredibly important topic. And there's a massive difference
- 0:47between just doing like AB testing and true
- 0:49experimentation. Few E Comm businesses fully
- 0:52embrace experimentation, and often that's due to skills gaps
- 0:54or resources to do it properly, like time pressures because of
- 0:58daily trading. Every business can learn from
- 1:00those who are more mature in their experimentation journey.
- 1:03Today, you'll get insights from a leading practitioner who's
- 1:05been working at Train Line as team lead for more than three
- 1:09years now. So warm welcome to Hannah.
- 1:11How are you, Hannah? I'm all right.
- 1:13Thank you. Thanks for having me.
- 1:15I'm really looking forward to this because it's an expertise
- 1:17that I don't have. I I understand experimentation,
- 1:20but I've never been responsible for it and nor would you want me
- 1:23to be. So before we start asking lots
- 1:25of questions and sharing lots of advice, can you just think it'd
- 1:28be really good to talk our listeners through?
- 1:30You know who you are, what your background is and what your
- 1:33current remit is. Yeah, cool.
- 1:35I will do so. My background is it has been
- 1:39experimentation through and through for probably 13 years
- 1:44now. I started with my degree in
- 1:46experimental psychology. I did a PhD which was research
- 1:49based, so ran a load of experiments that was in
- 1:52cognitive neuroscience. It was lots of fun.
- 1:55And then after I'd understood how brains work, I moved into
- 2:02how trains work. I did go via another company but
- 2:06basically switched from academia into industry in a couple of
- 2:10places and I currently work at Train Line.
- 2:15I've been there since January 2020 and I have basically been
- 2:21developing the AB testing process, platforms and and
- 2:25products for almost all of that time.
- 2:30Just a small amount of things to do and because train liners are
- 2:33obviously has insane amounts of data behind it and lots of users
- 2:38as as we told you last before, I've been a train line customer
- 2:40for many years and I've always been impressed with the
- 2:42iterations in the app and, and the improvement.
- 2:44So yeah, I'm I on a personal level, I'm going to really enjoy
- 2:47this selflessly today as well. So we're going to start with the
- 2:50kind of like the strategic view. Then we're going to get down to
- 2:51some of the practicalities about how you you run things.
- 2:54And let's start the the strategy and skills questions like what
- 2:57is the role of experimentation in e-commerce?
- 3:00And how does experimentation help improve things like
- 3:02strategic planning or prioritisation?
- 3:06So I think my answer to that one is that experimentation like it
- 3:12is strategic planning and prioritisation and
- 3:14experimentation is product development.
- 3:18And I think I'm going to give a very brief analogy about the
- 3:22brain if, if you don't mind, just from my research previously
- 3:26as a, as a PhD student, a lot of what I did was looking at what
- 3:30went on in the brain when certain decisions were being
- 3:32made. But actually there's something
- 3:35interesting that in certain regions of the brain, they show
- 3:39like really strong activity always during certain tasks.
- 3:43And you might conclude from that that like, fine, that brain
- 3:45region must be doing that task because it's like really active
- 3:48whilst it's happening. But actually what what you find
- 3:53is that with people who have damage to that region, so in
- 3:56other words, they don't have that region intact in their
- 3:58brain and therefore it can't be supporting them.
- 4:00They are still able to do those tasks that you might expect not
- 4:04to be able to do. And the point is that even
- 4:09really, really scientific methods of analysing data just
- 4:13sort of as you go along, can be misleading, like brain scans.
- 4:17You think, yeah, this really tells us the answer to
- 4:19everything, that they're very scientific, but actually they
- 4:22don't necessarily because they're just a recording and
- 4:24they're not giving you any causal relevance of what's going
- 4:27on. And so in an analogous way, I, I
- 4:31hope that most organisations are using their data to understand
- 4:37their customers, understand their business, understand the
- 4:39decisions they should be making. And that's great.
- 4:41I'm not saying that it's something we shouldn't be doing
- 4:43and it's obviously something that we also do at train line,
- 4:46but to actually understand causal relevance of the things
- 4:49that you're doing, like how good is this path we're taking?
- 4:52Should we take this strategy or that strategy or that strategy?
- 4:55What are it's implications for product development?
- 4:57The only way you can have a causal understanding of that is
- 5:00through experimentation. Because you do, some people see
- 5:03it, some people don't see it. And you can literally isolate.
- 5:05Well, you can with enough, you know, volume and, and low enough
- 5:09noise, you can isolate that factor and say, OK, this matters
- 5:12and this is what we should be pursuing and this is how we're
- 5:14going to do it. Are you ready to beat the
- 5:17market? With Scales Commerce Platform,
- 5:19you can launch advanced promotions within seconds,
- 5:22generate reliable revenues with subscriptions, open shops, and
- 5:25new channels and countries with just a few clicks and much more.
- 5:29Go to the link in the description to visit Enterprise
- 5:32Evolutions from Scale and see all of our latest features.
- 5:37I really like that example. I love that all of the, the, the
- 5:42psychological and neuroscientific theory that sits
- 5:44behind this as well. Having done psychology at
- 5:47university, but not obviously at the advanced level.
- 5:49You have, it's just fascinating the, the, the data bits, one
- 5:52side of it, but the behavioural understanding is another part
- 5:56that influences this planet. And so it leaves me on to
- 5:59another question. And I've worked with a a lot of
- 6:01businesses in different types, B to BB to C and many who do
- 6:05testing. But what I find it's often
- 6:07fragmented. So there'll be some testing run
- 6:10in an E Comm team, but the learning from that isn't shared
- 6:12outside and there'll be other tests run elsewhere that that
- 6:16the E Comm team could benefit from.
- 6:18And you don't, you don't seem to have this overall ownership.
- 6:20So in your, in your opinion, who should own experimentation and
- 6:25why? Because it does impact other
- 6:26areas of business like it can inform product decisions.
- 6:30It it can and, and it should obviously.
- 6:33And so in terms of ownership, I I think experimentation is like
- 6:38this huge multifaceted like ball of stuff.
- 6:42And so my view is that the we would call the experimentation
- 6:47data products should be centralised.
- 6:50So by experimentation data products, I mean things like if
- 6:54you're using an assignment tool, an analysis tool, if you've got
- 6:58experimentation playbooks, you've got dashboards to analyse
- 7:02the results of the experiments, probably others as well that I
- 7:05can't think of at the top of my head.
- 7:07Those should be centralised. They should be accessible to
- 7:09everyone. Like everyone can see them,
- 7:11everyone can contribute to them, everyone can do them.
- 7:15And then the usage of experimentation, as you say, it
- 7:19needs to be company wide. But it, I, my view is it and,
- 7:22and I can have some empirical evidence for this, but it can't
- 7:26be company wide unless you've got that centralisation of the
- 7:29core tools that are going to be used.
- 7:32So no barriers, central ownership, but but everyone,
- 7:36everyone is able to access it and use it, I suppose.
- 7:40Sorry, just one thing to add to that is there's also I think a
- 7:43question even in like very mature experimentation, like
- 7:48businesses that are very mature in their experimentation
- 7:51program, there will always be some people who have more
- 7:53experience than others, new joiners, people who've done it
- 7:56differently elsewhere, etcetera, etcetera.
- 7:58And so I think it's important to make sure that there is always a
- 8:01core group of people. And you might define that
- 8:04differently depending on which organization you're in, but a
- 8:07core group of Trail Blazers or guides or whenever you want to
- 8:13call them who are going to support people just to make sure
- 8:17they're on the right track. So everyone's there and using
- 8:19these tools, but there is support out there if if they
- 8:22feel like they need it. I think that I really like the
- 8:26fact that you, you focus the point on the core tooling and
- 8:30the documentation and other that intelligence database.
- 8:33Because I remember years ago I did some work with, with Sage
- 8:35and they're obviously a massive organization with and that's the
- 8:39thing. They struggled because there was
- 8:40no central repository. So a market like France couldn't
- 8:44see what the UK had done with testing on e-commerce for like
- 8:48pop ups on the site. So what work, what dim work to
- 8:51not that they should copy, but to then help them inform their
- 8:54own testing strategies. And it's amazing actually how
- 8:57often you that's the stumbling block.
- 8:59There's no centralised database. So so information's fragmented.
- 9:04Yeah, and to be honest, even if the even if the database or
- 9:09whatever is centralised, there is often still work to be done
- 9:14to make it accessible to everyone.
- 9:16And I think it's fair to say that's a journey we've been on
- 9:19in the last few years with experimentation at Train 9.
- 9:23So what my aim has always been is that someone with no
- 9:29statistical knowledge, no knowledge of the product area in
- 9:32question, possibly limited knowledge of trains, but I think
- 9:35most people at Train 9 know how about trains.
- 9:37It just happens when you work here for long enough, but but
- 9:40you know, the premises shouldn't need to know anything and can
- 9:44come to a dashboard, for example, and get that piece of
- 9:47information. Like I saw that this test was
- 9:50run. I can see in plain lot of
- 9:53phrasing what the hypothesis was.
- 9:56I can see at a glance what happens and I can infer from
- 9:59that whether or not it was released or or what happened
- 10:02next. It's it's not necessarily easy
- 10:05to do, even when you've got that central tool.
- 10:08Swap is the ecommerce operating system, powering brands looking
- 10:11to scale from shipping tracking and package protection to
- 10:13returns and cross-border. Swap improves every pain point
- 10:16in your operations journey, trading your tech stack for the
- 10:19one system that can manage it all.
- 10:20This is ecommerce evolved. Yeah, a very good point, which
- 10:25you mentioned hypothesis that kind of segues neatly onto you.
- 10:30I know that you you talk about sit impactful factors for
- 10:34growing experimentation culture and I wanted to drill into some
- 10:37of those and really get get your insights about what good looks
- 10:41like. So how you go about it.
- 10:41So let's start with hypotheses because not all all equal and
- 10:46there is a difference between AI guess a lazy and and a a a good
- 10:51or well polished hypothesis. So in your mind, what what makes
- 10:54a hypothesis strong and useful, and how do you weed out
- 10:58something that that's a bit weak and hasn't really been thought
- 11:00through? So in my view, I think the the
- 11:04key definition of a strong hypothesis is that it's
- 11:08something that is backed by data to start with.
- 11:12And again, something that I have talked to many people at Train
- 11:16Line and Beyond about is we use Bayesian stats.
- 11:19That only matters because Bayesian stats gives you
- 11:21literally the probability of your hypothesis being true or
- 11:24not. So if you have an idea, it may
- 11:28well be a really sensible idea, but if you've not got any data,
- 11:32nothing to ground it in, then sort of a priori, you've got a
- 11:365050 chance of that hypothesis being supported.
- 11:38Like it's equally likely to be true as not true.
- 11:42And as soon as you start to ground your hypotheses in
- 11:46previous data, either from like your organization's data or from
- 11:51qualitative research or from previous experiments that have
- 11:54been run, you're already increasing the chances of your
- 11:57hypothesis being supported. And that's literally what you're
- 12:00testing in basic sense. Like I say, it's like you are
- 12:02literally looking at the probability of it being
- 12:04supported. And I think there's often a
- 12:06tendency for people who are newer to experimentation to hope
- 12:10that there's some sort of statistical magic that will
- 12:12happen that will take their well-intentioned but potentially
- 12:17mediocre idea and do some wizardry to it.
- 12:20And you get like, oh, hypothesis supported.
- 12:21And that's just like, the only thing that's going to influence
- 12:24it is the hypothesis itself. And the other, another key thing
- 12:30that I've talked, I've talked about a lot is the importance of
- 12:33testing concepts rather than testing things.
- 12:37So it should never be the case that your hypothesis is, we
- 12:41think this design is better than that design because that's a
- 12:44thing. There's no reason like.
- 12:45And even if you were right, what have you learned?
- 12:48You've learned a really isolated fact like, well, yeah, they like
- 12:51that better than that. But why?
- 12:52How can we build on this? What can we do next?
- 12:54What's the natural result of this information?
- 12:56You just don't, no. Whereas if you're testing a
- 12:59concept, I think the key way of knowing that you're testing a
- 13:04concept is there's many different ways that that concept
- 13:06could be manifested, many different designs, possibly even
- 13:09many different places within your product that you could
- 13:11test, test this hypothesis. So that's another, it's another
- 13:16key one that we go for. And I just just filling
- 13:20finishing that ramble. It's usually quite easy to see,
- 13:24to be honest, it like it's quite easy to see a sort of
- 13:27retrofitted hypothesis where someone has an idea that they
- 13:31want to test and they've kind of just written a sentence about
- 13:36it. Because actually, as soon as you
- 13:37start to pull apart, what can you learn from this?
- 13:41It quickly becomes apparent whether you can or not.
- 13:44Yeah, The thing is so important about the day thing to try and
- 13:47give your yourself a chance of having a strong hypothesis.
- 13:49Because I've seen it before where where people are over
- 13:52eager to get into testing. So they come up with all sorts
- 13:54of hypothesis that aren't really strong enough.
- 13:57And most of them get disproved and the the tests fail that
- 14:00everyone gets disheartened. And then someone at the higher
- 14:02level goes right, cut, cut the CRO because it's not working.
- 14:06And you, you get all these like false positives in the, in the
- 14:08decision making. So I think that's really good
- 14:10advice. And that leads me to the next
- 14:11one, which is next. One of your impactors is knowing
- 14:15why you're tested. So can you expand that?
- 14:17Like what level of detail do you need to get in to to know why to
- 14:20define goals, object to success criteria, etcetera.
- 14:24So I have to a shout out to a tool I found really useful
- 14:27actually for this is it was developed by Spotify.
- 14:30It's the it's the thoughtful execution framework.
- 14:33So it's, it's still a hypothesis framework, but I think it helps
- 14:36you really to ground your idea and understand exactly what
- 14:39you're testing. The actual level of detail, I'd
- 14:42argue doesn't need to be particularly high.
- 14:46But you know, I love data. That's why I do my job, really,
- 14:50really like it. And so, and I'm sort of in in
- 14:52the happy position now where during my PHDI would spend
- 14:56absolutely ages collecting the data and making it into a clean
- 14:59format, whatever. Now, just like because of the
- 15:02job I have, I get to get involved with it right at the
- 15:04end when all the lovely data are already there.
- 15:07And you just get to have a look at them and, and see what
- 15:09they're telling you. And so I know as well as anyone
- 15:12else, it's so tempting when something doesn't jump out from
- 15:17your data to start clicking around and saying, what about
- 15:21have we excluded that one or what have we only looked at, I
- 15:25don't know, peak time travel, let's say, or what have we
- 15:27focused on a certain region of the country, etcetera, etcetera.
- 15:32And so that's kind of what I'm getting at.
- 15:34Like you should know why you're testing.
- 15:36And really crucially, your contingencies should be
- 15:39specified upfront. You should have your analysis
- 15:42and then that analysis tells you what you do next.
- 15:45And there should be no room for ambiguity or questioning by the
- 15:49time you've got those results. Yeah, that makes a lot of sense.
- 15:53And I've had a lot of conversations with people at the
- 15:56time in terms of making sure that they provide back their
- 15:59their testing privatisation to their overall strategic
- 16:01objectives. But that's e-commerce from other
- 16:03business because you often find somebody gets excited about
- 16:06running a test that might be called test, but it's not
- 16:08aligned with the strategic focus.
- 16:10So it might achieve something, but it's not achieving what the
- 16:13business wants that team to achieve, basically.
- 16:16Yeah. And that's actually linked to
- 16:17the thing you said a couple of minutes ago about people that
- 16:20itching to get starting, get started and having lots of
- 16:23ideas. Like I think the other things,
- 16:26nice data are empirical. So sure, like sometimes throw in
- 16:30a curveball, why not, you know, one in 10, maybe just throw in a
- 16:34random hypothesis just to see what happens.
- 16:35It's like the exploration, exploitation trade off, although
- 16:38obviously it should still be grounded in some data.
- 16:40But yeah, most of the time you just need it to be in line with
- 16:44strategy data backed and not something that you just feel
- 16:47really strongly about. So on to the next one of these,
- 16:52these six key points. So test fast test like test big.
- 16:55I like this one, but let's let's break that down because I guess
- 17:00I, I often talk to people about tactical versus strategics like
- 17:04testing, because I'm at a much more Noddy level.
- 17:07So I'd do things in a very simplistic language.
- 17:09But you could do a tactical testing on a page in, in your
- 17:12website where you might have a KPI that's not performing well
- 17:16and you want to prove it versus strategic, which might be around
- 17:19proposition pricing something something bigger.
- 17:22When you say test fast, light and big, what do you mean?
- 17:25And what factors impact that ability to execute at speed?
- 17:31OK. So taking your tactical versus
- 17:33strategy point just then, I think one thing that I say
- 17:36reasonably often is testing one thing of size 10 is better than
- 17:44testing 10 things of size 1. Because if you've got that one
- 17:48really impactful thing, the likelihood is with your tents
- 17:53significant in whatever statistical methodology you're
- 17:56using. But smaller things, you're
- 17:58probably just seeing a sort of reactionary peak and it's going
- 18:00to die down. Again, not always, but that that
- 18:02seems to be what happens. And it kind of goes back to the
- 18:04whole idea of concepts like making tinkering around on a
- 18:09page to try and you know, get the numbers up in your quarter
- 18:14goal for, for whatever metric it might.
- 18:18You might get those numbers up, but the likelihood is you've not
- 18:21done anything substantive and long term.
- 18:24But then I'm also saying test fast and test light.
- 18:27And what I mean by that is actually again, in the example
- 18:30you just gave, you know, pricing is a bigger factor, but the real
- 18:34bottleneck from my experience is not the development of the
- 18:38hypotheses. So if you're coming up with a
- 18:40whole new price structure for your product, then that would
- 18:42obviously be difficult and it would take a bit longer.
- 18:45But nonetheless, you would, if you're going, especially if
- 18:48you're going to experiment, you would hope you would nail
- 18:50something down conceptually, like let's charge people who are
- 18:54already paying at the top end slightly more because we think
- 18:56they're committed. And, but I don't know.
- 18:57I'm, I'm making that up on the fly because obviously we don't
- 19:02control the prices of train tickets or train line, but
- 19:04something along those lines. You could have, and then you
- 19:06could test it. But the, the, the thing that
- 19:10takes a long time often is actually just the development
- 19:12work. So I know some companies will
- 19:15still do like allow people just to like edit it live without
- 19:19going through a code release process.
- 19:21But I think more and more people, you know, centralise
- 19:24their testing releases in in line with all of their other
- 19:27code releases. And so actually development work
- 19:30adds up. And if you're adding development
- 19:32work and design work and making a really substantive change to
- 19:36the mix, that's going to take you for an idea that you might
- 19:40have come up with in a few days by exploring the data.
- 19:43It's going to take you a few weeks just to implement it.
- 19:46And again, when you get to that point, people are itching to
- 19:49know the answers. Like it's that you almost build
- 19:51that, that lag builds momentum and it makes then a result with
- 19:54the hypothesis not being supported all the more, you
- 19:58know, crushing. So long winded.
- 20:02But if you are testing a concept, there is merely always
- 20:05a way to manifest that product. Sorry, manifest that concept in
- 20:10your product without it being the fully fledged thing that you
- 20:15would release if your test goes the way that you want it to go.
- 20:18And in doing that, you can save yourselves.
- 20:20And again, it might be another day of sitting down and thinking
- 20:24about it. I would argue it's one of the
- 20:26most challenging things to do when you're coming up with a
- 20:29test is actually working out what's the quickest way you
- 20:32could test it without sacrificing, you know, without
- 20:37losing sight of the actual thing that you're testing.
- 20:40But yeah, that extra day reaps the rewards longer term because
- 20:45you've you've massively reduce the development time in
- 20:48releasing it and then sorry, no. I think that's a really good way
- 20:52of looking at it is because there's because speed to market,
- 20:55it comprises different stages and if you rush the planning bit
- 21:01and end up spending longer in the execution, you could have a
- 21:04slower speed to market than taking that extra time.
- 21:06But often I think the challenge is it's people in the testing,
- 21:09whoever's responsible, being able to hold off the
- 21:12overeagerness from maybe a higher level of we won this test
- 21:15live by Friday. Yeah.
- 21:18And actually going back to an earlier point, I think that's
- 21:20one of the nice things about having certain things
- 21:24centralised in a company because for a long time, you know,
- 21:29that's always the case. It's going to be the case in any
- 21:31company. But when the people who are sort
- 21:35of setting the advice and writing the experimentation
- 21:39playbook and so forth are divorced from that pressure,
- 21:42which is always going to exist, it makes it much easier to to
- 21:47provide that push back and, and to to provide that reasoning.
- 21:51Because it's not everyone on the same team receiving the same
- 21:53pressure. It's some people on one team
- 21:55receiving pressure and someone without that pressure hopefully
- 21:59giving like a cruel and logical response to actually be slowed
- 22:01down by three days, your speed up by three weeks.
- 22:04Yeah. And I'm going to come back to a
- 22:06point you made earlier, you you talked about almost this seemed
- 22:09a democratisation, like building out centralised tooling
- 22:13documentation to help people. But beyond that, culture and
- 22:18behaviour play an important role in this as well.
- 22:20And people buying into this process and experimentation,
- 22:23supporting the open strategy. How what are your, how do you go
- 22:26about that? Like how do organisations get
- 22:28people to buy and break down the silos that often exist?
- 22:31And I know it's not a one-size-fits-all solution in
- 22:34businesses. It's not.
- 22:40So I think the only thing I could say is you have to play
- 22:46the long game because if you could, you know, if you, you
- 22:51could like put culture on your road map and solve it in 1/4,
- 22:56that would be very easy. Everyone would like that.
- 22:58But you can't. Culture is just something that
- 23:00needs to be driven by consistency, by responsiveness,
- 23:05and by, I guess, reliability that I think that sounds a bit
- 23:07boring, which, which yet again, goes back to the dedicated
- 23:12central resource for experimentation.
- 23:15I there has never been. Yeah, I I never want to paint a
- 23:20picture of there being a bad culture versus a good culture at
- 23:24Train Line. But what I will definitely say
- 23:27is I have seen that culture grow and grow and grow over the time
- 23:31that I was at Train Line. And the one exception to that
- 23:36was actually, and this sounds quite arrogant, but it's when I
- 23:39went off on maternity leave, when I came back, there was just
- 23:44a noticeable difference in the way that people were experiment
- 23:47doing experimentation, which we built back up again.
- 23:50And I don't attribute that to me.
- 23:51I attribute that to the consistency and the support and
- 23:55the resource availability. Yeah, it's it's hard because
- 24:00culture changes over time as well with when people leave and
- 24:03join and when the change is at higher level as well at people's
- 24:06home. Yeah, you obviously have
- 24:08politics to deal with, but also knowledge bases to build up and
- 24:11people aren't haven't come from an experimentation background
- 24:14and come into an organization where that's strong.
- 24:18Yes, although again, if you are able to democratise, there
- 24:21should be enough key players who have a level of knowledge across
- 24:26the organisations like distributing it across the
- 24:28network. But I also agree that the, there
- 24:31are, there are crucial factors are, you know, you're the people
- 24:37who are most senior in the organization.
- 24:40Their attitude towards experimentation really matters.
- 24:43And again, I, I've, I've seen a really like that that's
- 24:47positive, I think a train line and that that really matters.
- 24:50Yeah, it does in the same with any other element of culture
- 24:53feeding down in terms of behaviour, how people work
- 24:56together 100%. The the final one of the six
- 24:59impactful factors I really want to drill into is know your data.
- 25:03Like, I'd love to hear processes and tools that for you are
- 25:07critical to gaining that right understanding because we all
- 25:09know there is so much data out there and there's the ability to
- 25:12be massively distracted by vanity data where it goes, oh,
- 25:15that looks good. It's shiny and you can do
- 25:17nothing with it. So yeah, what what is your
- 25:19advice? How do you get to the the heart
- 25:21of what's important? So I'm going to tell you an
- 25:26anecdote that might seem certainly random, but I promise
- 25:28it's relevant though. I, I joined Train Line in
- 25:31January 2020. And you might recall that in
- 25:35March 2020, there was a lockdown that lasted for quite some time.
- 25:40And I think it was like genuinely illegal to get a train
- 25:44at one point if you weren't a key worker going to work, for
- 25:47example. So that obviously meant that
- 25:51given that no one was on the trains, our focus for
- 25:53experimentation at train 9 and this was three months into my
- 25:56new job was a bit different. And the like, the advantage that
- 26:01has never stopped paying off that it gave me was that I was
- 26:06able to sit and explore our data and work out, you know, we had
- 26:11ways of calling tests, but they were, they were sort of generic.
- 26:17So doesn't mean that they are bad, but it does mean that
- 26:21they're generic. For example, calling a test
- 26:23because your P value is below nought point, nought 5 is
- 26:25generic, right? It's just a, it's just a
- 26:27criterion that people use. So what I did and all I needed
- 26:32was a Jupiter notebook to do this is explored our data
- 26:36analysed. I did a load of, you know,
- 26:40synthetic a a tests. I analysed them.
- 26:43I looked at previous AB tests and I built up a really good
- 26:47understanding of just what happens by chance alone using
- 26:49the statistical methods that we were using.
- 26:53And then we used that information to go on and
- 26:58influence the way that we call our tests so that how we
- 27:02actually decide whether something is better than another
- 27:05thing. And the reason it mattered so
- 27:09much is because it allowed me to see things that surprised me how
- 27:14often you would get a really big uplift in a certain, well say in
- 27:19conversion rates through chance alone and how much that varies
- 27:23in key areas of the site or the product.
- 27:26So we can analyse things differently and have different
- 27:29criteria according to like where an experiment is taking place
- 27:34and everything is now done in a bespoke way, ultimately based on
- 27:39that exploration. I think that's such an
- 27:43interesting area as well because I know you did a talk
- 27:46experimentation at Lee, didn't you?
- 27:48I think was it last year about avoiding being misled by noise
- 27:51and big data And it is a massive challenge.
- 27:56But what I guess how how easy is it to spot false flags and spot
- 28:02when the data's inaccurate? Because you can have all of the
- 28:04modelling set up and have really reliable modelling statistical,
- 28:07but then input data is inaccurate and leads to like
- 28:12distorted read. How do you approach that to make
- 28:15sure that the input data is reliable for you?
- 28:21So in terms of input data, I mean firstly, the tools that we
- 28:25use to assign, you know, we've obviously been through a bunch
- 28:29of tests to ensure they are actually random, that they are
- 28:34putting people in the same bucket when they return to the
- 28:36site, etcetera. So that size is not nothing that
- 28:38we worry about. I actually it's very easy to see
- 28:41when there's something wrong in the input data.
- 28:43I don't think there's ever been. Yeah, when when there's
- 28:48something wrong in the input data, it's, it's just usually
- 28:51you'll see something that looks either too good to be true or
- 28:53too bad to be true. And if it looks too good to be
- 28:56true, it always is. Like, that's just that's the way
- 28:58it is. And then you can go and explore
- 29:01it there. We try to also, we have a again,
- 29:06because we've, we've built all of our tools, our dashboards.
- 29:09So this is a common problem that people need to solve.
- 29:12And it's a very time consuming problem to work out whether your
- 29:15data look better because there's an effect in it or do they look
- 29:19better because something's gone wrong in the way they've
- 29:21implemented the test. So we have a number of factors
- 29:23and I can't remember them all off the top of my head, but it's
- 29:26it's like a check box, check box list.
- 29:28Like, are the sanity checks satisfied for this given test?
- 29:32And you get a red flag. You can go and look at that
- 29:34specific factor just to double check.
- 29:36And sometimes it will be fine. Anyway that that can be edge
- 29:38case tests that just look a bit squiffy, but they're fine.
- 29:42But yeah, saves time and gives us quite a clear and concrete
- 29:45answer. If I may sorry it I I think I
- 29:51just want to say one other thing about being fooled by data and
- 29:54I've already touched on it, I know a couple of times, but you
- 29:59are not. So for example, a probability
- 30:02value of nought point nought 5 is not that you're likely to get
- 30:08a false positive one in 20 times.
- 30:11But the problem with big data is even the smallest fluctuations
- 30:18in like, let's say, again, conversions, a number of
- 30:21conversions between the two groups with big data that will
- 30:25very often come through as significant.
- 30:27So it's not like it's wrong, it's just such a tiny effect
- 30:31size that it's not worth worrying about.
- 30:34And I know you know, many people already take this into account.
- 30:37They don't just look at AP value, they also look at an
- 30:39effect size. But then going back to the point
- 30:41I made about the exploration of our data, I really like about
- 30:44the way we conclude our experiments.
- 30:46It's expected loss. I know that other people use it,
- 30:48but it's literally the probability of your hypothesis
- 30:50being true and the magnitude like how, how true is it?
- 30:54How much better is your variable of interest, let's say
- 30:57conversion? And it brings them together.
- 31:00And so it's baked, baked into the quantity we use.
- 31:03And we just set the threshold accordingly based on what we'd
- 31:05expect to see in that area by chance alone.
- 31:09And that just helps us. It just means that, you know, if
- 31:11you get something like a nought point, nought to uplift of
- 31:14whatever, you're not concluding that that's significant because
- 31:18it just isn't. It's not going to make a
- 31:19difference to your business. Yeah, I've seen that before in
- 31:25in businesses where where they are doing, they're running a
- 31:28campaign and suddenly the the the revenue is five times what
- 31:34previous campaigns are done and they get overexcited.
- 31:36But it's just one single order for like 20 times the standard
- 31:39AOV. And that's why you're right,
- 31:41it's so important to have those checklists of OK, so yeah, it
- 31:45is. What is this really telling us?
- 31:48Have have we got anything that's that makes this less valid and
- 31:52that we need to discount it? I think that's important advice
- 31:55for anybody who's who's on this journey and not as advanced as
- 31:58you guys are to, to make sure they build checklists into their
- 32:00process to help them evaluate more effectively.
- 32:04The next question I've got was around the, the tech stacks
- 32:07because again, like every, pretty much every bit in my
- 32:09speech has got a slightly different tooling kit around
- 32:12experimentation and testing and you've built your own testing
- 32:15and stacked versus like taking off the shelf.
- 32:18So like what are the pros and the cons?
- 32:20Like what are the tools you're using?
- 32:22Like what? What made you go down this
- 32:23route? I'll be completely honest, the
- 32:27reason we went down this route was I already decided before I
- 32:31was at train line and I, I just did it.
- 32:33It was one of the first tasks that I did was to get us off a
- 32:36off the shelf tool for assignment on for assignment for
- 32:40web, I should say, and onto an in house tool the the like.
- 32:46The advantage of that is it's super cheap.
- 32:48We're in control of it. It does exactly what we need it
- 32:51to do and no more. So there is, I think, a strong
- 32:55argument for having something separate that does its jobs well
- 32:59for assignment versus analysis. And particularly we're injecting
- 33:02all of our own logic through our Bayesian library into our
- 33:06analysis versus assignment, which is already very
- 33:10challenging as as everyone will know, you know, we're trying to
- 33:13make sure that people are getting their train ticket
- 33:16information extremely quickly. People are often looking at
- 33:20their train tickets like whilst they're hurrying off to the
- 33:22station anyway. But I'm obviously talking in the
- 33:24in the region of milliseconds. We don't want anything that
- 33:26slows anything down. We don't want anything that if
- 33:29it, if it breaks, we need to make sure that our product is
- 33:33not going to break. And I know that many off the
- 33:36shelf tools offer this as well. It's just that we, we decided
- 33:40that this streamlined, inexpensive, rapid does one job
- 33:45well. Solution was the solution for
- 33:48us. I think the the downside is
- 33:51obviously it's you know, the maintenance is really
- 33:55challenging and there yeah it's it's it's like you don't you
- 34:04don't it's hard to allocate the right level of resource.
- 34:07I suppose it's the way to put it because usually it's fine, but
- 34:10then there's some sort of upgrade somewhere and it
- 34:14influences the code like our our tool and so we need to upgrade
- 34:17it. That's from the assignment side.
- 34:19In terms of the analysis side, I have to say I see no downsides
- 34:24whatsoever. I, I love having our dashboards
- 34:26in house. And I've already mentioned it
- 34:28that we have the Bayesian library which rewrote we
- 34:31developed. We are concluding our tests in
- 34:34the way that we know is right for us.
- 34:36And we are confident that this is giving us like the best
- 34:42signal that we can be getting based on our own data and based
- 34:45on our understanding of it. And also we're visualising it in
- 34:49the way that people need. Going back to an earlier comment
- 34:52that I made about really ensuring those dashboards work
- 34:55for everyone. We can, we can design it so that
- 35:00that that works in that way. And again, that's, that's in no
- 35:03way to say that off the shelf tools are not designing
- 35:07beautiful dashboards because they absolutely are that each
- 35:10business case has a separate need and we can just iterate,
- 35:12develop, change, change back and and do it exactly how we want
- 35:16to. And sorry, one final thing is it
- 35:19allows you, we talked again earlier about sharing knowledge
- 35:25across the business. And I think it's sharing tests
- 35:29is one thing, but sharing knowledge is another.
- 35:33And developing these tools in the way that you that that we
- 35:37have gives scope to sort of aggregating them into buckets of
- 35:43insights that are meaningful for us, but you probably wouldn't
- 35:47get from an off the shelf tool. Yeah, I'm not surprised.
- 35:52It's a common theme actually for the for when, when I speak to
- 35:55people in other areas who build out their own tackies.
- 35:58It's the flexibility. It's, it's the huge benefit, but
- 36:02there's always a pros and cons and it's always the maintenance
- 36:04bit because you end up with a smaller team rather than a, a
- 36:08third party who's got an entire development like team or
- 36:13multiple teams working on it. But yeah, I, I like the fact
- 36:17that you split out that, that assignment piece to the
- 36:19analysis. That makes a lot of sense as
- 36:20well. Let's come.
- 36:22Onto to actual tech because I you know, I'm I said to you, I'm
- 36:25a train line user. I have been for years.
- 36:27I love how the apps develop. There's I I know if I compare
- 36:29from what I started using to how much more useful data is in
- 36:33there. And I can only imagine how much
- 36:34fun it is behind the scenes to get the data from operators and
- 36:38various places to be able to pull it in at things.
- 36:41I even like shown the platform. And so I have this all the time.
- 36:43You say about millisecond. Yeah, I'm getting to water.
- 36:46They're like, shit, I'm late. I've got to get home to pick the
- 36:48kids up. What platform's it going
- 36:49changed? I love the fact you've got all
- 36:52this real times up, but I think what people would love to hear
- 36:55before we we end this episode is what are some of the most
- 36:58interesting, impactful kind of tests that you you've run.
- 37:02You talked about some of the you talked about split tickets in
- 37:05when we talk when we were planning this episode.
- 37:07I think people just love to hear the kind of tests that you run.
- 37:11Yeah. Well, split ticketing is
- 37:13obviously a feature that we have in the app, sorry, in the
- 37:17product. And I think that's that went
- 37:22into, I think that was launched around the end of 2019, maybe
- 37:26early 2020. So I kind of came for the tail
- 37:30end of that launch. But in terms of sort of, you
- 37:34know, big changes, it's not like it's not like a complete
- 37:41redesign or anything. In fact, the way that the way
- 37:43that the split ticket manifests itself in the app is usually, I
- 37:47think, just a little icon, but it was it's nonetheless.
- 37:51Obviously a lot of decisions need to be made about how
- 37:53quickly to move, how quickly you can get the test live, what you
- 37:57can learn from it, etcetera. Like I say, that was actually
- 37:59launched just as I was coming into train line, so I wasn't so
- 38:02involved in that initial launch. But there are other things that
- 38:06the there I can't, I don't think I can give specifics, but in
- 38:14general, there are tests that surprise me all the time because
- 38:17you think it's so such a small effort and it has an impact that
- 38:23you just really weren't expecting.
- 38:24So it's simple copy change, for example, when it's done for the
- 38:30right reason, the right concept, like it's that's the thing like
- 38:33you can manifest a concept through copy change and you can
- 38:37have a huge impact through doing it.
- 38:39And I see it happen, you know, reasonably frequently.
- 38:43And I suppose there's probably a flip side where there are some
- 38:47things that just it wouldn't, wouldn't probably be obvious how
- 38:52much effort it is on, on our side to manifest it.
- 38:56But again, that's where I enjoy being involved because you're
- 39:00trying to find the best way to test it and break it down and
- 39:02test it. Like like I said, apologies, I
- 39:04know it was quite a vague answer but I think I can talk about
- 39:08this, no? Tracy, actually do that.
- 39:10I've got a related question. How do you factor user feedback
- 39:13in? Because you obviously a lot of
- 39:16what you do is about honing the product to, to provide a better
- 39:19user experience as well to drive the conversions or whatever
- 39:22metrics you're gauging success on.
- 39:24So how do you get customer feedback to help validate you
- 39:29know and and define hypotheses? So this goes back to the point I
- 39:34made earlier about data-driven hypotheses that qualitative data
- 39:37should absolutely be factored into that and people should and
- 39:41do use it. I think one thing, one way I
- 39:45like to frame experiments, particularly I think at
- 39:48something, maybe it's, maybe it's in general, but
- 39:51particularly at train line where people choose to use train line
- 39:56over another mechanism for buying their tickets, but
- 39:59they're not necessarily choosing to go on the journey.
- 40:01You see what I mean? Like they have to get that train
- 40:04and their choice is just how they are buying it.
- 40:08And it means that you get, yeah, it's, it's not like someone is
- 40:13buying a new dress that they, they could choose not to buy.
- 40:15If they didn't like it, they didn't like the experience.
- 40:17So I there's a slightly slight nuance, but it's important to
- 40:21factor in and and it means in particular that a lot of the
- 40:24time you're asking. With an experiment, do more
- 40:28people like this than dislike this?
- 40:32Because designs are designs and the way of manifesting things
- 40:36matters. There will always be people who
- 40:38don't like it, but because they have to buy a ticket anyway,
- 40:42their behaviour and how they react to something they don't
- 40:44like may be slightly different than.
- 40:46Like I said, I'm using buying a dress as an example.
- 40:49And so in that sense, the reason I'm giving this anecdote is
- 40:53because it matters that you don't overly index on
- 40:59qualitative research because there will always be people in
- 41:02any. I'm not talking just train 9
- 41:04anywhere. There will always be people who
- 41:06don't like a certain feature and there'll be others who do.
- 41:09And actually, the crux of what we're doing is counting how many
- 41:12like it and and how many would prefer it was a different way,
- 41:15and trying to find the happiest solution for everyone.
- 41:20Yeah, that's interesting. And and obviously you you get
- 41:23you can get some real distortions in coal depending on
- 41:26your soul then as well. And all it takes is a few
- 41:28disgruntled customers to take the put it off kills.
- 41:31Yeah, I mean. Particularly if you're just
- 41:33looking at reviews. Like that's hugely biased in and
- 41:36of itself. You you tend not to leave a
- 41:38review unless you're pissed off about something.
- 41:42Yeah. Or or you're just one of those
- 41:44over excited, happy customers who leaves a positive review
- 41:47every single time everywhere. Yeah.
- 41:50It's nice that if people exist to balance out the others, but
- 41:53The thing is also people when I when I say I work at train
- 41:57lines, like why train tickets, the price they are or whatever,
- 42:01or why is my train on strike or positive things as well, like, I
- 42:06don't know, my train was on time.
- 42:08Yeah. I mean, you get both not
- 42:10responsible for either the good or the bad there.
- 42:12There's nothing we can do about it.
- 42:13We're just, we're just selling the tickets and I so another
- 42:16thing about feedback is being clear that the person who's
- 42:20leaving the feedback understands where your business has the
- 42:25influence somewhere it doesn't. Yeah.
- 42:27Yeah, nice examples, Hannah, that that brings a sense to the
- 42:31end of the. I mean, I could carry on asking
- 42:33questions forever, but it's Friday afternoon and you have a
- 42:35birthday party to get ready for. But I'd really appreciate you
- 42:38taking time. I've, I've loved it.
- 42:39I've found it really interesting.
- 42:41If anyone, if anyone listening in is is a fellow that
- 42:46experimenter or CRO enthusiasm wants to reach out and connect,
- 42:49what's the best way? To to connect with you brilliant
- 42:53LinkedIn, I'd say call on LinkedIn.
- 42:59Cool, I shall. Put a link in the landing page
- 43:01for the blog. Thanks everyone, as always for
- 43:03listening. I hope you've got as much out of
- 43:04it as I have. Don't forget to follow us on
- 43:06LinkedIn and subscribe to our LinkedIn newsletter that goes
- 43:09out every two weeks. Until next time everybody,
- 43:14you're listening to the Inside Commerce podcast.
- 43:15Hosted by experienced consultants James Gerd and Paul
- 43:18Rogers, Inside Commerce is your fast track to digital knowledge,
- 43:21featuring interviews with leading e-commerce retailers,
- 43:24industry thoughtleys and the C-Suite, key tech companies to
- 43:27explore the latest digital strategy, customer experience
- 43:30and tech. Discover what the latest trends
- 43:32really mean and get practical advice based on experience to
- 43:35help fine tune your e-commerce strategy and techs.
- 43:38Follow us on LinkedIn and reach out to James and Paul if you'd
- 43:41like to discuss an e-commerce project.
- 43:43We hope you enjoy the latest episode.