Latest / Inside Commerce: Ecommerce Strategy, CX and Technology Podcast / EP175: Building CRO Into Replatforming Projects And New Tech Releases To Improve Success, with Dave Mullen
Transcript
- 0:02Thanks for tuning in to re-platform podcast, sponsored
- 0:05by traction, hypersonics and hosted by myself change, scared,
- 0:07and Paul Rodgers. Warm welcome with.
- 0:09It's your first time. Thanks to our regular listeners.
- 0:12We've got a lot of time and effort into creating this
- 0:13content would love to keep this 3 and keep all these amazing
- 0:17guests come in so you can help us refers to other people in
- 0:20your network. Let them know we exist.
- 0:22We always want new listeners. Share our posts on LinkedIn
- 0:24Twitter wherever you see them. And also, one of the biggest
- 0:27thing you can do is write us on Apple Spotify.
- 0:30YouTube or wherever you listen really helps boost our
- 0:31visibility. So thank you very much and we
- 0:33hope you enjoy our latest episode.
- 0:35We are cracking, add topics a building, but a cro and tested
- 0:40into re platforms and new technology releases to improve
- 0:43success. So, there's a let me just
- 0:45quickly set this up. I don't you to do our guests
- 0:47say, Dave, but businesses, often underestimate how useful
- 0:51client-side testing to be in refining.
- 0:53The process releasing new features and new platforms for
- 0:57potentially go waste and huge round them up, time and money or
- 1:00And not feature prioritization wrong.
- 1:02And it's fascinating to see how consistently the desire to get
- 1:05products over the line, be proud of a new site and you launch can
- 1:11take the focus off like the customer experience peace which
- 1:13can then Compromise result. So our guest today knows how to
- 1:17help businesses fill the knowledge.
- 1:18The skills got to do this I guess cost-effectively and said
- 1:21instantly. So we're going to be covering
- 1:23integrates intestines, a full site re platforms, and new
- 1:26technology has sensible approach is to validate your new sides.
- 1:29They're all testing should play off the launch and how you can
- 1:32do things like brother proof of concept.
- 1:34So big, welcome to today's modern who's an independent CR.
- 1:37CR own consultant. Hey Dave, how you doing, sir?
- 1:39Like James, I'm very well, thanks.
- 1:41Thanks for having me on. Well, I really enjoyed it
- 1:44because we had a very brief LinkedIn discussion of a post.
- 1:46I made about this and they spun out on episode and I think this
- 1:49is really valuable to businesses of any size.
- 1:52I know you've got a lot of exposure to your background,
- 1:54working for a well-respected, analytics, and optimization
- 1:56agencies like, Po wdia w. A All.
- 2:00So, you want to just give people a bit of a flavor, but who you
- 2:02are and what you do? Yeah, so I think I just checked
- 2:08on LinkedIn for the last 12 years.
- 2:10Looks like I've been focused on optimization experimentation.
- 2:16I've worked with companies like Cannon interflora protein Works
- 2:21around some Warehouse, also look at involved with other things
- 2:25like institutions and Charities, like the University of London,
- 2:30currently doing some work with the teenage cancer.
- 2:32Trust, And yeah, the focus of my work is understanding users,
- 2:41building experiments based on that understanding and over.
- 2:45And over again, finding out what works and what doesn't work.
- 2:50And I think that's the core that process is about having the
- 2:54biggest impact in the most efficient way, which I think if
- 3:00you were to do some kind of character studies is very re
- 3:03much my my ballpark. So I'm yeah, I'm a big fan of
- 3:09the, the process of testing of finding out when I'm right, when
- 3:14I'm wrong and being able to actually demonstrate the impact
- 3:20and already, a lot of the things you said, James resume, people
- 3:24want to be proud of their sites and get them over the line.
- 3:27So, yeah, really looking forward to discussing today.
- 3:35Right analysis questions. So starting with the commercial
- 3:39imperative, why should businesses fact to testing into
- 3:44launching new functionality or technology projects be that, you
- 3:47know, a complete redesign or, you know, just launching like a
- 3:50new search solution, something like that.
- 3:56Yeah. Why should you use testing?
- 4:00Well, testing in itself is just a process.
- 4:03It's just measuring the impact of one thing against another so
- 4:09he can be really good. It can be a total waste of time.
- 4:14But yeah, done. Well I I would I'd almost
- 4:18reframe the question to start with and say why on Earth would
- 4:22business as usual be to make a A big investment and just went for
- 4:27the best but I guess taking a taking a step back as I say this
- 4:34is the process of testing and but around testing there's a I
- 4:40guess a discipline and an expertise that really goes hand
- 4:44in hand. The nature of testing is lead
- 4:49you to be about as user focused as you can possibly be.
- 4:53As you essentially, you have to get on with people but beyond
- 4:57that you're you results are judged by what users actually
- 5:02do. And so if you work in testing
- 5:06then you you have to hone in on what really, really matters to
- 5:13people. And you have to be able to
- 5:18develop that into either spotting or designing changes.
- 5:23that will not just Mass up to people, but Master enough to
- 5:27change their behavior in a measurable way to Change
- 5:31converging or progression or whatever your metric is by, you
- 5:34know, two three, four percent and you have to also have to be
- 5:40really quite ruthless and identifying what what's unlikely
- 5:45to move the needle? So, working in testing.
- 5:51There's it's a process that naturally forces you to be
- 5:55incredibly laser focused on the things that are going to really
- 5:59make a difference because Is over and over again.
- 6:02You've got to measure that and yeah if you just keep getting
- 6:07caught up in Buzz words or things, you've been reading
- 6:10about one LinkedIn or whatever, but they don't move the needle.
- 6:13Then useful is, is to test and learn and find out what doesn't
- 6:16work. You're not going to be working
- 6:19in that field for very long. So if I was going to be making a
- 6:26big investment in a technology project, then I, Would
- 6:31absolutely want people who are in that specialism of saying.
- 6:36Yeah. But what's going to affect my
- 6:38boss and wine and people who were in that feedback loop and
- 6:42it becomes refined over and over again, of does it affect.
- 6:46So, I was wrong. There was right there that moved
- 6:49it, more than I expected thought that would be positive, but
- 6:51actually, it was negative. So I would absolutely want to
- 6:55factor in Both the people who were shaped by the process but
- 7:00also the process itself it. Yeah.
- 7:05It's a very big big question. Why should we Factor it in?
- 7:08But fundamentally, I'd say, you want to factor in the expertise
- 7:12of what's going to affect your bottom line, and as much as
- 7:17possible, if you're making a big investment, that should be
- 7:20because you think is going to have a big impact.
- 7:23Hopefully, it's not just cuz it's shinier than the last
- 7:26thing. If it is going to have a big
- 7:28impact, I've done enough testing to to know that sometimes that's
- 7:32positive. Sometimes that's - especially
- 7:36when it's new. Because whenever your old
- 7:38project was, however, however dated or whenever it was however
- 7:43much it may be wasn't your cup of tea.
- 7:47It was at least out there in the wild and people were getting
- 7:50through it and it had its rough edges rounded off.
- 7:54So if you're going to make a big investment, Moment.
- 7:58Then to me, it'd been almost Madness to do, not find out what
- 8:03its impact is, and position yourself to either pull back and
- 8:08fix or double down and maximize the impact as much as possible
- 8:13and make sense. And then I guess the bit that I
- 8:16missed off the last question, whereas I think testing is often
- 8:19an easy cost line to cut and a lot of people have, you know,
- 8:24there are not counter-arguments around the cast the time.
- 8:27On the scale was needed. To do testing prop play, how do
- 8:32you go about kind of creating a business case and kind of
- 8:36countering those counter-arguments?
- 8:39Yeah, yeah. And I think that definitely
- 8:43comes up at the time of big projects.
- 8:45Especially I certainly hear a liner hate that idea.
- 8:49A lot is we're looking to get into testing but not not yet
- 8:54because we're developing a new sites where really focused on
- 8:58that. Now, we don't want to waste our
- 9:01energy and our efforts. So it sounds nice to say we
- 9:09should test everything, but unless you're an Amazon kind of
- 9:13scale, it's just not realistic. And it also can be a complete
- 9:19waste of time. But a lot of people have been
- 9:21burnt by testing projects that seemed exciting, but dragged, I
- 9:25didn't have that much impact. I'm one of the one of the
- 9:32challenges I guess is I would say either do it really well or
- 9:38don't bother at all. And you you have to be able to
- 9:42find things that make a big enough impact that you can
- 9:47measure. And the the time it might take
- 9:49to identify a 1% difference, you know, you might have to test for
- 9:53a year or two To get some statistical accuracy on that
- 9:57difference. Whereas, if you can hone in on
- 9:58something that's going to make a 3% difference, you might be able
- 10:02to identify that in within weeks.
- 10:05So, to the to the skeptic who isn't sure if they should waste
- 10:10their money on testing right now, I'd actually have a lot of
- 10:14a lot of sympathy in one sense in that, that there definitely
- 10:19are times where, if you're only going to kind of Half do it or
- 10:26make them put a tiny bit of effort into it then it hap
- 10:30solutely. Could be another cost and
- 10:32something that drugs things out the same type going back to my
- 10:41previous answer. If you if you are making an
- 10:44investment then your options really are to experiment to test
- 10:49and find out what works or just to hope for the best.
- 10:54And testing, I guess there's two way to angles, you can test.
- 11:01When the thing is done to find out its impacts and try and fix
- 11:05it then or make the most of it then.
- 11:07But there's also a huge amount of anything people underestimate
- 11:11how how lean and quick and focused testing can be at the
- 11:17beginning of a project. And during that time where you
- 11:21might have endless stakeholder meetings, And reviews and
- 11:25opinions going back and forth. You could have a fairly lean
- 11:30Focus, testing team actually out there sitting, not just what
- 11:35people think, or what people say they'll do, but finding out what
- 11:39they'll actually do, and I've seen lots of instances, where a
- 11:45kind of proof-of-concept test has been live.
- 11:48That's taken days, they less than a week to get live much
- 11:53smaller. Expense them that the full
- 11:57investment on a technology project where you just hoping it
- 12:00works. And so yes it's an investment at
- 12:05the star often but it can it might save you the effort of
- 12:10wasting the whole budget if the project is going to be a failure
- 12:15or it might hone in on the bits that matter.
- 12:17So that you do still have to have those stakeholder meetings.
- 12:20But you can cut half of them and say we know this is the bit we
- 12:25want to focus on or this is where we want to have our
- 12:28investment. And I guess I'd say to the
- 12:33person saying we're investing all this money right now, we
- 12:37can't be investing it in cro and you make my life, so easy.
- 12:43If you bring me in, for instance, after launching a new
- 12:47website, it's all out there. We have to go through piece by
- 12:52piece and we have to build the Case for each change, whereas
- 12:58involving testing and the expertise around it.
- 13:02In the, in the ideation stage, during the design stage, you,
- 13:08you get that expertise at the time.
- 13:11You can implement it. I guess, just as, if you were
- 13:15going to build a big shopping center, put the foundations in,
- 13:19put the windows in, put the displays in, and then build an
- 13:22expert in shopping center design, Inning at the end that
- 13:28will be absolute Madness. You don't want to be knocking
- 13:30down those walls and so on. So if you if you have a
- 13:33discipline and an expertise around that discipline, that's
- 13:37laser focused on what's going to make you Blossom line, then
- 13:41bring it in. While you can get the by far the
- 13:44biggest Roi, which is where you making those decisions and help
- 13:49them to. Yeah, let them help you to
- 13:51minimize your mistakes and maximize your benefits.
- 13:53That At that point, I tracked your Complete product Discovery
- 13:59growth engine create relevant shopping experiences that
- 14:02convert into sales and grow online Revenue with
- 14:05personalized, search merchandising, and
- 14:07recommendation Solutions powered by a I find out more at a
- 14:11tract.com. Now, let's get into some
- 14:15specifics. I think that always helps people
- 14:17and you've mentioned that running proof of concept and I'm
- 14:21a big fan of proof of concept stuff during like Feature
- 14:24requirements, planning's influence new platforms.
- 14:27When you have lots of stakeholders and we want this.
- 14:29This is a must and trying to see how you can do some of that
- 14:33validation prior to making expensive development decisions.
- 14:37So can you talk us through some examples of how he come teams
- 14:40could approach proof of concept and also them like they're swap
- 14:43tools and skill sets and needed a delivery.
- 14:44Like we're anything where you've done it and it's worked.
- 14:49Yeah sure. Another big question.
- 14:53Well yeah let me let's get into some actual examples I guess so
- 15:01suppose suppose you want to expand your delivery options as
- 15:06a feeling internally within the team that it would be great to
- 15:09give people a broader range of delivery options and the again
- 15:15there's a belief internally that that will increase Conversion
- 15:19because people feel it appeal to them.
- 15:23But in many cases is a complex thing to implement because
- 15:28you're not, just adding a few buttons, you've got to change
- 15:31your, your supplier relationships for your delivery
- 15:35relationships, your infrastructure, you Logistics
- 15:39and I, I've seen a client who'd, it would release done all of
- 15:46this without experimentation and it didn't work people.
- 15:53Don't want to think about delivery for that long.
- 15:56It's a pain or they want to go. Yeah, that'll do and move on.
- 15:59Whatever is giving them this big list of options with really
- 16:03focusing, the men and making them consider all the there
- 16:07complications and the expense and so on.
- 16:11So what we did was we, we pulled right back and found it to be
- 16:21far more effective to have just a few And now well I would
- 16:25advise someone to do in that process is rather than say oh we
- 16:30can't be doing testing right now.
- 16:32We've got to get this big projects out of the door.
- 16:34We can of the expensive testing. We've got to get this out.
- 16:37Is if you have two delivery options, your standard, you free
- 16:44delivery and you premium and you're single.
- 16:46But what if we gave people options in the middle, a kind of
- 16:50thing you can do is create Those buttons that say I want
- 16:56somewhere between premium-free, I want this but not this, let
- 17:00people select those and in the short term, what they don't
- 17:05realize is what they're actually getting these premium.
- 17:08So you've not changed any of your infrastructure, you've not
- 17:11had any discussion with your Logistics partners with your
- 17:15Warehouse team or any of that headache.
- 17:17You've not had to bring it all the stakeholders and debate
- 17:20because this is a permanent feature.
- 17:22And how do we feel about That and what wording should we use
- 17:25there and does that work and someone thinks you get a better
- 17:28deal there and all that you just say, right?
- 17:32We've given customers an extra three buttons they can click or
- 17:36take or whatever we have put some time working out how we'd
- 17:41want to make those peeling and what they're actually getting is
- 17:45the better option. So we've got a slight cost in
- 17:49the short term we're we're giving them all premium rather
- 17:52than a Kind of midpoint in between.
- 17:55So we might be adding extra 20 P or order.
- 17:58We don't want to run this for a long time, because there is that
- 18:02cost, but that's the kind of test that you could have live.
- 18:09No exaggeration. Potentially on the day that you
- 18:12have conceived. The idea certainly within a week
- 18:16of conceiving the idea. You could have those other
- 18:18buttons there and you can be finding out.
- 18:22Do people tell Clem. Do people actually want them?
- 18:25How does conversion compare and kiss?
- 18:28Your, a be testing? It, you're only showing half the
- 18:30people. Those, those buttons.
- 18:34You're not just launching it and then trying to weigh up with
- 18:37other Market, facts. As what the difference is, you
- 18:40saying, right? We offered people, the load mob
- 18:42lessons and conversion when down thank goodness, we didn't spend
- 18:48those months and all that debate and all those commercial
- 18:51relationships and all that. Efforts to find out that it
- 18:57doesn't work. So that's a, that's a negative
- 19:00example. A positive example I've been
- 19:07working with a client recently and then that add to basket
- 19:12process had become quite complex.
- 19:15And we're starting to hear from users, that all this feels like
- 19:20a lot of work comments like that.
- 19:23Now obviously they have to do the work, we have to find out
- 19:26where they live and when they want it delivered and so on but
- 19:29that the hypothesis was that there was they were seeing too
- 19:34much in one go. It was off-putting at that point
- 19:37and that maybe a lot of that should actually be moved into
- 19:41the checkouts. Now again that that's a big
- 19:44project that's not your dentist phone at the depth guys and say
- 19:50yeah would you mind just moving all this to the checkout and
- 19:52they say yeah. We'll have it done way, Tuesday.
- 19:55That's a proper. You know, you hear that build
- 19:57and always where they, they draw a breath and goals but and they
- 20:02really wanted you to know the pain that will be involved in
- 20:05that. So he said, well, how do we, how
- 20:10do we test this as a proof of concept?
- 20:12How do we find out? If people saw a simple looking
- 20:16at the basket process of people landed on the product page?
- 20:20And at that point felt this was going to be really easy.
- 20:23Stay here, would they progress would they click through?
- 20:28So, what we did was rather than go through that again that huge
- 20:32technology projects of moving in all that thinking, about
- 20:35scalability and one lip held at work in this situation, what
- 20:38about Christmas? And so on, we took all the
- 20:43subsequent steps little bit of CSS, hit them.
- 20:48So what you see is a really simple product page and then it
- 20:53It's a temple short term hack but I put all of that into a
- 20:57modal, so it was still on the product page.
- 21:00We hadn't had to move anything change any of the API calls.
- 21:05And if it was it definitely wasn't the, the optimal end
- 21:09solution. But again, this is something
- 21:11that can be built in in a couple of days and you can have Lolly
- 21:16and you can actually say right when we show people this vastly
- 21:20Stroup streamlined experience. Thing, they click those first
- 21:25two buttons and a higher rate than when they could see all the
- 21:28other stuff afterwards. And yeah, this was one that in
- 21:35terms of developing was, was months of work where us the
- 21:40proof of Concepts within a couple of days can have
- 21:44something live and can very quickly say, I am this area
- 21:47really does help. And in this area, actually, it
- 21:51doesn't as much so that this is where we want to.
- 21:53To want to focus our efforts. Yeah, I don't know.
- 21:58Jane do was a big question. I don't know if you want to
- 22:00guide me as to where you'd like me to focus in next.
- 22:05No, I think I think those are good tags result.
- 22:08I like the idea of the I guess I call it hacking it where you're
- 22:12not making fundamental operational changes because
- 22:14you're right the complexity of that will just really piss off
- 22:17he called has off. But yeah, I saw an example where
- 22:21somebody wants to do then notify me for our stocks and to do that
- 22:25properly. End to end with automation is
- 22:27not a simple thing, depending on what your back office systems
- 22:29are? Yeah, well, your email programs
- 22:32on what they can do and they just simply just Embedded.
- 22:35A simple form on the PDP, which yeah, I don't know, where there
- 22:40was a Google for whatever might be, just a cheap form Builder
- 22:42with notify me. Add your email, bang off you go
- 22:45and did it, will manually and did it on a few projects
- 22:48products to check what the appetite was 4 and then 4, if
- 22:51they then manually followed up, when stock came in, would that
- 22:54have an impact on sale? So, yeah, I like that idea of,
- 22:58you know what? Sometimes you can, you can do
- 23:00manual hack or nods ready to hide, but you do it cheap and
- 23:03you don't affect it. So now it does on Oscar and
- 23:06sonar scope, poor thing. You had a question instantly.
- 23:10Yeah, absolutely. Yeah, so Dave, I guess who
- 23:14should own testing internally if a bigger project like which into
- 23:19assuming there's no like cro manager or dedicated resource,
- 23:24who would you typically be working upon this stuff?
- 23:28and, Yes, a good question. I I can't pretend to be a an
- 23:34expert in e-commerce team design, but I think in terms of
- 23:40the principle, it's experimentation is about helping
- 23:47you to make better decisions. It's about finding our when you
- 23:51decisions are harmful or finding out when you decisions have been
- 23:56good and that there's something in there, you should make More
- 24:00love. So the, the benefit you get from
- 24:05experimentation base, probably directly proportional to how
- 24:11linked it is to the decision. Menu consists of the scope of
- 24:15decision but can be experimented on.
- 24:18So I think cro. It's unfortunate that it has a
- 24:23three letter acronym, but doesn't even completely makes
- 24:28sense, but that's a whole sideline.
- 24:31A phrase that of neural, an alarm Bell for me is yet.
- 24:34We just, we want to be able to pull another another lever.
- 24:38And I hear businesses think about SEO, PPC, cro, as these
- 24:44ways of just spending a bit more money to get a bit more traffic,
- 24:48or get a bit more conversion. And if, if its treated out in
- 24:54that way, if you have someone more Junior saying yeah we just
- 24:57thought we should get some experimentation.
- 25:00I didn't pinch conversion a bit, then your impact is likely to be
- 25:05limited to a bit of copy here or you know, style within a banner
- 25:12or something. Whereas I would argue if
- 25:18experimentation is about making better decisions, then you each
- 25:25should be owned by the people. Making the most important
- 25:28decision. Certainly the decisions that
- 25:31affect users. So yeah I don't think I could
- 25:36list a specific role or a title that your bone experimentation
- 25:41but it needs to be someone who is fully in the X in terms of
- 25:47how decisions are made and it needs to be someone.
- 25:51He gonna do these proof of Concepts there, they can be
- 25:57quite a lie to a typical team even if they think we're super
- 26:01agile. And so on even say the
- 26:05development process of cro of an experiment, he is very
- 26:11different. So if you're, if you're a
- 26:13traditional developer then you'll know all about scale.
- 26:16And robustness, and you might want to spend weeks and weeks
- 26:23developing something that for a proof of Concepts.
- 26:27You'd actually want to spend days on and have live and learn
- 26:33and be able to move on. So, again, if your, if your
- 26:39contact within the company or your decision Baker, whoever is
- 26:43owning cro and experimentation is a Able to make those quick
- 26:48decisions and say yeah, we should do this.
- 26:52Oh yeah. That's gather proof of concept
- 26:53live or yeah. I know this isn't how we
- 26:59normally develop, but it would be so valuable to find that out.
- 27:03And so, yeah, let's do it then. You're going to have an impact
- 27:10on many magnitudes more. So, yeah.
- 27:13So I guess it's someone bullying the mix.
- 27:16In terms of to say and someone with the yeah the clearance to
- 27:24say yea or I'll just check with so and so but yeah in principle
- 27:30we want to have that proof of concept live and let's get it
- 27:32like there's quick as possible, okay?
- 27:35And I guess a lot of people see their tests inside as part of
- 27:39like a BAU program or stream of work, can you talk us through
- 27:44how you would build test? And learn into a full
- 27:47re-platforming and redesign without blowing loads of budget
- 27:51and impacting timeline like what's your recommendation?
- 27:57I've been. Yeah, so testing is a BAU thing.
- 28:05I guess reflects the idea that it's a you know a lever to pull
- 28:08its kind of a thing to have but not one.
- 28:12We making important decisions. This is Big Boy time.
- 28:15Now let's get the three letter acronym stuff out of the way and
- 28:19do the serious things. So I would certainly argue very
- 28:26strongly against the BAU. Perspective.
- 28:33So if you if you're going to try and bake it into a new project
- 28:38then I think the first the first step I would recommend is
- 28:45breakdown. What do you actually intend to
- 28:47do and why to do use a phrase that makes some people feel like
- 28:53they're at school what is your hypothesis or what are your
- 28:56hypotheses and give? people that a simple framework
- 29:01that says, you know, because of this evidence, we think changing
- 29:04this will impact this and I would get people to actually go
- 29:11through the exercise of saying, Breaking it down at that level.
- 29:17What do we intend to do here? And using it a structure for?
- 29:21That's really good because it it means the structure points out
- 29:25the week ideas rather than you. And so while it can feel like a
- 29:31painful mating or a piece of paper work, it is really reading
- 29:34valuable to get people to lay out their reasons and their
- 29:37evidence, or their lack of evidence, and not everything has
- 29:42to have evidence. We, we don't have Be trapped in
- 29:46some kind of data-driven Terror knee, but at least if it is just
- 29:52a feeling that things should be modernized because the compass
- 29:55is look newer than us and we just sort of think we should
- 29:57have a new site. It's still really helpful to get
- 30:01that out there that we don't have the evidence.
- 30:04That is the case. And yeah, once you once you got
- 30:10that framework, you can take a bit more of an objective.
- 30:13Look, and you can separate out the The things that are just
- 30:17thrown in there because someone on quite a highway, just
- 30:20suggested them versus the things that there's actually a lot of
- 30:24evidence behind, but might not be as a shiny and his glamorous
- 30:28and but have the potential to have a bigger impact and then,
- 30:35for each of those, you need to ask the question.
- 30:39Well, do we already have such a strong case that we think we
- 30:44just be wasting. Our time trying to refine and
- 30:47build more evidence, maybe we've just seen it work in some ways,
- 30:53or is it quite weak? When we look at it, objectively?
- 30:58In which case, is there a proof-of-concept approach?
- 31:03Is the something we could do? It doesn't just have to be
- 31:06testing, but is that other ways? Is there something in our
- 31:09analytics data that might help us confirm or not?
- 31:16And in some cases you can test before as well but again is good
- 31:20to clarify that. So if you say well yeah we
- 31:24really believe we want to implement this this new feature
- 31:28but the functionality is so complex that we really would
- 31:32have to build it in order to be able to then experiments on it
- 31:36and once we built it well we've already got it and we would we
- 31:40like it we want it to be live. But in that instance, Just as
- 31:47with my kind of big shopping center example, it's so helpful
- 31:51at the star to say to the people building it.
- 31:54By the way we've got these areas of uncertainty and we'd like to
- 31:59be able to potentially test having it.
- 32:02Here, I having it here and so that when they're, when they're
- 32:06thinking about those foundations and how they create any sit,
- 32:09there are aware that it might not just be set in stone and
- 32:13used in that exact way. That it might be that people
- 32:18want to be able to test and do that and the might be
- 32:20considerations in how you how you code it, the might be
- 32:27considerations in terms of where your API calls are there.
- 32:31Often things that limit what you can do on can a rapid
- 32:35client-side testing. So, yeah, I guess fundamentally.
- 32:39It's breaking down. What are we actually doing here?
- 32:43What's our evidence? What's our hypothesis?
- 32:46Acknowledging that uncertainty because we don't necessarily
- 32:50know what impact they'll have. And then for each of them being
- 32:54deliberate in either saying, we want to test this now or we want
- 32:59to be prepared to test this. Or we think this is a high
- 33:03priority. We think this is a lower risk
- 33:08and with that process as well, you can even then baked in how
- 33:12we're going to monitor it. If there is that uncertainty, do
- 33:14we have the Tracking in place? What if that assumptions wrong?
- 33:19So, yeah, I excess me things. I think I work in this field
- 33:25because I get to get on with stuff and make an impact.
- 33:28So, I'm really not someone who advocates for excess process and
- 33:33meetings and paperwork, but I think boiling down your changes
- 33:36in sir. What are we doing and why?
- 33:38And what's our plan based given? We don't know for certain what
- 33:42will happen. It should be an We valuable
- 33:46process and it in and of itself and prepares you to make the
- 33:50most of experimentation, hypersonics helps, e-commerce
- 33:54companies make more profit every day.
- 33:58This AI driven platform delivers recommendations for pricing and
- 34:01inventory that lead to bigger profits.
- 34:05Visit hypersonics, dot AI, forward slash podcast to get a
- 34:09free trial. I think that point about setting
- 34:13expectations developers about Are you might want to go test in
- 34:16the Futures at that. That's a really good ones about
- 34:19here, actually, because you might not have the time frame to
- 34:22do test and validation during a bill depending on what the some
- 34:25businesses have a fixed lingerie.
- 34:27You know, their guns in Newmarket.
- 34:29The new platforms got to be ready by then and actually that
- 34:32preclude real sophisticated testing but actually and knowing
- 34:36that where you might want to go to.
- 34:37So you make sure you pick the right tools or apps.
- 34:40That will support that process. I think that's that's a good
- 34:43thing but I had to be equal. One specific use case.
- 34:46I'd love to hear you talk about how you would guide people
- 34:51through and test it. So this is and I've seen the
- 34:54speed times the project there's big design piece done.
- 34:58They've like completely redesign the user Journeys, like,
- 35:01navigating you named are everyone's massively, excited
- 35:04about the new designers all, sign up for brand point of view,
- 35:07two sites in you 80 and then the team gets a bit nervous and
- 35:11they're like Christ, we've tagged you reach ancient site
- 35:13and we we haven't really put enough.
- 35:14They need to figure Vince this, how would you bring testing into
- 35:17the process? To help them identify any deal
- 35:20breakers. Bearing in mind that they can't
- 35:21is too late to do baking in the style.
- 35:24This is a, what can you do? Massaging you a teacher, just to
- 35:27see, there's no absolutely egregious.
- 35:30Do red flags to improve the chance of success a launch, but
- 35:33it needs to be light and set than, you know, that very light
- 35:36proof-of-concept easy testing. Yeah.
- 35:41Yes. Only if your upper or and it's
- 35:43not driven by user insight, you're probably right to have
- 35:46the juices and a kits. People want to build a site that
- 35:53they're proud of and that's very understandable.
- 35:58But you're one thing I very often say is a client's is your
- 36:02among the worst people in the whole world to know what it's
- 36:06like to use your site's, your typical user might give 30
- 36:12seconds or six minutes or whatever for to your site
- 36:16whereas you've given you know days months years you know it
- 36:21inside out what your value proposition is what your product
- 36:25sale where to find them. So I probably affirm those
- 36:29gestures to start with if it if it really is an opinion driven
- 36:36launch but and I think that's where At the start, I was
- 36:43talking about the, the expertise that naturally comes through
- 36:46testing through endlessly having your ideas validated or not.
- 36:51And going back to user data, new hypotheses, new testing.
- 36:57So there's a lot of, I think one of the worst kind of themes of
- 37:03the moment in social media is testings just a process.
- 37:07You never know if you're wrong or right?
- 37:08And so on and like, I totally Buy in the testing can surprise
- 37:14you, but what I don't buy into do, is that working in testing
- 37:18long-term can't give you a much more honed Instinct into what,
- 37:24what will make a difference. So one of the first things I do
- 37:29is get people that run tests day in day out to have a look at
- 37:34your site. Certainly, if we, if we're
- 37:36pretty last minute and there's not that much you can do then.
- 37:41You learn through testing that, for instance, stakeholders can
- 37:44worry a huge amount about the homepage.
- 37:47If it's below the fold on the homepage and less and you're an
- 37:51e-commerce site with a broad range of products.
- 37:55I don't know that I've ever seen that.
- 37:57Make a significant impacts on conversion.
- 38:01Whereas, if your Jitters are about the wording you use to
- 38:07describe delivery, then yes. They have those gestures because
- 38:12over and over again through testing or seen that.
- 38:16That is an area that some massive red flag and you shoved
- 38:20living sleep. If you've really redesign my old
- 38:23section based on what what you want and what you, how you want
- 38:28to be proud of it. The next thing I do is get some
- 38:34actual users to see the sign hopefully you doing user testing
- 38:38but again I would I would Get user testing, run by people who
- 38:44are laser focused on conversion, there's a great depth of stuff,
- 38:51people are interested in ux and so on.
- 38:55But you just says at that point a really about what's going to
- 38:58happen to our metrics when we put this live.
- 39:01So I would get experiment focused people who are home
- 39:04through that process to do user testing across the site and pick
- 39:08out further red flags from Kind of combination of their
- 39:12experience and what users are saying.
- 39:17And then something something I've seen I've helped in, and
- 39:23it's been really effective, is if you're able, you can put
- 39:28that. If that uat site, is, is
- 39:31effectively ready to go. Then you can host that on a
- 39:36subdomain and rather than launching it to everyone.
- 39:40You can Split test a small percentage of your traffic to
- 39:45that new sites and that like make, you know this, but since
- 39:49it's still a smaller impact than putting it by to everyone and
- 39:55you can do that for too long or that really does become a weird
- 40:00user experience. And there are those some
- 40:03trade-offs in giving people that strange experience of having a
- 40:07different site and then the old site, maybe if you turn it off,
- 40:10but They didn't care at nearly as much as you think they do.
- 40:14As long as they can find their product and buy it.
- 40:16Then be impacted. That will be fairly small.
- 40:21And if you gonna do that, what you want is to decide in advance
- 40:24how you're going to measure it. Because so many people have so
- 40:29much data and they it's not that we don't have data, it's they
- 40:32don't know where to look. So you ideally want you
- 40:35tracking? Yeah, another key Point you're
- 40:38tracking needs to be spot on. On not an afterthought.
- 40:42It's not something that gets rushed out and be put in place a
- 40:45week after the site goes live. But if you gonna do that kind of
- 40:49A/B comparison, then you you're not, you want, not just good
- 40:54tracking, but comparable, tracking see.
- 40:56You want an idea of what's a typical journey through your
- 41:00funnel on your current site and you need to think in advance.
- 41:03How are we going to compare that?
- 41:06What how we going to have the funnel on the new site?
- 41:09And I guess on those are Is that you flagged up either from your
- 41:14your own sense of what might be dangerous or testing expert
- 41:19sense based on their experience. Hopefully backed up by user
- 41:22data. As well from user testing on the
- 41:25site, you can't have those areas that you think a dangerous
- 41:31potentially just temporarily mind.
- 41:33You might go overboard but you want to track loads of stuff.
- 41:37You want to be able to go something.
- 41:39Unexpected is happening here. Yeah, what's going on?
- 41:42And you want to, you want to dig into that there?
- 41:46Again, it's something I could ramble about for a, for a long
- 41:50time. So I'll stop there and let you
- 41:54Pro where you were want to go, right?
- 41:57And we're last question, I guess the big topic on a lot of
- 42:03people's minds around crio. And I've been asked this a lot,
- 42:07but I'm probably not qualified to answer it.
- 42:09Certainly not to the level you are.
- 42:11And given your experience with smes and the more Enterprise
- 42:16Branch. You've worked with which
- 42:18Solutions do you think people should be looking at after the
- 42:21sun setting of Google optimized? And how does it differ based on
- 42:26level of brand or retailer? Oh yeah.
- 42:30So what what tools should we be using for experimentation?
- 42:34And I actually, I don't think tools nearly as important as
- 42:43understandably. The tool providers, make them
- 42:45out. I think for, for a long time,
- 42:50what you really need to run an experiment, if it's if it's
- 42:55server, side is Javascript. That's so really quickly.
- 43:00So you can serve you test fast and make those changes without
- 43:05people seeing a visible flickering.
- 43:06Liquor so you don't want something that's so slow.
- 43:10That you see the page briefly and then you see the changed
- 43:13version. So that's fundamental you, you
- 43:19need to consider your cookie system.
- 43:23So if you have a fairly strict cookie barrier, that doesn't let
- 43:29any tool fire until you've clicked, ok.
- 43:32Then that potentially means you're going to see one version
- 43:35of the site. Then click ok.
- 43:38Then you test code implements suddenly everything changes and
- 43:42that really could skew user Behavior because that that is a
- 43:46weird experience. So if that's a concern, the one
- 43:53provider I've worked with managed to develop a way of
- 44:00getting around, that is convert convert.com and they will They
- 44:06will let you kind of customize their setup so that you serve
- 44:09the change before you set a cookie and then it's only after
- 44:14users, click accept that they're then recorded as being part of
- 44:19that change. So that's a fairly critical one
- 44:22but I'm sure other tool providers are catching up on
- 44:25that. Once you start to get into the
- 44:31dreamy world of server-side testing and using feature flags
- 44:35on and on and so on. And then one, I'd only do that.
- 44:40Once you've really making the most of client-side testing
- 44:44because server-side testing is much slower.
- 44:48Whenever anyone will tell you in reality you've much more limited
- 44:53and to make a significant change, there's a lot more work
- 44:59involved, you have to be scalable and so on.
- 45:04And I don't think I really have Hi.
- 45:06This is the go-to tool for server side because it's so
- 45:09depending on your, on your Tech stack.
- 45:13But if you, if you starting out in this field, or you currently
- 45:17using Google optimize and want to know where to switch, then
- 45:22probably carry on with Google optimized.
- 45:25As long as you can. As you things, new offers come
- 45:28out, I'd certainly give convert a look.
- 45:31They worked well for me. But for anyone spending a Lot of
- 45:36time on this, I'd encourage you to spend your money on
- 45:41experience optimizers and Specialists test developers and
- 45:47not worry too much about him tools.
- 45:51Yeah, I think I think that's really good advice because when
- 45:53you do it as scale, the the process, I mean, obviously you
- 45:56have to have the technology they can deliver what you want to do,
- 45:58but the process and the process control in the discipline is
- 46:02massively massively important. Well that's it's gone through
- 46:06all the questions. David appreciate you taking the
- 46:08time to come on stage, share your insights and knowledge.
- 46:11But anyone's got any questions they want to probe.
- 46:13You more whether it's on the technology or the process or if
- 46:16they've got a project where they want to sound you out, how do
- 46:18they reach out? Yeah.
- 46:21Thanks James. You can find me.
- 46:25You can just Google Dave. Mullen experiments are seeing
- 46:29are 0 and should put me there, but you can also go to data,
- 46:33inspires.com, and that will just take you straight to my LinkedIn
- 46:39profile and feel free to connect especially if you put a note.
- 46:43When you connect saying you'd like to chat then I'll I'll
- 46:47follow the fairly quickly and yet always It's always good to
- 46:51meet new people and stare discuss these things.
- 46:54Kill a Detachment thanks very much for having.
- 46:56These are nice. Yeah I hope people enjoyed it.
- 46:59If you got any questions do you ask and thanks for listening,
- 47:02keeping your eyes open for our next episode.
- 47:03We drop it every week. Do subscribe if you haven't
- 47:06already. And we'd love that rating on
- 47:07Apple's Popeye, YouTube or wherever else you are watching.
- 47:10The missing. Take care everybody.
- 47:18More information on this topic, head over to re-platform dot.
- 47:21F m, for our audio podcasts to discuss a project or if you like
- 47:25to chat about any of the topics covered in this episode in more
- 47:27detail. Please reach out to myself James
- 47:29card and my co-host Paul Rodgers via LinkedIn and Twitter.
- 47:32Thanks again for listening and keep your ears peeled for the
- 47:34next episode.