Latest / Inside Commerce: Ecommerce Strategy, CX and Technology Podcast / EP164: Optimizely On Using Ecommerce Experimentation To Improve Digital Customer Experiences
Transcript
- 0:02Thanks for tuning in to replant podcast.
- 0:04Funspot attracting hypersonics, you're listening to myself James
- 0:07gird, my those Paul Rodgers is often exciting.
- 0:10Duties, a hello to regular says, thanks tune back in a warm.
- 0:13Welcome to our first time. We hope you like it, we'd love
- 0:16you to subscribe to get new episode.
- 0:18Let's every week which open up said every Tuesday and do give
- 0:20us a like on YouTube Spotify Apple except ra if you enjoy the
- 0:23episode. So our topic today is a cracking
- 0:25one it's really really important.
- 0:27It's using e-commerce experimentation set to Of
- 0:30Digital customer experiences eight, there are so many reasons
- 0:33why experimentation test is important but primarily put in
- 0:36data behind whether it's punches or its opportunities.
- 0:40Identified three cursory search for pain, validation by, and
- 0:43what you're doing and do a lot of e-commerce things to test
- 0:46into a. He commits of groomers, but few
- 0:48are fully Embrace experimentation.
- 0:50And we've got an experimentation and Technology later on today.
- 0:55So, we're done to do the instructions a minute we focus
- 0:59on On the optimizely platform. Used to be at the server IP
- 1:03server. Then acquired optimizing, they
- 1:05rebranded as optimizing the and today optimize the represents
- 1:09orchestration, Commerce experimentation.
- 1:12So quite interesting kind of ecosystem, and we're going to
- 1:15cover that experimentation and understand what.
- 1:18It really means type of business, it's best suited to in
- 1:21the impact of testing, on site performances.
- 1:25So first off, I want to thank David.
- 1:27Bowen, who is the man? The myth?
- 1:28The legend, the V. A product based in the UK I've
- 1:31known for years who set this up here, very kindly introduced,
- 1:34his colleague, David Carlisle, it will sell out in a minute.
- 1:38He's a senior director of product strategy based in the
- 1:40US. He's got a lot of experience.
- 1:42So we've got a really good practitioner on today.
- 1:45Is that retail e-commerce experience at Neiman Marcus work
- 1:48with clients across the product, start with experiments
- 1:52experimentation strategies in his current role.
- 1:54So warm, welcome David, how are you sir?
- 1:58Thank you so much. I'm doing great James.
- 1:59I'm very excited to be chatting with you today.
- 2:01Yeah, I love it. It's so nice to have somebody
- 2:03who's got the the Hands-On practitioner Viewpoint of
- 2:06experimentation. But also now helping other
- 2:09people to can you accelerate their technical capabilities
- 2:12around it. So before we start I'll see you
- 2:14all the fun questions during your people intro, bit about
- 2:18yourself, you roll, and then you know for those who don't know
- 2:21optimizes well how would you position optimize me for an
- 2:24experimentation. Absolutely.
- 2:27Yeah. So I have you know, a little bit
- 2:29more than the decade of experience here in the, you
- 2:32know, analytics e-commerce and experimentation space.
- 2:35As you mentioned, most of my background has been working in
- 2:39retail specifically on testing programs and then ultimately, I
- 2:43made my way after Consulting to optimizely to help customers
- 2:48adopt and use the platform to give value.
- 2:50And I've been really lucky to work with some of our largest,
- 2:53you know, Global customers. See them at Topped this process,
- 2:56see their struggles, and their pitfalls and, you know, learn
- 2:59from that and try and take that to more and more customers here,
- 3:02over the last six years that I've been in optimizely in the
- 3:06last few. Now, transitioning into product
- 3:08and and leaving our product strategy, for the
- 3:11experimentation part of our stack, which I'm really excited
- 3:14to share with you today, you know, for those that maybe are
- 3:17familiar, but don't know. Deeply what optimize he does.
- 3:20Our mission is really to enable customers to unlock their
- 3:23digital potential. You know, a lot of Customers
- 3:26have apps websites or platforms and you know, they use a suite
- 3:30of tools. They have a bunch of processes
- 3:32that they go through. They make decisions about what
- 3:34content to push, what features to build their trying to
- 3:37optimize and improve those platforms in this work is, is
- 3:40not easy, takes coordination, and effort, and development, and
- 3:44investment. And, you know, we really seek to
- 3:47build tools and platforms that help customers, you know, on
- 3:51that journey of optimizing their platforms to create the best
- 3:54customer experiences. So, That's kind of the Sixty
- 3:57Second elevator pitch of what we do and that encompasses content
- 4:02management that in that encompasses sort of Commerce
- 4:06eCommerce and B2B platforms. And then, lastly the part of the
- 4:09stack that I'm really responsible for is is the
- 4:12testing and experimentation tools that sort of sit on top of
- 4:16that stack and allow you to use data to try out, new
- 4:19experiences, receive this feedback from your customers
- 4:22behaviors in the metrics to track and and use that to make
- 4:25decisions. Improve the platform's eggs have
- 4:27not add noise to expand this about, I'm really interesting
- 4:30because I've known that by new episode, every for a long time.
- 4:33I've not seen so much of where the experimentation B has gone
- 4:36since optimizely acquisition. So I'm intrigued as well quickly
- 4:41though to help because they're, we have people of all different
- 4:44levels of seniority and experience and backgrounds on
- 4:46here. Listening, could you just
- 4:48succinctly explain from from your your perspective?
- 4:52What does experimentation mean versus testing?
- 4:54So how is Station different testing.
- 4:58Certainly I mean I think sometimes the others words are
- 5:01used synonymously, you know perhaps experimentation is more
- 5:07the cultural mindset. Hey like you know we're going to
- 5:10use this mindset to make decisions.
- 5:13You know, we're going to use this as sort of the way that we
- 5:17remove roadblocks. A lot of times I'm working with
- 5:19large organizations where they have deep-rooted.
- 5:23Traditional ways of making decisions, deep-rooted
- 5:26traditional ways of Getting to the results they want.
- 5:30And a lot of times we use experimentation to sort of pivot
- 5:34and move them into a new growth oriented mindset, you know.
- 5:37Hey, why not just test it, why not, you know, experiment here,
- 5:40what do you have to lose? And I think, you know,
- 5:42throughout today, if I say testing or experimentation, I'm
- 5:45referring to the sort of process of letting the tool.
- 5:51Sort of divide traffic into two groups and comparing the
- 5:55behaviors of these two groups. And You know, this is akin to,
- 6:01you know, what they do with pharmaceutical drugs or any sort
- 6:04of, you know, science experiment where they they have a
- 6:07hypothesis. They divide this into two groups
- 6:10and they introduced a new drug or new chemical end to the a, or
- 6:15the B group and then they monitor the results of that and
- 6:18they extrapolate that out to, you know, to actually define
- 6:22whether this medicine is working or not.
- 6:24We do the same thing on a website.
- 6:25You know, if I come to New Markets, And it knows that I
- 6:30traditionally shop these brands or perhaps it knows my gender or
- 6:33perhaps it knows my geolocation. Maybe I buy different clothes in
- 6:37California than you'd be buying in the u.k. right now because
- 6:41it's a little colder where you are.
- 6:43And so, you know, these small changes make the website, feel
- 6:46more personal and more relevant. And that often leads to more
- 6:49engagement, more customer satisfaction or in the
- 6:53e-commerce World more purchases and no, all organized.
- 6:58Citations or e-commerce data sets up for experimentation.
- 7:01Because as you say it's a cultural shift that people to
- 7:04draw decision making rather than just how we've traditionally
- 7:07done it. What type size or saw a business
- 7:10does your to specifically optimized sleaze permutation
- 7:13platform suit. So does it cover all sizes is
- 7:16more aimed at Mid Enterprise Market?
- 7:21We're all sizes, you know, we have, you know, customers that
- 7:25are small. We have customers in the mid
- 7:26market and we work with, you know, some of the largest global
- 7:29brands. At the Enterprise top, I think
- 7:32the size is, you know, the size is not as important.
- 7:34I work with testing teams, that's a mighty team of one.
- 7:38It's one person who pushes all the changes to the website is
- 7:40one person who designs, all the tests one person that reads, the
- 7:45results of the tests, you know? It's and I've worked with
- 7:48companies to have it. You know, 30 or 40 people on
- 7:51their testing team across different parts of the stack.
- 7:54So I don't think the size as important.
- 7:56The size is important as, as perhaps having some sort of
- 8:01environment that you're trying to improve or optimize and some
- 8:07way to track whatever you're trying to optimize.
- 8:11You know, for example, we have customers that manage scrolling
- 8:16text on a digital billboard, you know, they use our CMS Platform
- 8:20to manage how the text change on that digital billboard and, you
- 8:25know, that's tough to measure the impact of changing the text
- 8:29on a billboard unless you offer some sort of number that they
- 8:32can call in or text, unless you have some sort of metric you can
- 8:36measure from that. But really what's important is
- 8:38you have some sort of environment you you're wanting
- 8:41to improve and you can track a metric that you're trying to
- 8:44improve so you can measure success.
- 8:48But the key question is the skill set that's needed because
- 8:52you said about actually resourcing, you can have it,
- 8:54we're just one person versus large teams.
- 8:58I know side with great tooling comes, great responsibility.
- 9:00As you invest in technology got better user.
- 9:02So what skills would you advise people need?
- 9:04If they're getting started with tool like your experimentation
- 9:07black or what are the skills that they need whether that's in
- 9:09house or through? You know, third parties are
- 9:12working through me. I think there's tangible and
- 9:16intangible. Bill's here.
- 9:18You know, something that's tough to measure, but really valuable
- 9:21is just curiosity, you know, you know, wondering if you have a
- 9:25hypothesis, what the outcome will be in, just generally,
- 9:28being curious adds to it, but that's tough to measure.
- 9:30But I think you know, skills that are measurable are
- 9:34certainly having an understanding of how to read and
- 9:36interpret data. Having an analyst on your
- 9:38testing team, always helps, you know, someone who understands
- 9:44how the internet works. I know that's that's silly but
- 9:47If you don't understand that, you know, I might be on my phone
- 9:51here in California and I go to nike.com And that reaches out to
- 9:56a server and pulls down code, and brings it back to my device.
- 9:59And just a general understanding of that, really helps this
- 10:02typically is the engineer or the developer, or the technical
- 10:05resource on the testing team that has understanding.
- 10:08And then, you know, someone in marketing who has an
- 10:12understanding of personas as we sort of move Beyond testing into
- 10:15personalization having on a, Standing up to the customer
- 10:18bases and knowing how to communicate to your different
- 10:22personas those are valuable skill sets to have even at
- 10:26optimizely. We we have the probably the
- 10:30largest collection of of A/B Testing subject matter experts
- 10:33in the world. You know we have a lot of people
- 10:36that have been doing this for a long time and so we have to
- 10:39constantly remind ourselves that testing testing and easy.
- 10:42You know this isn't easy and we have to make sure that we're
- 10:45communicating to our customers at the right level of Of
- 10:47maturity whether you're just getting started or you've been
- 10:50doing this for five years and so you know that's sort of this
- 10:55does not all encompassing. That's not sort of all the
- 10:59resources that you need. You certainly, if you're working
- 11:03in an iOS application, with our feature flagging solution, you
- 11:06might need you know, QA resources to test the sort of
- 11:11experience before it gets rolled out to the App Store.
- 11:14And I'll certainly if you're working with our Sting content
- 11:18than perhaps somebody who understands how to push that
- 11:20content live through the CMS. These are all sort of roles and
- 11:24resources that are valuable to testing team and I think you
- 11:27will one thing that's different about testing or experimentation
- 11:30versus maybe other platforms is that it doesn't have a central
- 11:34owner. A lot of times our stakeholders
- 11:37could be someone in e-commerce and digital or it could be the
- 11:42engineering and CTO or it could be the analytics and data
- 11:45science team we often Have customers, were the primary
- 11:49stakeholder of our software is is different for each customer.
- 11:53And so I think that's always interesting to me because if
- 11:56it's owned by the analytics team, while you might need, you
- 11:59might need Technical Resources, added to your analytics team to
- 12:02build those experiments versus. Well, if it's owned by the
- 12:06engineering team, you already have those Technical Resources.
- 12:08So perhaps you need that, the analyst resources to read the
- 12:11data, to make sense of the outcomes.
- 12:13And so it kind of does shift depending on the primary owners
- 12:16of of that party. Stack.
- 12:19Attract, you'll Complete product Discovery growth engine, create
- 12:24relevant shopping experiences that convert into sales and grow
- 12:27online Revenue with personalized, search
- 12:29merchandising, and recommendation Solutions powered
- 12:32by a I find out more at a tract.com.
- 12:37Yeah. It's an interesting.
- 12:38I'm glad you pulled out the the customer inside piece because it
- 12:41they you know, I don't have the same expertise you but projects.
- 12:44I have been involved in where a lot of the testing falls down is
- 12:48what they're testing. In the principal and hypothesis
- 12:50is sound. And the analysis of where the
- 12:52issue is, is sound or the opportunity.
- 12:55But the execution is totally not then line with the customer.
- 12:58So if you get folks read. So, having your ideas, it's a
- 13:01joined up perspective to have the right skills to execute.
- 13:05And yeah, you see in any growth in in, like, product strategist.
- 13:10Owning this piece within any client, you are worried because
- 13:12I've seen a lot more in the UK specifically, in Europe, more
- 13:15product owners, being imported with an e-commerce rather than
- 13:18an e-commerce man. Ager.
- 13:21Yeah, absolutely. So this product owners are
- 13:24responsible for developing a roadmap and defining the future
- 13:28of either. The platform they own or the
- 13:30area within a platform that they own.
- 13:33And, you know, they're managing resources to build features and
- 13:37then rolling out those features and without a platform like, you
- 13:40know, optimizely or without a testing and experimentation
- 13:43mindset here. It's tough to understand the
- 13:46impact of the feature that you built and this is often how the
- 13:49those product owners are. Managers are measured as the
- 13:52impact that they're bringing to the platform investment in those
- 13:56features. So yeah, we have a lot of
- 13:59product specific use cases. Certainly with our feature,
- 14:02experimentation and feature, flagging tool set, which, you
- 14:06know, something we can dive a little bit deeper into later but
- 14:10we're really starting to push into that Persona as well.
- 14:13And and trying to make sure that people know, we're not just
- 14:16built for marketers that, you know, we have a suite of tools
- 14:19for many personas This just shut my dog's my dog decided to walk
- 14:24in and hear what you're saying. He's obviously into
- 14:25experimentation David. Yeah.
- 14:30What another question that that, that comes up a lot when I speak
- 14:34to it to the smaller teams, you don't have the widest.
- 14:37Kill says, the Seduction on the idea of like, code free testing.
- 14:43I just get platform. I could do it all myself.
- 14:45I'd love you to be able to, like, I'll teach later people
- 14:48fight. If you're an e-commerce team
- 14:49listening and you don't have a to develop a resource.
- 14:52What can you realistically test yourself without before you have
- 14:56to code? But executing test Yeah, this is
- 15:02a great question. So often, it's oftentimes, we're
- 15:07testing Program start. They start with a visual editor,
- 15:11making small changes. And then, you know, as they
- 15:13build the muscle, as they build their maturity here, then they
- 15:17start hitting two more complex tests, use cases and sometimes
- 15:22they're able to accomplish that through Visual editor and
- 15:23sometimes that takes them to the next level of needing the
- 15:26develop a resource to write code for them.
- 15:28But, you know, I would say here is It's, you know, that they'll
- 15:33sort of run in versus back-end argument or client side versus
- 15:37server side, with a visual editor, something like
- 15:41optimizely web or anything like that.
- 15:44This is modifying what we call it, the used the UI or the
- 15:47client side and so customer builds a platform they host that
- 15:51code on the server when I'm on my phone and I open that
- 15:54application, I pull that code down from the customer server
- 15:58whether that's Nike or fill in the blank for whatever customer.
- 16:01And that code loads on my phone or my computer, and it makes a
- 16:07nice looking website or application.
- 16:10But what loads there is what's modifiable by the visual editor,
- 16:15you're just modifying the code that has come from the server.
- 16:19You're not fundamentally changing, you could hide things
- 16:21or add things, but you're just modifying sort of the visual
- 16:24experience, so the color of things, the size of things,
- 16:29perhaps their relative positioning Things you're not
- 16:33necessarily changing business logic, you're not changing the
- 16:37search results or the underlying algorithm.
- 16:39That might be feeding the product recommendations.
- 16:43You're not changing any of the underlying Technologies like,
- 16:46you know, if someone performs an internal search on an e-commerce
- 16:49website, that uses an API to reach out and get search results
- 16:53and come back. So, with the visual editor
- 16:55testing, you're only changing the UI that front visual layer
- 16:59and you're only changing the code that has already.
- 17:01Eddie come back from the server and so you know that's different
- 17:06than our server-side solution which is integrated on your
- 17:09customer server. And that's where you'd be
- 17:11changing code on the server. But just, you know, just a laser
- 17:13focus on the question. I often find that the visual
- 17:17editor use cases can take a customer, you know, probably
- 17:22through their first year or 18 months of testing.
- 17:26You know, if done right and certainly you can use the visual
- 17:30editor as Really mature customer but typically after 12-18 months
- 17:34you're starting to have use cases, that would work would
- 17:37probably benefit from Custom Dev and let me kind of give you an
- 17:41example. You know, if if you use our
- 17:44visual editor and you drop ten thousand lines of code and the
- 17:48visual editor, we're going to execute 10,000 lines of code.
- 17:51We're going to give you a message that says, hey, this is
- 17:53probably not a good idea, but we execute 10,000 lines of code.
- 17:57That's probably not a good customer experience.
- 18:01For the code to come back from the server.
- 18:03And then you to wait again, for 10,000 lines of code, to be
- 18:06applied to the page that would probably cause a we call the
- 18:09flicker a probably cause, you know, bad customer experience
- 18:12and soaped. You're usually after about 12 18
- 18:16months worth, while to why's it getting the point where your use
- 18:18cases are getting more complex, they're getting larger, you
- 18:21doing fool page redesigns, you're changing larger,
- 18:25functional your site and that's when it behooves you to maybe
- 18:28think. Do I need to add a developer
- 18:30resource to code this in In as this may be a test use case,
- 18:34that's better for the server-side testing platform.
- 18:37Perhaps, this is something that we don't want to do with the
- 18:39visual editor. I mostly card States at I'm
- 18:43assuming now that the the experimentation piece is fully
- 18:47integrated across the stack is Acres.
- 18:49Let's take an example when it was a piece ever before
- 18:54rebranded. Yeah, the the business identity,
- 18:57so personalized search, it was getting Advanced merchandize and
- 19:00there's a lot of really good Rich features.
- 19:02You know, there were lots of other Acquisitions around like
- 19:06content personalization video, Etc.
- 19:09So is the experimentation piece integrated across all of those
- 19:12Those facets. So you can you do, can you
- 19:14change test different search algorithms?
- 19:18Can you test merchandiser strategies across the site?
- 19:21Now, Yeah, we've been attacking this for a few years and I'd say
- 19:27it's kind of been multiple layers.
- 19:29The first layer would be, you know, the ability to natively
- 19:34toggle on and and you know one button Implement other products
- 19:39in the, you know, e-commerce platform.
- 19:42So you are a user of optimizely Commerce platform.
- 19:46You want to add the visual editor Which we know is
- 19:51optimizely web, just toggle the button and it installs, the
- 19:55visual editor and you're off and running.
- 19:56That's probably the first layer of integration when their
- 19:58products. And so, in a certainly we've
- 20:00achieved that today and then second layer of integration
- 20:03might be. Okay.
- 20:05Now that I've installed an implemented sort of this testing
- 20:09platform and I'm making changes. You know, what are ways that I
- 20:13can maybe automate? And deploy features on behalf
- 20:18the customer, the helps them. So if you push something in the
- 20:20Commerce platform, can that automagically have feature Flags
- 20:25wrapped around it so you can toggle that on and off in our
- 20:28feature management platform. And so that's kind of the second
- 20:31layer of integration. We're Beyond just being able to
- 20:34install with one button. We also want to make it easier
- 20:37for customers to use multiple products at the same time and
- 20:41that's where we are today, you know, really focusing on the
- 20:43harmony between these platforms. Yeah, that makes sense.
- 20:48So we have an ally here but a few coins of work with you had
- 20:53sba's but I know single page applications and I know you give
- 20:58explanation of what that is and how it differs.
- 21:01And I know these are kind of phasing out and not as many
- 21:03people are using these days, but they found that implementing
- 21:07experimentation to Lynn and testing tool in was more
- 21:09challenging, did you talk to show you?
- 21:11How do you cope with a house optimizely solve that and all
- 21:14their additional challenges people?
- 21:16Understand in terms of integration.
- 21:19Yeah absolutely absurd hope you mentioned SP a spa single page
- 21:25applications you know historically coding website.
- 21:30It each time someone would click through to the next page on your
- 21:33website maybe from the home page to a category page, it would
- 21:37refresh the entire page all the code on the page and the purpose
- 21:40of the single page app is to maybe only refresh the parts of
- 21:43the pages that are necessary. When When a customer is clicking
- 21:47through. So instead of refreshing, the
- 21:49whole page, only refresh the elements that need to be
- 21:52visually changed so that you're improving the performance.
- 21:57And, you know, showing these changes and quickly, more more
- 22:02quick to the user because you're refreshing less and less of the
- 22:04page. So this is great for
- 22:07performance. But it's challenging for tools,
- 22:09like experimentation tools that really depend on this full page.
- 22:13Refresh to tell us when the customers navigated to the Next
- 22:16page. So if you're on the home page
- 22:19and you click on an element on the home page that takes you to
- 22:21the next page. If, if there's not a whole page
- 22:24refresh, sometimes tools, don't understand that the customer has
- 22:28navigate to the next page. Therefore, we don't reach out
- 22:31and get the most recent file to apply to the next page.
- 22:35Now you can see where the complication comes.
- 22:36And so, you know, specifically with optimizely, we do have
- 22:40capabilities to work with single page, application Frameworks
- 22:44react.js and the likes. And specifically to throw some
- 22:49buzz word Technologies out. We have their it's called
- 22:52mutation observers which look for any change in the code base.
- 22:56So we're no longer reliant on a full page refresh.
- 22:59We're no longer reliant on the URL, actually changing.
- 23:02We just have some technology that looks for any changes on
- 23:05the page to occur. And if we see that we can signal
- 23:08back to our servers, to send the most updated file backed up to
- 23:12my Z, so we know. Okay, the customers on a new
- 23:14page, here's what needs to be. Like so specifically, with with
- 23:18our testing platform, we do work well with single page apps on
- 23:23the web side. And on the server side, we have
- 23:27native sdks to work through the react and, and JavaScript, so we
- 23:30can kind of very tightly integrate with the way that
- 23:33single page applications work. I love this buzzer up ago,
- 23:36mutation observers, I'm going to use.
- 23:38I'm going to drop that into meat and randomly and see if I get
- 23:41blank faces of no excellent. One of the Key concerns that I
- 23:48guess people who are not on the technical side of don't
- 23:51understand. The the integration capabilities
- 23:54is that you stick more technology on you.
- 23:57Slow the page down, you slow performance.
- 23:59You compromise your overall user experience and you talked about
- 24:03the flicker of the client side versus the efficiency of the
- 24:06service side. Here to talk.
- 24:07So powders. Optimizes architecture minimize
- 24:11and help customers deliver testing without screwing up that
- 24:14page loads. Yeah, I think, I think sort of
- 24:18the concerns for performance of probably always been around but
- 24:21they've really ramped up the last five or six years at least,
- 24:25with our customer base. And I'm sure everywhere I think
- 24:29probably somewhere a large traditionally respected company,
- 24:33you know, that was a market leader published metrics around,
- 24:37you know, the percentage increase or decrease on their
- 24:40conversion based on 100 or 1000 milliseconds of delays and, you
- 24:45know, it for them, I'm for that large respected.
- 24:48Market-leading company. It was millions and millions of
- 24:50dollars and I think a lot of people saw that and thought
- 24:52what's my performance and okay, how can I increase this to
- 24:56improve conversion? So you will, I generally think
- 24:59that the focus on performance is is great.
- 25:01I often find in practice that there is, there is an
- 25:06equilibrium point here, where you know, performance gets
- 25:09about, as good as it can get without you re investing in
- 25:12technology and changing the way you fundamentally do things.
- 25:15So You know, overall, I think apps and website, website should
- 25:21be fast, but I do have customers that disregard performance
- 25:25because they can, for example. The airline's industry, they're
- 25:31almost always going to sell all the tickets on a plane and if
- 25:35you need to travel somewhere, you're probably going to wait a
- 25:38few seconds for that page to load so you can buy your ticket
- 25:42and I think now they've started introducing ways to help you
- 25:46understand. The page is loading if you go
- 25:48to, you know, an Airlines website now or you even if you
- 25:52load that and try to book an Uber you know, if you try to
- 25:55book an Uber it tells you searching for nearby Drivers,
- 25:59and it just shows you this loading icon.
- 26:02And when you're booking a new bird, you're willing to sit
- 26:04there and wait on it to do. Its thing to find you a car.
- 26:07And I think this is an example of how not everything has to be
- 26:11precise, performance. Possible.
- 26:12As long as you just tell me you're doing the job in the
- 26:14background, I'm comfortable to wait for you to do what you need
- 26:18to do. And so, I do think generally
- 26:22faster is better when it comes to Performance here, but I do
- 26:24think there's ways to handle it. If you need more time to figure
- 26:28out, The results of something. And so, you know, we've been
- 26:31focusing for the last five or six years really on this.
- 26:34So we could be ahead of our competition.
- 26:36We were sort of at first people to go to market with an
- 26:39enterprise-grade server side offering this is different than
- 26:44sort of the visual editor. This is different than
- 26:46optimizely web testing platform that we have been talking about.
- 26:49This is sdks that are natively deployed in your code base.
- 26:54They sit on your server and you make changes at the server level
- 26:57not. Right up on the front and on the
- 26:59visual level and these are virtually zero latency because
- 27:05they're executed on the customer server, you know, just to kind
- 27:10of double down into this. The reason that latency or the
- 27:13reason that performance can be slowed with a testing platform
- 27:17is because after the website loads client-side, testing
- 27:22platforms, then apply their changes. and it happens very
- 27:25quick, it happens within 100 or 200 milliseconds, but depending
- 27:30on many factors, you know how good your mobile device or
- 27:34desktop internet connection is what's the size of the changes
- 27:39that need to be applied You know, just general, you know,
- 27:45processing power on your device, all these things can affect the
- 27:48speed at which the changes are applied and if it's too slow and
- 27:53the user sees those changes applied, this is called the
- 27:56Flicker, and this is a bad customer experience because you
- 27:58saw something and then it changed before your eyes and
- 28:02that that makes you a bit skeptical.
- 28:03So by going the server-side route by natively deploying,
- 28:08sdks on the server, we don't have a secondary.
- 28:11Network call, we don't introduce downloading and applying changes
- 28:15because the changes are shipped with the page code through the
- 28:18customers app. So you know with full stack we
- 28:22completely negate the concerns around performance with.
- 28:26You know, that's our feature experimentation server-side
- 28:30testing platform with optimizely web which is this client-side
- 28:33visual editor that kind of been talking about.
- 28:36We are also the first to offer enterprise level Edge
- 28:42capabilities. So, you know, I'm not an expert
- 28:46in CD and Technology, but seating technology itself has
- 28:50been around for quite a long time.
- 28:52You know, these are companies like Akamai and cloudflare that
- 28:55have servers distributed all over the world.
- 28:57And so if I'm here in California and I go to website, I get a
- 29:01server in California, that feeds me that website.
- 29:05And if you know, somebody's in Germany or wherever and they go
- 29:07to the same website, they get a server in.
- 29:10That feeds them the information for that website and so this
- 29:12distributed network of content that's sort of Close to You.
- 29:17Geographically makes things load faster.
- 29:19This house idioms work and Edge is a little bit newer a little
- 29:24bit faster CD and Technology. It's a bit of a different
- 29:27technology and we launched the feature set where you can use
- 29:32optimizely web on edge networks. We have the only that I know of
- 29:37sort of enterprise-grade edge network, but in addition to
- 29:39that, That there's a lot of built-in features that help with
- 29:44performance. Hear something we call Auto page
- 29:46trimming. So, if you have setup Pages,
- 29:51audiences and events and optimizely, and they're not
- 29:53being used in an active experiment, they're sort of
- 29:56archived and removed on your behalf.
- 29:58So this takes them and make sure that there's not unnecessary
- 30:01things loading, increasing the performance.
- 30:03And also, we have Integrations with the four major Cloud
- 30:07providers. As I mentioned, Akamai, Cloud,
- 30:09Fair, fastly and another's where you can self host the SDK.
- 30:14So instead of us hosting it on the My zcd end.
- 30:19You can take on the responsibility of hosting it on
- 30:22your CD n where your website or where your application is
- 30:25hosted. Therefore, reducing one outbound
- 30:28Network request. Basically, minimize the impact
- 30:30performance. So, because we've been doing
- 30:33this for the last five or six years, we have It had the great
- 30:37opportunity of working with customers and designing
- 30:39Solutions here that fits their performance needs.
- 30:42And so we I think we have some great perspective on how we can
- 30:46you know get to that right performance for each customer.
- 30:49Is it? Good point though?
- 30:50The the speed isn't the only performance measure here and I
- 30:54think that's such an important thing.
- 30:55Because the success criteria, you've got to have the right
- 30:59metrics. Nail down to know that slower
- 31:01speed could not impacts like commercial metrics and you could
- 31:04get a better overall conversion. Zhi re by improving the user
- 31:07experience, but slow the speed down and there have been good
- 31:09case studies show that. So yeah, I'm glad you, I'm glad
- 31:11you pulled the how I guess it's interesting in the server side,
- 31:15point of view, enables businesses, who the impact of
- 31:19maybe making large strategic changes like changing search
- 31:23technology. For example, enables them to
- 31:26test that effectively. So you could, you could
- 31:29basically a be run, two different search engines against
- 31:33each other and integrated apps. Better speed.
- 31:35Has anyone ever done a huge? Yeah, this is a huge use case
- 31:40here testing technology versus technology.
- 31:44And I tell this to every customer that will listen to me.
- 31:48You know, maybe they'll do it, maybe they won't, but I
- 31:51certainly say they should is to sing your way into your Tech
- 31:53stack. And we have a number of customer
- 31:56examples where they had this incumbent search vendor and they
- 32:01had a new sexy vendor come in and tell them, they really
- 32:04wanted their business and And they would give them better
- 32:07technology for a better price. And so our customer would use
- 32:10optimizely feature, experimentation to A/B test one,
- 32:14vendor versus another. And something really interesting
- 32:17happens when you do this, because when you go to your
- 32:20incumbent vendor and you tell them you're going to test their
- 32:23technology against a new vendor, all of a sudden they show up
- 32:27with a bunch of new strategies. They show up with a bunch of
- 32:30ways to improve your experience in your like, where where has
- 32:33this been the last decade we've been paying the bill.
- 32:36But what else happens is, when you run the test, let's say the
- 32:41new, the new technology actually does improve performance and
- 32:45does improve metrics, and you can measure that Roi, you know,
- 32:49exactly how much return on that investment you're going to get.
- 32:54So you kind of know how much you want to pay for that technology
- 32:57if it's going to bring you a million dollars in incremental
- 32:59search Revenue, well, you probably don't want to pay more
- 33:02than a million dollars for that technology.
- 33:04So it gives you some leverage in the The contract negotiation to
- 33:07say, hey this is what your Technologies Worth to us.
- 33:09And so this is what we're going to pay.
- 33:11And this is really, really cool and mature way to use our
- 33:15software to test your way into your Tech stack.
- 33:17Yeah, I think that's a really powerful thing because not
- 33:20disrespect to, to some technology companies out there.
- 33:23But, you know, people put full positive test runners in case.
- 33:26Those, of course you do that's marketed.
- 33:29Well, the the claims of, you know, double your conversion.
- 33:32It's just the use cases are so varied in the impact.
- 33:35Changes from from business to business.
- 33:37There, you really have got to know you're right up.
- 33:40It's an interesting view, but actually there's strength in the
- 33:43negotiation, not just the confidence to change interested.
- 33:47Let's talk about the operational implications because that for
- 33:51people who aren't used to run in, no large-scale testing and
- 33:55experimentation strategies. I guess they can think about, oh
- 33:58my God, this sounds like there's gonna be a lot of work.
- 34:00How will I manage it? How I stay on top of it.
- 34:03Yeah. How do you coordinate 0 and
- 34:05probability? Eject manage, can you talk
- 34:06through the program management tool in the optimizely provides
- 34:10or impact? This has on my a planning and
- 34:12management? Efficiency.
- 34:14Certainly yeah, I love this question because often it's
- 34:19often overlooked, you know, when we go through an RFP with a
- 34:23customer and their planning now, how much they need to spin and
- 34:25technology, or they often Overlook what's necessary to
- 34:29actually coordinate and run and the collaboration piece
- 34:32accessory, I use the the analogy of an eye Iceberg here with
- 34:36customers a lot. You know what most people see in
- 34:40your organization is the tip of the iceberg.
- 34:42They see the hypothesis and the outcome, it's like, what did we
- 34:46think was going to happen? And what actually happen is all
- 34:48they see, but underneath the surface of the iceberg certainly
- 34:53has greater mass and there's all this work and alignment and
- 34:57collaboration and planning and documentation.
- 35:00The development of the test, cueing the test Reading the
- 35:05results and putting this together.
- 35:06Like, there's so much work that goes into the outcome of a test
- 35:10and most people just see the Okimoto test.
- 35:12And we noticed this and this is one of the biggest requested
- 35:17features. We're using a voice of the
- 35:18customer here and they just kept saying like, hey, your platform
- 35:21is great but like, I just need something to help me coordinate
- 35:25this, because it doesn't really fit in jira, and it doesn't
- 35:28really fit in Microsoft Excel, and it doesn't really fit in
- 35:32here and Slack. I just don't how do I manage
- 35:35this. And so we began building a set
- 35:38of features to help you know project manager experimentation
- 35:42program we call it program management and then over the you
- 35:46know course of the last few years we've been actively
- 35:49working on building in our sort of CMP workflow capabilities
- 35:54into experimentation. So the same sort of workflows
- 35:57that are our content management system uses to help content
- 36:03producers. I berate with designers and
- 36:05content producers and publish content live, we've created a
- 36:09set of workflows specifically designed for, you know,
- 36:11experimentation where it can help you gather the hypothesis
- 36:15and assign stakeholders, and manage this test idea of through
- 36:19your testing program. It also, you know, becomes one
- 36:23Central repository for your learnings, which I think is
- 36:26often overlooked as well. You know, you could be testing
- 36:28and having a lot of tests wins. But if they're just going all
- 36:30over the place and they're not kept in a central Repository,
- 36:34You know, everybody who comes along that joins that testing
- 36:37team starts from zero. They don't have sort of this
- 36:41Collective knowledge of the test that haven't haven't worked.
- 36:44And I think this is a really important piece of your Healthy
- 36:47testing culture is to have a repository where somebody can
- 36:50read a little all the previous test outcomes and then begin to
- 36:53make better judgement about what they want to test because they
- 36:57you know, have the understanding of what was testing the past and
- 36:59so it we believe this is a big part of the undertaking of A
- 37:04testing program and we're investing heavily here and, you
- 37:07know, Shameless plug, but you could go to optimize e.com and
- 37:10look under the orchestrate Tab, and, and sign up for this for
- 37:14free and test it out today. And you can see, sort of the
- 37:18sort of workflows that we have built out to allow you to manage
- 37:20content or experimentation. I think that's a really
- 37:24important point about its glad you folks in the lair as well
- 37:28because documentation and the sharing of knowledge Within In
- 37:34larger businesses around Tess. I work, I did some work with the
- 37:37massive account, Bible accountancy company ones, and
- 37:40they're just, there was no knowledge base.
- 37:42And you had people who've done really good testing in one
- 37:44country or even different teams within the same country and
- 37:47nobody had visibility there. So you had to publication of
- 37:50effort and no ability for what works somewhere to be
- 37:54transmitted down. So there you do the testing lead
- 37:57in France could see what the UK's on ago.
- 37:59Well that's a great idea for Tess actually while we test in
- 38:01that 88 skated subtle. It's such a brilliant thing to
- 38:05do when you're right. I mean I said I say small
- 38:06technology selections again is the operational implications
- 38:11Process Management. Documentation, I how will you do
- 38:13that? How are your maintain control?
- 38:15So yeah, interesting to hear, thank you.
- 38:17Yeah, here's a fun question for you.
- 38:21What are some of the most effective uses of optimizely?
- 38:25You seem from working across different lines, I any big wins
- 38:28or any. I Eureka most only on all.
- 38:30That's a very, very School bit of testing.
- 38:36Well I think I think yes I think it's what we typically try to do
- 38:43here is teach Frameworks as opposed to specific strategies
- 38:48and I'll give you some examples to build this out.
- 38:52I think it's better to talk about these high-level buckets
- 38:56or strategies and then to help customers develop use cases
- 38:59within each bucket. As opposed to just give them a
- 39:02couple of use cases, you know, for example, The framework that
- 39:07we use is called the value driver framework and will often
- 39:09go to customers and say, hey, we took a look at your, you know,
- 39:14at the platform and the last 100 tests that you've ran have been,
- 39:17you know, focused on increasing revenue and that's great.
- 39:20But you do know, there are other areas you can drive value
- 39:23through experimentation. Let us share with you some of
- 39:25those areas and so you know increasing revenue and
- 39:28engagement is a very effective strategy because usually that's
- 39:32what it's going to make an executive happy, you know and
- 39:35Obviously you have to make Executives happy but you know
- 39:38also there's say sort of say a strategy of reducing the
- 39:43customer interaction costs. You know.
- 39:46Let's let's think about in the in the use case of retail, do
- 39:50you want to offer a chatbot to all customers or do you want to
- 39:53offer the sort of the chat functionality for just customers
- 39:56that are logged in? Or do you want to offer just
- 39:59that that ability to chat and get information to customers
- 40:03that are logged in that are your highest spending?
- 40:05Whirs, you know, what is the cost of that chat?
- 40:07Bottom, should we offer it to all customers that come to the
- 40:10website, or should we offer to the customers that are most
- 40:13valued customers that have the highest lifetime value?
- 40:16And how does that change the marginal spend on this
- 40:19technology, right? And so maybe reducing customer
- 40:22interaction cost is a strategy that you need to have multiple
- 40:26tests to try and go improve or mitigating risk.
- 40:30You know, this is really big sort of value driver in the
- 40:33server-side feature. Experimentation, Feature
- 40:36management space. You know, we talked about
- 40:38product managers, as a Persona, that we often work with their
- 40:41building features, they're willing out features and their
- 40:44greatest fear would be. It comes time to rule out there
- 40:47feature and they roll out their feature and it doesn't work.
- 40:50There's a bug in production, it couldn't handle production, load
- 40:54traffic, whatever the circumstances.
- 40:56And so you know, through a healthy feature testing feature
- 41:00rollout process that you do with our software, you can minimize
- 41:04the risk of having Bugs in your production code.
- 41:07And even if one does happen to slip through, you can use a
- 41:11toggle switch to roll that back very easily and optimize League
- 41:16feature management. And so, you know, this might be
- 41:18an area of where it's not just one use case, but is the
- 41:21practice of progress or delivery of Feature, rollouts Feature
- 41:25flagging to give you that risk mitigation, that's important.
- 41:28And, you know, I always make the same joke here.
- 41:32A good testing strategy, a good testing Road.
- 41:35App is like a good stock portfolio.
- 41:37It's Diversified. And I think you need to have a
- 41:40set of use cases at each one of these value drivers so that you
- 41:45can show that you're adding value across multiple areas of
- 41:47the business. But also you don't have all your
- 41:50eggs in one basket because, you know, if if you have all your
- 41:53eggs in one basket, while you're much less likely to have as big
- 41:56an impact as you could. So that's how I feel about that
- 42:00and I know, you know, testing and experimentation is a little
- 42:04bit different when it comes to planning.
- 42:05Ting a roadmap. You know a lot of times when
- 42:08you're planning your in your in the product or like myself and
- 42:11working on a product you have the luxury of planning a year
- 42:14out. You know, here's what we want to
- 42:16accomplish in 2023 where with testing experimentation
- 42:19hopefully you're incorporating your findings along the way and
- 42:22so because you're continually optimizing and incorporating
- 42:25those those insights with this materially changes, your roadmap
- 42:30every month, every quarter. And so, I think planning a
- 42:33little bit shorter, you know, having it published Really
- 42:36accessible for the next. Three months of here are test
- 42:39ideas. And here are the buckets are the
- 42:40strategies that those tests ideas are seeking to optimize.
- 42:44And then each quarter sitting down and sort of refreshing that
- 42:47to make sure you're incorporating.
- 42:48The learnings that I think is really powerful strategy that I
- 42:51think spotted actually cuz I'm really Keen to you because you
- 42:54worked see ya retailers started naming Marcus.
- 42:57And you come into two big technology company and work
- 43:00across different clients. So you've got a lot of
- 43:03experience with roadmap planning.
- 43:05What? Tall in can Ecommerce teams used
- 43:09to help manage robots, or factory because I've I've seen
- 43:12small business where a Trello board, push it into a task
- 43:18management system works. I've seen others save who
- 43:22basically invest in more software, but other people using
- 43:24Google Sheets. What were your go to tools for
- 43:27having sensible roadmap management?
- 43:32Yeah. I mean everyone probably
- 43:36defaults to Excel and you know, we are trying to get our
- 43:40customers, we're trying to provide them the opportunity to
- 43:43get out of excel if they want to get out of excel.
- 43:45We're not we're not trying to get them out of excel, but if
- 43:48they say, hey, like we're we're so tired of all these documents
- 43:52living in Word and Excel, we'd like for it to live in one
- 43:54place. You know, that's what we're
- 43:56trying to build. So we have We have sort of
- 44:01different visual views within our program management
- 44:05capabilities. You can you know you can submit
- 44:08all your test ideas and organize these and it kanban format so
- 44:11like under review or in flight or completed.
- 44:15If that's the way you like to view your roadmap and we also
- 44:17have the ability to visualize a Gantt chart where you can put in
- 44:21the dates of. When these tests are experiments
- 44:23might launch. And you can visualize that over
- 44:25time and see sort of the, the waterfall of all your tests
- 44:29running over time and so you could use natively our platform
- 44:33do that. Today, I often see customers
- 44:37using something like are table, you know, few years back
- 44:42actually. I personally worked with are
- 44:44table to help build out some of the experimentation templates in
- 44:50their Marketplace. And so, if you go to are table
- 44:52today and you search for experimentation, you should be
- 44:54able to pull down templates, that include experiment.
- 44:58Idea, submission, form experiment, template planning,
- 45:02and experiment, road map template that help you know.
- 45:04So I haven't Sierra table. But really I'm laughing because
- 45:09most of the time it's in an Excel that makes its way as a
- 45:12screenshot into a PowerPoint and and that's that's where it
- 45:16lives, you know. I think I'm not sure on this.
- 45:19This might we might need a mythbuster this but I think
- 45:21Excel was originally intended for a way to store and capture
- 45:27recipes. They were at the original
- 45:30Inception of excel was for people to write down the
- 45:33ingredients of a recipe and save it there and wow, look how far
- 45:36it's come. It's now the tool.
- 45:39We use to manage almost everything in our day-to-day
- 45:41Tech life or had called a good few years back.
- 45:45But I work for a massive multi-channel retailer and they
- 45:49were like, billion-plus business and their entire content
- 45:53management process to upload content to their side.
- 45:55I homepage content banners was done through A Visual Basic
- 46:00macro within an Excel spreadsheet.
- 46:02So Billy a helium atom, company's entire website content
- 46:06is dependent on one person. In an IT teams ability to
- 46:09support a macro which was it work, make it work bloody well
- 46:13but it was it was mind-boggling to see that the your rhyming
- 46:16Excel is the default. Go to Google Sheets.
- 46:20Rulesheets, the I had go and I got one last question for you
- 46:24which is, let's continue the robot thing, but apply it to
- 46:28optimize the These product itself.
- 46:30Where is optimizing the head in? Are you focus on any new feature
- 46:34releases? Or is it more about it?
- 46:36Extending capabilities in existing features?
- 46:41well, so Maybe it, maybe a little context of our journey to
- 46:46would help here. Just before I share explicitly,
- 46:49what working on, but you know, we're probably in maybe our
- 46:52third our third iteration as you know, maybe our third phase if
- 46:58you if you don't like the word iteration but we're a third
- 47:00phase. You know, we spent in a first
- 47:02phase, we spend many years building, you know, platforms,
- 47:06you know from scratch to be Best in Class Point Solutions.
- 47:09And you know this the traditional CMS and commerce
- 47:13sides of the world. And then over time realizing
- 47:16that these worlds are pushing closer and closer together and
- 47:21you know, more and more people are adopting and experimentation
- 47:24mindset in their Commerce strategy.
- 47:26This is what's helping them when you know.
- 47:29So, their decision was made that we want to move into sort of the
- 47:32second phase as a company. And we want to move into this
- 47:35acquisition and merger phase where we, we buy really powerful
- 47:41technologies that can add synergy.
- 47:43To our current platforms. And so we moved into that second
- 47:47phase and that's where the acquisition of optimizely and a
- 47:49number of other Technologies like a CD p and a Content
- 47:52management platform sort of happened and then we sort of
- 47:56moved into the third phase, a couple of years ago, we said
- 47:59okay we have the pieces of our product family that we think are
- 48:02really, really important. We feel like we have the Best in
- 48:05Class pieces and now what we want to do is focus, our energy
- 48:09on implementing that. So it's seamless for our
- 48:11customers to get that Synergy. And so We can create this
- 48:15harmony, so that, if you're using multiple of our products,
- 48:18it feels like you're using one product, even though you're
- 48:21using multiple parts of our platform.
- 48:23And so the last couple of years that's really defined, our
- 48:26roadmap is is sort of weaving these Technologies together to
- 48:31make it feel like one cohesive platform so that you get the
- 48:34Synergy of having a Commerce platform.
- 48:37With experimentation build tip, or you get the Synergy of having
- 48:41a customer data platform that can an push, you know, customer
- 48:46cohort data down to the experimentation tool.
- 48:48So you can do personalization, right?
- 48:50You can see how having more than one tool here, adds a lot of
- 48:52capability and so for last couple years we've been focusing
- 48:55on buttoning up those capabilities to really make sure
- 48:57they all work. And then we constantly have to
- 49:02focus on staying relevant and Best in Class in the market, in
- 49:06terms of our privacy and compliance.
- 49:09And in terms of our approach to Performance because those are
- 49:13really key. Ends for most buyers that most
- 49:15customers out there today. And then, lastly, it is about
- 49:19thinking, how do we, how do we push the market ahead?
- 49:22And I think I mentioned a couple times today, our use of edge
- 49:26technology, Edge CD and technology, which is pretty
- 49:28Cutting Edge. I think we have one of the only
- 49:31enterprise-grade Edge networks that we can deliver content from
- 49:34as well, as we recently brought that to the feature
- 49:37experimentation side of our stack.
- 49:39So having Edge sdks. So if you want to do this server
- 49:42side but still use ad UK on And so we are definitely continuing
- 49:47to evolve and try and innovate. And the question is, you know
- 49:51how how much do we put into each of these buckets?
- 49:54If we have a bucket of harmonizing and integrating
- 49:57existing Technologies and a bucket of innovating and trying
- 50:03to drive the market forward, what's the right mix of our
- 50:06product roadmap? And I think we're probably, you
- 50:08know, probably probably 60 40 and that regard, you know, sixty
- 50:12percent of our product roadmap is Educated at a continue to
- 50:15integrate so that we provide synergies for our customers,
- 50:18that use multiple our products because we have a lot to do, as
- 50:21well as you know, forty percent focused on building new
- 50:24features, releasing new features taking customer feedback, we
- 50:28have a public-facing feedback portal.
- 50:31If you go to feedback that optimize e.com and you have a
- 50:34feature request, you can log it there, you can view.
- 50:36Log in, you can upvote that feature requests, you can see
- 50:39other customers that have upvoted that feature request,
- 50:41that's our voice of the customer tool.
- 50:43We incorporate that into our planning and so you know
- 50:46customer requests definitely go into that last bucket and then
- 50:51Innovation, you know, how do we We didn't even get into you know
- 50:56all the all the ML and the AI changes in a chat GPT
- 51:00conversation but you know how much research and data and
- 51:04information is out there that we can pull from to try and inform
- 51:07the user about what to test. I love this website I Quant
- 51:11icons out cam you go there you can you can put in a website and
- 51:15it will use its models to show you where it thinks that
- 51:17eyeballs would be attracted to on the website.
- 51:20And you know like that information exists this modeling
- 51:22and exist. And how do we pull?
- 51:24That kind of stuff into our platform to show the user hate.
- 51:27I know you're testing on that area right there, but most
- 51:29customers don't see that. They see this area at the top
- 51:32right of your page. Here's where you should be
- 51:33testing. How can we, how can we kind of
- 51:35inform our customers on better testing strategies.
- 51:40Yeah, interesting. Point you made about the the
- 51:44customer feedback and rate in the features.
- 51:46I really like I think gorgeous doesn't really notice as well
- 51:49and then you've got the visibility, I think it's really
- 51:51good because it gives visibility to people and they can get the
- 51:55inspiration. From other people's ideas will
- 51:57be a good feature. They might not have thought
- 51:59about and then upload it. So you're really nice.
- 52:01I did that really enjoyed that David.
- 52:05Thanks so much for taking the time.
- 52:07I've learned a load of new stuff as well about where it's wise.
- 52:09He said, and what It's what's capable?
- 52:10So thank you if anybody wants to but he wants to like Probot any
- 52:16of the things we talked about or they've got a potential projects
- 52:19to about how do they reach out. Yeah, absolutely.
- 52:23I'm you can find me on LinkedIn. You can email me with David at
- 52:27Carlisle and optimize e.com. You can go to, you know, the
- 52:30website and click the contact us form.
- 52:32If it's more General sort of product formation, but any of
- 52:36those means it's great to get a hold of me.
- 52:38LinkedIn, always works as well. So, and I'm excited to hear
- 52:42where you sprinkling mutation observers and your casual
- 52:44conversation. Not, I'm just gonna do my
- 52:47friends in a pub, so it's going to work but that's excellent.
- 52:52Appreciated, loved it. I'm sure the people listen.
- 52:54I've got loads of practical in size H.
- 52:56I thank you very much. And thanks everyone for
- 52:59listening as well, and to keep your ears open for next episode,
- 53:01we drop, but every week, do subscribe, if you haven't
- 53:04already. And we'd love that rating on
- 53:05YouTube app or Spotify or wherever else you listed, thanks
- 53:09very much. Take care everybody.
- 53:18More information on this topic, head over to re-platform dot.
- 53:20F m, for our audio podcasts to discuss a project or if you'd
- 53:24like to chat about any of the topics covered in this episode
- 53:26in more detail. Please reach out to myself.
- 53:28James scared, and my co-host, Paul Rogers, via LinkedIn and
- 53:31Twitter. Thanks again for listening and
- 53:32keep your ears peeled for the next episode.