The Ecosystem Approach: Wyndham’s Blueprint for Digital Optimization

[Music] [Danielle Harvey] Thank you for joining us bright and early. Everybody's favorite session time of 8am.

Today, we have a really great presentation for you on the Ecosystem Approach. So it's Wyndham's Blueprint for Digital Optimization. We're going to go through how Wyndham leverages a heavily integrated data ecosystem to provide an optimized digital experience for their customers.

I am Danielle Harvey. I'm head of Travel & Hospitality for Quantum Metric. Sorry, I sound like this, two nights in Vegas trying to talk in loud restaurants, and I've got about half of a voice. So bear with me over the next hour. The good thing is I'm talking the least of all three of us. I'm joined today by my colleague Adam Dille. He's our head of Engineering. And then we are joined also by Erin Boyle from Wyndham Hotels & Resorts. I'm going to let Erin introduce herself and Wyndham. [Erin Boyle] Awesome. Thanks, Danielle. My name is Erin Boyle. I'm the Director of Product Analytics & Optimization at Wyndham. I've been there for about nine years, and I lead up Product Analytics, Marketing Analytics, Digital Optimization, and Digital Data Governance. For those of you that don't know Wyndham, we are the largest hotel company in over 95 countries with 9,000 hotels, and we welcome 135 million annual customers to one of our hotels in our ecosystem. So what we're going to be talking about more is how we at Wyndham try to use everything that we have in our ecosystem to help make our customer experience that much better. The customer is the heart of our business. Our goal is to make travel possible for all, so how we're able to execute that efficiently.

So prior to joining Quantum Metric, I spent my entire career client side. So it's 16 years in digital and customer intelligence. I actually spent 11 years with Wyndham.

But was very focused on about a ton of technology I've been through. I don't know how many digital transformations. And ultimately spent a lot of my time trying to make sense of data.

And when data and systems are siloed, that can create some serious inefficiencies with very real impacts to things like your time, resources and costs. We did some research recently and actually found that the average digital team is wasting two entire months a year on issues that have absolutely no impact to their business at all. And many are spending more than two weeks just trying to actually manually reproduce issues. So this is both like surprising and not surprising to me as a person who spent a lot of time doing this kind of work. But when you see it rolled up like that, the impact of that in your business is substantial. And so then as we ask leaders, how are they thinking about spending and investing going forward when it comes to technology? They're very focused on tools that are going to streamline workflows, that are going to help them basically make decisions faster, really understand the issue faster.

And so that doesn't just apply to buying new technology, it also applies to making the most of the technology that they have today. So often that means integrating, and that's where our story comes from today. So when you integrate your technology, you get tangible benefits out of that. So you're getting usually faster time to identification or mediation of issues. You are getting better collaboration between teams, and as a result, then more productive teams. And then in general, better ROI on both of those platforms that you've integrated because you're leveling up those insights. You are essentially unlocking more value and more power from those products. So as I said, this is we're going to prove out today. Erin has some great stories. Then we're going to do like a little bit of show and tell demo from Adam so you can actually see what it looks like. Hopefully leave with some really solid understandings of how you can do some of this yourself. But I'm going to hand it over to Erin to do a little bit of foundation setting, share a little about the challenges that she faced at Wyndham that kind of started her on her journey today. Awesome. Thank you. So raise of hand, who's felt like this poor woman on the screen before while I'm trying to answer an analytics question from your senior leadership at 8am on a Monday morning, and you're like, "I don't have the brainpower to do this or the capability to do any of this." So I feel like this is me on a daily basis, sometimes hourly, sometimes minutely basis. But some of the things that we at Wyndham before we went into our integration with Quantum, we were getting questions that we just could not answer. So some of them was what content is important to a user? Some of the other ones around is the new design user friendly? Are we improving the CX experience? A lot of the other ones are around, does photos make a difference? Are photos important? What copy? And most importantly, it's, tell me why something's happening. We were so good at telling what was happening, but we couldn't really explain the why something was happening. So why is the front-end not doing something that we're expecting it to? Was it due to a back-end issue? We couldn't have that full view story before. So these were a lot of the dreaded questions that started us on this path over the last three years to try to help answer this question and most importantly, the tell me the why.

So going into the Wyndham toolbox, this is our digital tool stack that we have. So just going through, we have Amperity, which is our Customer Data Platform or CDP. We have user testing for our UX Research, Medallia as our voice of the customer, Datadog for that monitoring and API tool that we have, and then Analytics and Adobe Target for A/B testing and optimization. With all of these, that's a lot of tools for my team of, at the time, three, to help manage and understand the data to tell that story to our senior leadership of why something is happening. So there was still the gap. So I'm going to use the Grand Canyon as an example. We're in Las Vegas. So the Grand Canyon was a gap that we were trying to fill. We couldn't get all the data together. But with Quantum, we were able to integrate with everything. So I'm going to go back because that was the Hoover Dam, not the Grand Canyon. But I swear I'm smart.

The Hoover Dam, obviously, when we were going through all the different tools, Datadog's saying one thing, Target's saying another thing, the voice of the customer is saying a completely other thing, and we couldn't get these insights together. Quantum allowed us to break the Hoover Dam, and now the insights are flowing once we started to integrate with all these different tools. So once we integrated with all six, we have more insights than ever before. We're able to tell that true one story from a digital perspective to our senior leadership as a holistic team instead of, "this owner of this tool says this," versus, "I'm saying that." And it's made us more collaborative and more efficient in trying to improve that customer experience. I guess in the beginning, the customer is the heart of our business. We want to make that customer experience from start to finish the best. And we know the first touch point is the most important to a customer as well. So we need to make our digital platforms as that first touch point for some customers the best it can be to have the best journey across the board.

So-- That's me. Yep. You got it. So we're going to dive a bit deeper now into each integration. We're going to start with application monitoring, application performance monitoring. We may say APM because it's less words, and observability platform.

These help IT and engineering teams monitor performance digital platforms, but they typically focus heavily on the back-end. So they're looking at things-- They're watching for issues across like servers, API services. And while they're incredibly helpful and important in diagnosing what's happening, the challenge is that they're often pretty noisy. So it makes it difficult to figure out where teams should really focus, how to prioritize things.

And then it's also usually not easy to understand if that back-end issue is actually having an impact on your front-end customer experience.

Vice versa, right? How many times have you found an issue in Adobe Analytics and then you have difficulty tying that back to a root cause? And so that's where integration comes in really handy here. If you integrate econometric in your APM tool, you can actually link directly between the two platforms. So you can easily pivot between those back-end impacts and the actual impact on the front-end. You can then as a result, quickly and easily identifying if that back-end issue is having an impact on your customer experience. And then in addition, quantify whatever that revenue impact is. So the ideal, the result, you are identifying what's happening easier, understanding if it actually matters, and then ultimately fixing your issues faster.

Thanks, Danielle. So this is our most recent integration that we just completed with Quantum and it's tying into Datadog, so that APM tool that we have, to be fully transparent. I do not own Datadog, so I can't talk about anything in the Datadog realm, but bringing it into Quantum has allowed us to get more insights from just a digital product standpoint and for analysis. So our goal at Wyndham was to deliver a seamless digital experience, to improve that customer satisfaction. That's going to be probably on every slide, I'm going to remind you what our goal is here. But the challenges that we were having is that there was no smooth journey across all of our digital platforms from front-end frustrations and then to the back-end issues that we were trying to track. So like I said before, one of the challenges, is the front-end working correctly or what's causing the front-end to not have the goal that we're trying to obtain? So this integration is allowing us to try to answer this challenge that we have. So the resolution we were integrating with Quantum Metric and Datadog. So this approach allowed us to have more insight into that back-end monitoring and observability into Quantum Metric around that back-end system.

So what integrating did, it gave us the best of both worlds. So I am a big fan of Quantum Metric. I think they do great work as far as real-time customer qualitative data. They have the session replays. They have the interactions, and they have all this insight that we're able to capture from that front-end experience that no other tool can at the same time give. Datadog is really great at providing the deep visibility into back-end performance, tagging, and infrastructure, which is really important to the IT team at Wyndham. So what we did is we just said, let's make them work together to make one Wyndham mega tool here. So we have a portion coming from Quantum Metric, a portion coming from Datadog. So what they each do really well, combining them to make this best of both world approach for Wyndham. So we're able to connect the dots by doing this from the front-end to back-end. Is the back-end issue impacting the front-end experience, or is there is a front-end experience that might be causing some issues on the back-end? And it's allowing us to, create a faster resolution and prioritization with the team, because I don't know if you guys have the same pain I feel sometimes of trying to get tickets prioritized in the backlog to get fixed from a data perspective or from a digital perspective. But we're able to give this prioritization framework around, "Okay, let's focus on this back-end issue because it's actually impacting the front-end user experience to make that better UX experience for everyone." So what this integration looks like is we're able to send the Quantum Metric session replays directly into Datadog. So I don't know if you can see on the bottom, there's the blue square. That is the highlighting of the session replay. With Quantum, we have a 100% capture rate of all visits to the site. So we're able to get a session replay tied to every error, every event happening on the back-end. So our IT team can also see the session replay just like my team can see in Quantum Metric without having to actually go into Quantum Metric. It will deep link them into this exact session replay tied to this event or error.

The next thing we have are there's these dashboards that we have for a DevOps perspective, so giving that behavioral analytics signals. If something looks down, all the IT team can look into it and say what's happening on the back-end and kind of a back and forth communication between the tools.

So what this has allowed us to do, integration gave us the full stack insights, with truly record-breaking resolution times and quicker turnaround times for long standing issues. So there's three themes. I'm not going to read all of them, but one of the biggest ones that we wanted to highlight is that we have accelerated troubleshooting by 50%.

That's huge for us. Our teams run really lean, and trying to get issues resolved almost took an hour. Now it's taking close to 30 minutes, even less than that, depending upon the insights we're able to get from the session replays and from the signals from Quantum. So obviously, there are some times it'll take a little bit more effort, but on an average time, that mean time resolution cut by 50% is a huge win for the team. Another thing we were able to find is-- And again, I'm not an IT person, so these polluters were new to me. I didn't understand what polluters were, but there are polluters that came to the site and hit 45,000 consulate errors a day. There was no way for the team to identify them, to figure out how we can stop those from happening, and this integration allowed them to understand what they all were. They were able to prioritize and fix them, and all of those 45,000 console errors have been fixed in less than 30 days. So this is huge wins for us and just giving that full stack view. There's another thing that I personally love the most about integrating. I'm an Adobe user through and through. We have Adobe Alerts set up on every metric, every event in our tool. We also have Datadog alerts. There were times where I would say, "Hey, team, I got this Adobe alert." They would say, "I didn't get a Datadog alert," so we're not going look into it. So this, now we have Quantum Metric alerts, now the unified teams have one source of truth of what alerts we need to look into, so there's no fighting back and forth around, I see it here, I don't see it here. It's that more unified team experience of getting the alerts from Quantum Metric instead of going back and forth between Adobe and Datadog.

[Adam Dille] So we're starting at the most technical of tools in the Wyndham toolbox. And I lead an R&D team, and I have engineers that work on my team and use products like this. And they're often distracted by things like, I'm looking at log entries that happen often, I'm looking at servers that are hot, I'm looking at scaling things. And even at Quantum with the product that we build, we have to build in a little bit of making them understand how those back-end things actually have a customer impact. So I'll walk through how Wyndham does this and how all of our customers that use an APM integration like this do it. So I'm here in Datadog. Erin already mentioned it. I can click this session replay link and move directly into Quantum Metric. When I get there, you can see the Datadog Session ID in the Quantum Metric replay. You can also see that this is a two-way integration. So I can move back from a Quantum session replay into Datadog the other direction if my analysis started in Quantum.

But if we come out here to the replay, we can take a step back and just talk about what Quantum Metric does because we haven't done that yet. So our goal, Danielle mentioned it. Our goal is to get you to the why behind, why things are happening on digital. And we talk a lot, especially at a conference like Adobe Summit, about trailing metrics. So we're talking about revenue and conversion rates and task completions and that sort of thing. But when one of those things goes wrong, where do you go to figure out why that thing is going wrong? And that's the place where Quantum Metric wants to live for you. So I heard someone yesterday at the booth, one of our customers describe us as like diagnostic tool that you plug your car into when your car stops working. And I was like, "Okay," it's pretty good. I think we do more than that. But it's a good explanation of that car stopping working for you is kind of the trailing metric. And it could be 20 different things that caused it to stop working. Quantum Metric can help you figure out why. So the first step to getting to that why is one line integration of our SDK. It's available for web, kiosks, iOS, Android and Flutter. You make that one line integration and immediately we're gathering every visitor to your site, every visit they make, every page they view, everything they click, every form they fill out, every API call they make. And I stop here at the replay because this is the best place to see all of it and how it comes together in one place. Down the left rail, I can see all the pages viewed and events and things they clicked. Same across the bottom. And this is like a video player. I can click play and I can watch that user as if I was sitting there next to them use the application that I built. So this is incredible for building empathy for our customers, understanding what it looks like for them to use the experience that we've built.

But instead of watching it and using 17 minutes of our time here in this case for this replay, we've built with Generative AI something that we call Felix AI. So I can click that little fox icon at the top of the page and in just seconds, Felix AI is analyzing everything that I just mentioned that we collect in that data set and it's telling me why the user was there. So why did they go to Wyndham in the first place? What were they trying to accomplish? And then did they reach that goal? And if they didn't, what stood in their way along the way? So in this case, they were trying to book Super 8 in Winnipeg, and they had a problem with the search button. And if I look in the left rail over here-- Let's see if I can get this to work.

In the left rail over here, I'm seeing API Timeout errors. And put the whole picture together. We were looking at a problem on APM with some back-end things, and API is now failing. And all the customer knows is when I click Search, it doesn't work for me. So they're doing what we call rage clicking. They're clicking Search and it's just not happening for them.

And one more thing that I can do from Felix directly here, if we go to the bottom right corner here, there's a little button called Quantify.

Touchy. But there's a button there called Quantify. I can take the issue that Felix identified and then I can look at it in the aggregate. So this is micro, the replay, one session. I can go aggregate and look across all my sessions and I can see how much that single issue was costing Wyndham's business. Now you can also see a spike here in the middle. That's when something was going wrong on the back-end and our API timeouts were up. That search button was being rage clicked more often. And we've obscured all Wyndham's actual financials here, but I promise you under the dollar bills flying away emojis, there's dollar impact of this issue on their business. So we've gone all the way from sitting in a technical back-end oriented product, all the way to seeing how that back-end problem impacted the user, and then to quantify the business impact. That's what we call the quantified why at Quantum Metric. So let's do the next one. So the next two are going to be quicker hits. I want to make sure we dedicate time to Adobe Target and Analytics. The Customer Data Platforms-- Sorry again. They collect customer data from various sources, creating unified profile, which is then used for deeper customer analysis or marketing activations. Sometimes the challenge with that is that can be a little challenging to activate that but also to correlate with digital behaviors. You can bring clickstream data in. But sometimes the sheer volume of clickstream data can make it really difficult to understand, like what within that data set is actually important, what you really need to focus on and what's going to actually have an impact that matters. Where with Quantum, you can ingest data, your offline data from a CDP or a data lake, other data warehouse, stitch that with your digital session data, and just lend additional context to your user behavior. So it can help you gain insight into omnichannel experiences. And then ultimately, you can use that then as an input into your orchestration of journeys.

A lot of our customers will use like call center data, in store data. You could do like LTV customer segmentation data. But every travel and hospitality brand has a goal to shift more bookings direct to their direct channels and away from third-parties. So we actually started with ingesting some of Wyndham's reservation data. So I'm going to let Erin talk a little bit about how that went.

Yes, so we took our reservation data, like Danielle mentioned, into Quantum Metrics so we could look at where our customers were booking. Were they booking direct, through a call center, digital, or through OTAs or online travel agents? So what we're able to see is we identified differences in the browsing behaviors between those that shopped direct or those that booked ultimately through an OTA. So we have the journeys on the left through Quantum where we can see where the customers have booked the endpoint, how they're navigating through the site. But then we also have on the heat maps, you can see on the left that's the OTA bookers. They're more just looking to browse. They were looking at the pricing. They weren't really clicking our CTAs to get to the rooms and rates page to make that booking. But on the right, you can see the people booking direct were more likely to hit that book and pay later button. That brings you to booking. What we identified is that OTA bookers were more just browsing our website to shop around, ultimately going to an OTA to book a probably lower price if there is OTAs have discounted rates. But we're able to identify this through this bringing our CDP data into Quantum to observe this different behavioral shift that we didn't see before.

So the mechanism that Erin is talking about for bringing in offline data can be used for more than just booking. That's her use case. Another one that we've seen people use this for is to lower volumes of contact center. So calls and chats that are going into a contact center. The way that we do it is we bring in chat and call data into Quantum and we can marry that information with the sessions that generated those calls and chats. So this boils down to there's huge volumes of call and chat reaches out to support centers because the digital experience is not functioning like someone expected. So for example, how many of you have ever called into an airline or chatted into an airline because you needed to rebook your flight, you needed to change your seat, you needed to get reimbursed. That's the kind of thing that the digital experience didn't have the ability for you to do that, and so you had to reach out. So we can quantify the issues that are leading most often to calls and chats going into the support centers, and we can tell you the ones that you should work on so that you can have the biggest impact on reducing those calls, which is a huge cost driver for the business.

User experience research tools. So those help UX design teams collect qualitative feedback on digital products and experience. So helps get early insight right into new experiences. It's usually on prototypes or wireframes that don't have an analytics layer behind them. So then on top of that, you have that challenge of watching and listening to all those individual sessions and then manually collating insights out of that. So it's time consuming.

Distilling those themes isn't always the easiest. So an integration with Quantum Metric actually helps you expedite and simplify that analysis. You turn that qualitative into quantitative data, and then ideally you're validating your design decisions even faster.

Yeah. So what this-- Thank you. So we added Quantum Metric session replays again into user testing. You can see this is a theme for how I like to integrate first is always bringing the session replays in. But we were able to get quick access then for a UX team that does user testing as a team of one for our entire company. So helping her try to expedite these analysis sometimes would take her over a month to run a test for two days, analyze all the information, give it to her reports that me on the optimization team could take that insights and then create a strategy. So it was a lot of work for her to do, so adding in this Quantum Metric session replay has allowed her to expedite that analysis a little bit more. So as you can see here on this replay, that's step three of-- I have terrible eyesight, I think it's 25.

That's the prompts from user testing. She's able to watch one session replay that is able to then give her more insights into a macro level experience of how, let's say, 10 to 20 users were doing on the UX test.

We then created a dashboard for her. We have the user testing session ID so that we can take at a holistic level how did the test that Judy, was her name, shout-out to Judy, performed on her user testing. It's all at a macro level. Of the 20 people, how many had this error or this event or had this back and forth of flow within the user test. And so she doesn't have to sit through and watch the macro level trends. She can just go into the user testing and get more of those specific questions that she's looking for instead of then re-watching everything again and going through and trying to piecemeal everything from a macro level across all the users she tested with.

So I mentioned that my background is in R&D and engineering. So I know personally that engineers never make mistakes. But just hypothetically, if this new experience that we're testing and user testing isn't performing well because there's a bug that was introduced, this is a great time to find those issues before this new product goes to a wider release to a bigger audience.

Just like I mentioned things before, we're tracking these issues that happen in the user test segment so we can figure out before that gets to a bigger audience, just polish up the product and make it what it should be.

So how about VOC? So with the customer platforms, collect and analyze customer feedback, this base. Insights can be extremely valuable, the feedback is usually the vocal minority. Typically, a very small portion of your user base, not always representative of that broader customer experience. It's also frequently in the extreme-- Sorry, it's super positive or more likely super negative. Sometimes it's a rating with zero context whatsoever and no verbatims. And then my favorite, the ever helpful, your site sucks, with no actual actionable information there. And it's often one of the biggest culprits of those time wasters that I talked about earlier that impact your efficiency unless you integrate with Quantum Metrics. So when you do that, you can understand exactly what your customer has actually experienced. And then you can also quickly quantify how many are actually impacted by that. So is it like one person who doesn't really know how to use a computer or is it 10,000 people and you really have a problem? Then you can also use our GenAI summaries to help essentially increase visibility into these issues for folks that aren't in the digital team or the VOC team. You can send that out, help give them insight into the challenges that those customers are experiencing in a way that isn't super data heavy.

Yeah. So for Wyndham, Medallia Digital before was the most manual process I think I've ever had to deal with. We had a team of one again to look at all of the feedback we were getting from our customer feedback tool, and they would categorize into a digital bug product. Engineers aren't always right. A digital bug that goes into the backlog or a UX opportunity that we could take with UX team and try to do some optimization to try to figure out how to make that customer experience more important. So it was a very retroactive experience. There was always a two week delay, very manual, limited view into what the site sucks means, or if they just gave us a bad rating or a good rating but didn't give us any of that contextual feedback. My dad has always told me that feedback is a gift, if it's valuable and important and has more insights that you can action off of. So what integrating with Quantum Metric has done, we're able to get a more detailed view into the user's comments, because we added again the session replay links into our daily exports from Medallia, and it takes the guessing out. So instead of this site sucks, we're able to then walk through and watch that session that the user was performing because the session replay link is tied to that feedback.

So you can see they have the session replay link into Medallia for the voice of the customer team that owns Medallia tool. But me selfishly, I'm lazy. I don't want to go into more tools than I have to. So we added the session replay links into our Medallia readouts, but we also added in that Felix AI capability. So I don't have hours in my day to watch session replays to try to figure out what the needle in the haystack was that the customer was experiencing to give us this feedback. So we added in the Felix AI capabilities so that we can now just read this five sentence line. We can get some insights, and then we can able to quantify how many people also had a similar component within Quantum Metric. So this has been super valuable for us because now a manual process of two weeks now takes us my morning coffee at 8am on a Monday to read through all the comments from over the week in about 15 minutes. So a little bitter light reading for your coffee on a Monday morning is a lot better than sitting through and watching through hours and hours of replays.

So what we're able to do is we actually integrated this with our call centers. So how does this work outside of digital? We have a member services team that are for our loyal customers, and we were able to give them this Medallia readout with the Felix AI capabilities, and they're able to help improve the customer satisfaction. So before they were very retroactive, it would take them almost a week to reach out to a customer if they're experiencing an issue to try to resolve that issue. And at that point, it's almost too far gone. A week is way too long to try to fix an issue that a customer was having. So now with this Felix AI capability, they're able to get the feedbacks automatically as soon as they were submitted, and they could see in two seconds by reading what that customer is happening. So now we're being more proactive. We're reaching out to customers at the exact time we're getting that feedback and trying to make that resolution a little bit more quicker and more proactive to help improve that customer satisfaction and a customer experience. So what this did was it improved our loyalty bookings and our digital direct conversion. So instead of the user just going to book an OTA or go out calling the call center, they're able to now fix the issue with member services and they actually booked more direct. So we have more increased conversions, which again ties back to our goal of digital direct contribution, we want that to go up. So this isn't just for the digital people, but also for call center and specifically at Wyndham, our member services team.

So Erin showed the path from Medallia into Quantum to do this. Let's look at another path for the same thing. I get a ping on Slack that a customer left one of these 'your site sucks' survey. 'Your site sucks' is like the survey version of a drive by middle finger.

And so they didn't use their words here to tell us what is wrong. But just like Erin showed in Medallia, right here in Slack, I got the Felix summary next to the survey information and I've got a session replay link. So I might be done reading that Felix AI summary or I might want to dive all the way into the replay. And if I do that, what I got here is more rage clicking. So there's rage clicking here on this backdrop and this time, instead of from the Felix side, we can see how I can quantify this issue from the left rail of Quantum. And that lets me pick any item that's happening in the replay. In this case, it's an error. But I could pick an item that was clicked, a page that was viewed, anything there, those data points that I mentioned that we gather, and I can see more like this. And from here, I would do the same thing that I did in the APM flow earlier. I can quantify this issue, how many other people are experiencing it, how much it's costing Wyndham's business. And I've turned what was a useless survey into something that'll save a bunch of other users who are having that same issue that this survey respondent had.

Okay, I don't need to tell anybody in this room what Target is most likely. But I'm sure many of you have been in a situation where you can't really explain the result of a test. And a lot of that is from-- It's hard to anticipate every way that somebody may engage with a new test experience. And so making sure that you have everything tagged before you launch that can be really challenging. And so sometimes your target reporting your A4T even, they'll have some gaps in there. And then sometimes there's just some things that are too hard to honestly analyze in those reports. And so if you integrate with Quantum Metric, we have some incremental features that can help you do more robust analysis around that. We've got session replays. We've talked about rate heat maps. They can help give you that additional insight into understanding exactly how people are engaging. So you don't have to come up with like a list of 20 hypotheses and then try to put that into your backlog again.

And then on top of that too, we have the capability of auto capture and then tagless eventing. So if you do launch and you don't have tracking in place, you can actually still capture that data after the fact. You can basically essentially, if you didn't anticipate something that's launched, still be able to answer those questions and get insight into your test experience.

So this is our continuous product research and optimization workflow that we have at Wyndham. So I'm sure you guys all know this, but there's a discovery phase, the ideation, so building what that strategy is, the development of a test, analysis of that said test, and then you can either route to the test is done or we want to iterate or we can implement into the product fully.

So these are just some quick goals. We have six goals for digital optimization at Wyndham. A lot of them, again, ties back to the digital direct contribution, increasing conversion, increasing revenue. You probably all have the same exact goals that we have. And this is what we try to frame all of our testing too. We have a RICE model that we're able to utilize to help prioritize those tests with our testing partners. So going into the discovery phase, I'm just going to go through an example. This is my bread and butter. I love optimization. If you can't tell by more of my excitement of how we used Quantum, this is the first thing we integrated with Quantum, so we can get more insights for testing. So our current state, and this is just an example of our property details test, you could see on the left, that's our home page here on property details. And we were trying to figure out how we can increase people to go to the next page of rooms and rates. We know the highest drop-off rate in our flows or from property details to rooms and rates. So how can we try to figure out how to get more people down the flow? Once you get people down the flow, we can go to the next issue to try to get them to convert even more. So this is a true test around, page progression and getting more customers in the flow. So what we're able to do, and there's an opportunities tab in Quantum, which I love so much because they highlight opportunities that they have, and then they add in a revenue impacted number to there. So we can say this opportunity has this much revenue tied to it. So that also helps when we're trying to do prioritization of different strategies. Which ones do we want because we have this revenue tied to it? But from here, we can look at interactions. So this is a terrible screenshot, but I'm so sorry. But you can see how long our property details page is.

You can see at the top is that red, so the attention was so high on the top of the page. That dark purple, I hate that color in this screenshot, but you can see that only 0% of people scrolled 20% down the page. That's terrible.

20% down the page, you don't even get to the featured amenities on that screenshot on the left hand side. So all of our information around check-in times, the overview of where the hotel is, the maps of the area, the featured rooms and rates that we have all below the fold, 0% of people were seeing it. So we want to figure out how we can bring this information up. So using Quantum Metric of tools around more interactions, with heat maps and where people are engaging with on the site, we're able to come up with areas of things we wanted to highlight from the key importance that users are looking for.

So this is the ideation phase that we came up with.

Isn't that design cool? The half info, half photo on our screen. This was one of the tests that we loved. We were able to take those insights. We brought up information around the check-in time, check-in dates, the featured amenities, and just certain elements that guests were looking for. So we basically just shrunk that homepage hero, brought the important information up that no one was seeing before, and now 100% of customers were seeing this information.

So during the development phase, this is where we use Quantum a lot. We make sure we always create test segments. So we'll create segments for the variant and the control. And then we'll then build it into an A/B testing dashboard. During a test we can see how each variant is doing in our key metrics. So this is an example of one of our most recent ones just allowing us to monitor in real-time, how our test is performing while it's live, so we don't have to wait until the test is over to see how things are doing.

But I'm sure you're going to be shocked when I say this, this test lost. We at Wyndham were to bet money that this test was going to win. We said, "We don't even need to test. Let's just implement. This is great." But it wasn't. It lost. There was no progression down to the flow. It was literally negative in everything except for one metric.

Out of Quantum, we're able to see that there was an 18.9% lift in the photo carousel clicks button. If we didn't have Quantum integrated, this test would have been done. We would have buried it, said lost, let's try a new strategy. But with our integration with Quantum, we got that key piece of qualitative data that we wouldn't have gotten elsewhere. So what this meant and I will go to the next page so you can see the heat maps. The control on the left versus the variant on the right.

Customers were going back and forth on both arrow carousels. So what we're speculating here is that once we shrunk the image, we didn't shrink the arrow sizes on that photo. Now that it's all in one eye view, the users don't have to swivel their heads to see the arrows, they were going through and clicking back and forth on those photos. So that question, that dreaded question I had earlier, "Are photos important?" This would prove out, yes, based off this Quantum Metric heat map that we have here. So what this also is telling us is that users were not truly focusing on that search rooms and go to the rooms and rates page because the arrows were almost distracting the users from going to the key CTA here.

So what we're able to do is, on the right is the iteration that we took from this Quantum Metric qualitative insight, and we just put a little camera icon with one out of 23 on the bottom. We got rid of the arrows and we're like, "Let's just leave it on the bottom. Maybe this will help users progress a little further." So again, the integration or the iteration of this test to see if it will prove out the increase in conversion.

So good and bad news. I love when tests win. I also love when tests lose because we're able to get some more insights here. But from this test, we saw that the repeat visitors had an increase of 4% in revenue at 91% confidence. So if you're an optimization person in this room, you know that's a pretty good result. 91% confidence is what we aim for, the 90% threshold. But what this means is that the repeat users really enjoyed this experience. The new visitors did not. So what we're taking back here is, okay, what do we need to improve on for the new visitors? So again, going back into Quantum Metric and user testing, trying to find that niche thing that that new visitors are looking for so that we can improve this experience for all of our customer base at Wyndham.

So Erin talked about how you can use dashboards to report on A versus B in experimentation. That's the trailing metric that I talked about before. It's like, did this experiment do better or worse? But one thing she was saying in there is, there's a better or worse, and then there's all the minutiae within that page that explains why it might be doing worse.

Experimentation is such a visual, visceral thing. She's talking about redesigns and re-shifting things around the page. So we built an equally visual way of analyzing your page, and she's alluded to it a couple times. It's called interactions in Quantum. With interactions, I can draw a box around any part of the page that I want and I'm analyzing users who interacted with anything within that box. Now in Erin's case, she's going to analyze those users through the lens of booking because that's her ultimate success of this flow. And she can say within this box, so she's looking at, say, the image carousel that she was talking about before, are they getting to booking more or less than they did before on the image carousel? And that's how she can get to the minutiae to say, "Yeah, the test isn't doing as well, but within this variation, the actual image carousel is leading to better downstream conversion, better booking." And she can also, to compare left versus right, she can split this view here. She talked about building out her segments for A variation and B variation. She can do the same thing that I just mentioned, draw her boxes on the page and then compare left versus right, right in the same view. And most importantly, she's doing it all right within the page that she's used to working in every day. This is her job, making sure that this product page works really well. And she's comfortable with it. She's not working in a dashboard staring at a stat or a chart.

How about analytics? Last but very much not least, Adobe Analytics. So again, I'm not going to explain this one. I used Adobe Analytics for 16 years back when it was just Omniture.

I still may never forget them for sunsetting Ad Hoc, it lived in Adobe. It was critical for me to be able to do my job well, but that doesn't mean that it was perfect and doesn't mean it was the only thing that I used. So typical challenges that we often see people face with Adobe Analytics are that it's usually good at explaining or identifying what is happening, but not always why it is. Data isn't real time, but analysts typically find themselves using it for use cases where real-time would be really beneficial. So things like live monitoring for site issues. You want to know if conversion is down right now, not if it's down 90 minutes ago.

And then it's also just generally resource intensive to implement and maintain properly. So Quantum Metric actually offers a couple of different ways to make life a little bit easier, help you unlock some more power from Adobe Analytics. For one, you can send our data into Adobe itself, which can help you add that additional context into your analysis, better understand that why within Adobe itself, things like rage clicks and errors. You can also capture the session ID itself so you can then easily pivot from Adobe into Quantum, really understand why something is happening in real-time. And then again, that same auto capture and tagless eventing that I mentioned is helpful with Target. It can also help when maybe you're struggling a little bit with your Adobe Analytics implementation.

Think about when any of our breaks, now it's full of unspecified or maybe your product team released a feature and forgot to ask you for your analytics requirements or what I typically ran into was there's a project that had hit a deadline and had to cut scope and Analytics was asked to be a fast follower.

You can essentially always make sure that data is there. So you're fortifying your data foundation. Things aren't falling through the cracks anymore. You really don't need to say, "I don't know." So it's a kind of a nice complement to your Adobe Analytics implementation.

Awesome. Yes. So the behavioral analytics frustrations, signals that we get from Quantum are what we primarily use in our Adobe Analytics Workspace and different analysis. So we're able to add in those experiences around rage clicks or if there are any back-end errors that we don't see in Adobe Analytics that we get in Quantum, we can bring it back to try to do a little bit deeper dive. Again, that full stack view is what we're looking for. But you can see on the back top left, we're able to break down rage clicks by the session replay URL. Again, I love the session replays, so this is the fifth time I've said we integrated with the session replay links. But it goes directly back into Quantum. We're able to see quantify how many people are experiencing that same issue. We then look at the cohort analysis, so the retention. That's a big thing for Wyndham. One of our goals is getting repeat stays. We want to make you a loyal member, we want you to keep coming back and keep staying with us and keep booking direct. So what we were able to look as how was rage clicking affecting our customer retention? So not great. Only 2% of people come back a week later if they experience a rage click. So we need to try to improve those rage clicks so we can get those people coming back. So retention definitely does drop off. We were able to get this insight for a big project use case, and highlight how we need to improve that customer experience. Another one we have is just another sample of a workspace dashboard that ties in the rage clicks and the errors coming from Quantum, and then adding it to our most holistic dashboard that we look at on a daily basis.

One thing that Danielle mentioned is, trying to get all of their analytics, one, developed, two, developed correctly, three, getting the right values that you looked for in your requirements, and four, just making sure that it doesn't break. So at Wyndham, we have an analytics development team. Again, it's sometimes the first ones to get cut in a product, so we don't have view into anything. With Quantum, we're able to have every requirement in real-time, so we have Quantum as our main source of truth for some of our most recent enhancements, any products that we're launching, or any functionality that we were trying to prove out, we go back into Quantum for that because we just don't have it in Adobe Analytics. We're able to capture what we want from a requirements perspective in less than a day. I think it's four hours maximum has taken my team to do something like this. Instead of with the analytics development, it takes a project, it takes money, it takes resources and a whole lot of time, and sometimes the time's not even there to do. So this has helped us still get the insights that we're looking for without having to rely solely on the Adobe Analytics implementation, and we just use that Quantum Metric data that we're getting for those things. But what makes it even easier is that we integrate all of our Adobe Analytics tickets into Jira. And with Jira, it's a lot easier to get issues prioritized with adding in the session replays links again, and then adding in the revenue tied to that opportunity or that bug that we're seeing. So that opportunities tab, I think my team lives in that on a daily basis, trying to highlight any new opportunities that the tool is surfacing to us. It's basically handing us on a silver platter, "You should look at this." And we said, "Okay, let's go look at it. Let's add it into a Jira ticket. Let's add it into a backlog." And we are able to prioritize a little bit quicker, because we're able to add that revenue in the session replays. My five least favorite words are, "I can't resolve this," or, "I'm not able to replicate." "I don't know what this issue is. I can't replicate it." When the teams are trying to triage and try to pinpoint what the issue is. With session replays, I no longer hear that sentence and it is the best thing that happened to my job. We're able to watch everything. There's no guessing. There's no... I think this is it. It's we can see real-time the issues that customers were having and adding it into the Jira backlog and adding it into our prioritization. So now a resolution for bugs has been down to one week. It sometimes has been up to three months depending upon where the tickets were in the backlog order, but now being able to prioritize them more efficiently with a holistic team coming from insights, the qualitative insights from Quantum from the back-end and front-end, and everything we've talked about with all the tools we've been integrating has allowed us to have a more unified approach across team and across collaboration in trying to get these issues fixed.

Okay. And we're doing even more with Analytics. I'm incredibly excited about. So as a person who used to run Analytics and what we were just talking about is that the worst part of it is often tagging.

It's also arguably the most important part of Analytics because your analysis is only as good as your data is.

It's unfortunately incredibly time consuming. No analyst wants to spend all of their time in requirements and QA. Nobody wants to spend months and months and months, sometimes years, migrating to CJA. And so we have a few new ways to help with that to basically help you unlock even more power from Adobe and essentially just help make what's typically the worst part of Analytics so much easier. And Adam's going to take us through that. Yeah. We spent 50 minutes talking about improving the product, making life easier for the customer. And I think we all like to do that part, and no one wants to do what it takes to get there, which is tagging. So talking first about something we've already been doing for a decade. The reason that this is so fast and easy like Erin's talking about, I talked about a one line integration that you make with Quantum. That truly is the only time you add Quantum code into your application. From that point on, you're creating any custom events and errors that you need to create from our platform. So you can open up a replay like this one and you can mouse around and find the element on the page that you want. And you can say, "I want to create an event when that thing is clicked, or when something appears." So I can create an event for the booking button being clicked or an error message appearing on the page. And I can give it a name and I can push done and that definition of that event is pushed down to the clients that have Quantum Metric running and immediately gathering the data about that event. So Erin trusts us to answer a question that she has that popped up right away because we can do this so quickly. That's why she's able to get to us before she gets to Adobe Analytics to answer that question. But we want to make tagging even easier. We want to make it so easy that maybe you don't even need to think about it or do it at all anymore. So something that we've been working on that we call Felix AI Tagless. The same technology that understands the page that the user was on Super 8 trying to book a hotel in Winnipeg, it understood that's a site to book a hotel and it understood the structure of the page and why that user was there. It can also understand the structure of the page. And so it can scan the pages that are already coming through Quantum through session replays and it can find the items on the page that you should probably have evented and it can recommend them to you. So you get a ping from Felix AI Tagless that there's an event that you should create on this page. You push a button and say you do want that one. And again, through the same mechanism, it's down at the client being gathered. And importantly, Felix AI Tagless can maintain these events for you over time. So if you change something about the page that would have broken a previous event and you might not notice it for two, three weeks, Felix AI Tagless is going to be finding that event in the new definition for it so that you can have continuity in that event from old to new.

So if all this is easier, and it's already been easier than having your developer tag a page for a decade, what if we could use this to make the transition process to CJA easier? A lot of you are going through this migration path to CJA. And what if we could take advantage of the ease of tagging in Quantum Metric to get you quicker to CJA. So instead of retagging everything that you've already potentially had in Adobe Analytics for years, what we're working on now is a way to map the data that you already have in Quantum directly over into Adobe Schema. Now I'll mention here, I don't have to have my Adobe Schema created yet. I just need to know what I want to name that thing in Adobe. So if I have, for instance, a booking event and it's got five properties, I do three things. The first thing I do is understand what data points I have in Quantum Metric already and I pick out the ones that I want to map over into Adobe. I decide what I want them to be named in CJA, and again the schema doesn't have to exist yet. And then, three is we do a bunch of magic. So we call out through the Adobe API to create the schema for you based on what you just mapped. And then we automatically send those definitions, those mappings down to the client and we start to push them through the Adobe's Web SDK into CJA. So you've cut out an architect sitting in Adobe AEP building out a schema, and you've cut out a developer needing to go tag, like Erin keeps saying, we keep cutting this part of our process. You've cut out the need for the developer to go tag that to get it into CJA. So you're going to spend time making life better for your customers and doing all the things that we've been talking about here instead of worrying about the tagging that it takes to get to that end goal.

So to-- Actually, Erin, did you want to add anything about your experience? - Go on. - Yeah. So that POC that we're doing has been utterly life-changing. I don't know if anyone else is going through the CJA, AJO migration like we are. It is very daunting. The unknowns terrify me, and we eat, sleep, breathe data Adobe right now, and we want to keep that as our source of truth for digital. But what we're able to do is we're going through the process right now, and we're in, I think, six to eight months right now. We still have a few more months to go, but the POC of getting the schema set up I think took us two days? - Two days, yeah. - Two days. So we have two days and everything that we want from into CJA we already technically can get. So the two days is a lot better than eight months to almost a year, but we're able to capture all of that stuff that we already have, like Adam said, in Quantum and including all of that qualitative data that you're going to want anyway. So that was the first thing. We're like, "Let's do that. Let's add the qualitative data in there." And then also the hardest part I think was figuring out what we wanted in the schema. That was the most, I want to say everything, but obviously everything is a lot, so trying to figure out what to prioritize to prove it out. And when I said it was two days, it's two days. So we're able to activate off of this real quickly compared to, again, getting more resources and time and everything to set that up.

So just to wrap up, you can just click to the next slide. So again, just summarizing the key benefits that Wyndham has realized from that really integrated data ecosystem, better connecting data sets has had those tangible improvements. I think one of the most exciting cutting time to resolution in half, improving omnichannel understanding, increased ideation, democratization of data, but really that faster time to value and increasing the power they're unlocking from Adobe has been pretty impressive and exciting. And if you're wondering how do I do this, Erin, we've a couple-- Erin's going to share a couple of recommendations for you. Yes. So where to start? Look at your tool stack. Quantum can probably integrate with everything. Those are just the ones that we have at Wyndham. I'm not going to read through all these, so I know we're short on time. But what we do recommend is to drive buy-in, you need to find that one other team member to help you prove out why you should integrate. I'm sure you all don't own all of your digital tools just like myself, but trying to find that other person that owns another tool, prove out that POC. So what are the goals they're trying to do? What are the metrics they're trying to improve on? Working together, once they prove out their proof of concept, everyone else is going to fall in line. That's what happened with us. Everyone else fall in line. We now are integrated with all six of our tools, and we're able to activate on those to have that unified story. What I do recommend is doing more curated trainings based off those teams.

IT doesn't really need to know marketing insights from an interaction or whatnot, but if there's more of a use case that you can get from them, build that custom dashboard. If you're like Wyndham, everyone has an Adobe Workspace sent to them on a daily basis so they can log in and see an Adobe Workspace. We did the same exact thing in Quantum. So they have their own curated dashboards when they log in to Quantum, and they can see all of the insights that they have here. But my best piece of advice is to integrate with everything. It will only make the other tools have more value by integrating with Quantum, so everything's lifted up at that point. And it will allow for more cross team collaboration with insights continuously flowing. That dam has been broken at Wyndham, and it is just flowing nonstop now. So again, I just recommend integrating with everything so you can get all the insights that we at Wyndham haven't stopped getting insights since then.

Okay. So of course take the survey.

Thank you so much for joining us today. [Music]