[Music] [Brent Kostak] Well, hello, everyone, and welcome to the 2024 Adobe Summit event. This is the virtual session for Top Tips to Maximize Value with Adobe Target. My name is Brent Kostak. I'm a Senior Product Marketing Manager here at Adobe focused on AI, automation, all things personalization. Speaking with Ryan Roberts, he's the Principal Solutions Consultant for our Technical Validation teams here at Adobe. And we have a very great agenda covering a lot of the tips and tricks to maximize value with Adobe Target. Really want to grow your understanding of Adobe Target's AI and ML models, capabilities to uncover tips and tricks, to employ and get the most out of these capabilities of Adobe Target. So we're going to walk through how optimization has evolved, a lot of the fueling growth and trends in the market, strategies to improve and drive intelligent optimization from some insights and tips that we're going to give, and then finishing off the presentation here with automation and getting the most out of machine learning and automation within Adobe Target.

So first off, I wanted to talk about the evolution of personalization that we've seen from designing customer experiences, really starting from static, undifferentiated experiences, same experience for every customer, every visitor, to growing into more of micro-segmentation or one-to-one segmentation in real-time. Experimentation has really seen the foundation of driving a lot of insights in contextual data to personalization, but that's not enough. Consumers are demanding personalized experiences. We're seeing a lot of expectations being met when you think about personalization, and we did a study, Adobe and Insights and it showed improvements in conversion, average order values, revenue when you do scale into the real-time one-to-one personalized experiences based off segments and audiences, and how you're driving these insights and tools. Target trends fueling growth in terms of activation in customer journeys. Brands continue to really run deeper analysis into acquisition, efficiencies, and if you think about the scale of acquisition, adoption, retention, and loyalty, in terms of stages, Target's fitting into the value gap on intelligent optimization, it's fueling strong businesses to measure and drive impact to loyalty, thinking about repurchasing, new offers likely to refer. A lot of these product and marketing strategies driving potential customers to download in mobile app and activate into some experimentation that you're driving across, channels and doing some testing, and then adopting and growing into retention, focused on time to value, loyalty, and how a lot of these, you know, tips with Adobe Target will drive in terms of value across your customer journeys. Adobe Sensei is the technology that powers intelligent features across all Adobe products, and we're really going to focus in on our AI-powered capabilities of Adobe Target. So Target offers a lot of robust features in terms of testing and personalization, Auto-Allocate for dynamic allocation of visitor traffic to drive automatically to the winning experience you were testing, Automated Personalization, showing the right ranked order of offers and content to each visitor, Auto-Target really diving into the random force ensemble algorithms and Target to personalize the entire experience, right? Not just, individual content, but entire visitor experience for driving in contextual data, sequencing of a page or multiple areas and elements of an entire page. Think of it as the lobby experience that we see when you come to adobe.com an entire home page banner, mobile app, or website, multiple different locations for an Auto-Target test, and then recommendations, personalized suggestions for each visitor. So the core capabilities of AI-powered personalization and facilitating a lot of use cases that we'll see and get into here as we go through our tips and tricks, to really leverage KPIs and optimization metrics to lower acquisition cost, increase revenue, and marketing efficiency.

So strategies for intelligent optimization. First one here, I wanted to get into optimized recommendations. So embedding a Recs algorithm or test, within an A/B test for the testing and measurement. This feature lets you embed recommendations inside an A/B test, an Auto-Allocate or Auto-Target test, an Experience Targeting test, and the functionality opens up entirely new capability set when you think about determining best recommendations, placements, where the recommendations, algorithms, and different types could be used in terms of personalized for you, where the carousel or recommendations should or could be, driving more impact or revenue based off your optimization metrics, so not just thinking about product suggestions, but opening up to content, placements on your web page, your mobile apps. There's a lot of strategies that you can gain in the data-driven experimentation, insights from the A/B test to understand where your performance of the recommendations are driving those value. So reimagining how you can Target and personalize recommendations, applying to your business, but within an A/B test to foundationally see where these optimizations are best for your recommendations algorithms.

Next, I wanted to get into the differences here in Auto-Target and Automated Personalization. We get asked these questions a lot, you know, intelligent optimization, absolutely diving into a lot of the activity setups between these two tests. Auto-Target, you can think about this as the experience level. So we mentioned the entire page of experience in different elements, modules within an entire web page or mobile app. Automated Personalization really focused at the offer level. So the multivariate testing, different changes in terms of combinations of offers or suggestions. The benefit in terms of both, structurally, we're really thinking about the UX and the agnostic of elements, but the entire full VEC paged, multi-page experience within Auto-Target test. For an Automated Personalization, really looking at the offer-level, multiple different slots, or trays, and designs that you can have with an Automated Personalization test, but specifically within where you're trying to rank those specific offers or content. Auto-Target the entire page, think about sequencing of web pages here. Different trade-offs. Of course, if you're thinking about an aggregate experience for an Auto-Target test on the pager journey, you know, not understanding some of these insights at the offer-level, whereas, if you're thinking about multiple changes or thousands of products and combinations of services, Automated Personalization will be able to grow more traffic over time with those insights. Another one is traffic allocation for control. So when you're thinking about setting up these different tests, and really where the best practices are to gain the intelligence behind this optimization in these models. When you think about in a split test or a 50% test, your models will learn quicker, which is great. Speed to value is increased.

But when you're thinking about optimizing personalized experiences and maximizing ROI, when you set your control and have like 10 to 30 split here, the largest amount of traffic to those personalization models and those AI-powered activities in Target, you'll be able to drive more ROI, the models won't learn as fast, and you can see starting this allocation can increase time for models to be effective, which is one of the considerations for a personalized test. But the other flip side is when you're trying to drive insights from 50-50 or an experimentation test, it's good to start to understand how the models are being built. Once those are healthy, then you can transition over and maximize those outputs to the 10% to 30% split. So it's a balance between what tests you're running, but once those models have fully hit statistical confidence, you are ready to allocate to 10% to 30% split to drive a lot of the personalized experience on the insights that you found from 50% split.

Next page here, I want to get into personalized insights, AI model attributes and segments. So Personalization Insights reports in Adobe Target enables customers to understand how AI activities are generating lift by exposing key attributes used in modeling, some of the personalization models and even automated segments that were created from the AI models themselves so automated segments are groups of visitors defined by Target's personalization models. With automated segments and reporting, you can learn how each segment responded to Target's AI-powered features and Automated Personalization activity or an Auto-Target activity. There are different ways to leverage Personalization Insights, but you can really see the lift in terms of the important attributes that are generating confidence in lift scores from these models. So you think about browser types or device, any kind of first-party data that you're picking up with the Target tag. You're able to dive in deeper to the Personalization Insight reports. You can uncover a lot of the, you know, AI or the random force ensemble models in these tests to see what was performing, segments that are defined, new audiences and suggestions based off trends or behaviors from these segments. That's a huge opportunity to unlock new opportunities to marketing and product teams using the AI itself, trusting these Target models, once they hit Statsig and seeing some of these insights that are delivered through the personalization insight reports. Native Target reporting, but absolutely powerful and tip to use in terms of intelligent optimization within Target for personalization models.

Last one here, continuous metrics. Thinking about optimizing ML models in Analytics for Target. So Analytics for Target standing for A4T. The continuous metrics in A4T allows marketers to set a modeling goal to non-binary metrics. So not just conversion our engagement based but now revenue, time onsite, page views, for an Auto-Target and Auto-Allocate activity type. So the machine learning model is powering these activities. You can now optimize to the metric, maximize revenue, maximize some of the ROI here. Customers looking for more sophisticated bandit type optimization test and activities, really being able to deep dive into the analysis and lift, and provides enhanced reporting on some complex metrics in Target. These non-binary and continuous metrics apply granular personalization analysis and maximize business impact. Really great opportunity to dive into A4T, all the enhanced reporting that you get with analytics for Target reporting, but now for continuous metrics. Building those ML models to optimize experiences and really calculating some metric values around revenue, time on site, for more bandit type optimization tests. So I want to pass it over now to Ryan Roberts who will be speaking to automation and AI powered personalization. Ryan, pass it over to you. [Ryan Roberts] Awesome. Thank you, Brent, for those trends and insights. It's great information. Now I want to share with you some tips and tricks, get into some things that we would maybe want to do inside the tool, focusing on some of the automation and AI powered capabilities within Target. Let's jump in there. First thing that we want to talk about is, kind of, the first part of the process when you're talking about Automated Personalization or Auto-Target. So when I talk about one, I'm talking about both, is bring in good data. So enrich the features. So the features, the attributes, or the items that the model is going to look at to understand who they are serving to and try and figure out what they will respond best to. So bring in that good data. Don't just rely on the out of the box stuff because it's there, but you should bring in what you can that is maybe more enhancing or more personalized or a better understanding of the customer or where they are in the journey. We have a couple of ways for you to do that. One, if you're a RT-CDP or real-time CDP customer, or you're thinking about that, when you spend a lot of time getting your profile information there, it's nice to be able to easily get that over into the models. And there's a great way to do that when you share a segment or when you activate, excuse me, an audience to Target. You can add profile attributes about your customers from their profile schemas and send those over to Target. They will get sent over and streamed over, when those segments are manifest by those. So a great and easy way to send segments over-- Oh, sorry, to send attributes over for the model.

There's other ways, obviously, traditionally, and continues to be the case is that you can send information directly into Target. You can do that in a click-stream, whether it's via an at.js implementation, a Web SDK implementation, or a server-side implementation, you can send those attributes in here. You can, kind of, see an example of some at param1 and param2 attributes that are getting sent over in a sample click-stream request. Additionally, there is the APIs.

There is a bulk attribute API or the customer upload profile API, and then there is more commonly used, our customer attributes option. So if you have a flat file that you want to upload against a bunch of customer IDs, you can do that and bring in all of that information, and that will be enriching and available for the models to use and consume and to learn and listen on. So make sure that you are doing that. If you haven't put time to sending in extra information you know about the customers, your models aren't going to do as well. They aren't going to do as well as they could for you if you'll send in more data.

Once you've done that and you've got an activity running, then the next thing you need to know is understand what features the model's actually listening to. So we have a great set of reports that you may be familiar with, and these apply to Auto-Target activities, as well as Automated Personalization activities again called our insights reports. So there's an automated segments one, which is really insightful. But the one I'm wanting to talk and refer to is our important attributes report. So this, kind of, gives us an understanding of which features or attributes is the model using when it builds the force. So you may be familiar with the random force model is what we're using in underneath to make our personalization decisions. And that is, of course, made up of you can think of it as a bunch of trees. And each of those trees has branches on them, and each branch could be an attribute that is the decision. Does this customer yes or no on this attribute? Does this customer have a greater than or this or less than this, kind of, feature? And each one of those questions that it answers or branches helps it understand what's the best decision for that customer. And a feature will play a part in each of those branches of the trees in the forest. So if it's in a lot of trees and a lot of branches, then it's going to be a really important attribute because it's getting used, and it's getting heard a lot. And other features may only show up one time or not even at all. This report kind of gives you an insight into which ones are being listened to a lot and are playing a big part in the model's decisions. If you go in there and you see a lot of things that maybe seem generic or seem out of the box to you and, you know, you've put in some personal, like, some custom attributes that it really should be really meaningful, go ahead and identify those, and then refine the models. So we have this model API that allows you to see all of those features and then block certain ones. So if you think about Target at its core, it's always been an optimization tool. And so here, it's, kind of, in our DNA, we're giving you ways to optimize these activities and to optimize the way the models are running. Instead of just listening to all of the features available and making our best guess on which ones are meaningful, this gives you an important say in what isn't meaningful and allows you to say, "Hey, it looks like these are in the top 10 attributes, these things, I don't think, belong." So I'm going to try taking them out. Go ahead and take them out. Remove them. Block them from that activity. Let it run for another couple of weeks, see what new features are showing up. If it's still not the features you want to that you think should be there, go ahead and do another round of refinement and blocks those things and iterate and figure out what is going to work. So you may have to do it a couple of times to get the features that seem right and that end up giving you the best lift to work. So don't be afraid to get in, iterate, and make some adjustments that are not.

Again, that's the model APIs. Great feature. All right. Changing directions a little bit to next-hit personalization. This applies to Auto-Target, Automated Personalization, but also your experience Targeting activities, your A/B activities. Next-hit personalization is something we've talked about a lot in the last year or so, a new feature that allows you to do a lot of things that were pretty difficult to do before we had this capability linking our CDP, RT-CDP audiences to Target, and being able to be next-hit personalized. So once that visitor has done that action or perform that action on the very next-hit, you can make a decision and personalize them in a different way once they become known, which is an awesome capability and feature. You can do that with Target and sometimes with profile scripts if it was web-based data. But it was difficult in the end to write a script. And there's other use cases that connect it with offline data that you couldn't do very easily before this. So this has been a great featuring capability. But some people have run into the, "Oh, I want it to be more than just the last hour." So this, sorry, the last 24 hours. Edge audiences do have this constraint of 24 hours. So what I'm telling you here are two different audiences. You can see under update frequency, one is in edge, which is the last day's worth of information, but the other one is streaming audience. And streaming audiences aren't going to be next-hit, but they will be, you know, very fast. And so it's difficult to say, "Oh, I want to do this one or that one and make you have to decide that." Well, Target is coming to the rescue here, right? You can send both of those audiences into Target, and Target has the ability to combine them into a single uber-audience that looks at both of those things. So I want to show you that really quickly how I would do it in Target.

So we're going to...

Let me switch over here to my Target window. You can see I'm in an activity. I started composing it on the experiences set. Now I'm on step. Now I'm on the Targeting step. Under the Audiences, I can change my All Visitors to the Replace Audience. So let's do that. It's going to show me a list of audiences here that I have available to me. You can see some are from Target, some are from Experience platform. Right away, you can see I have a Lightroom Product used in the last 30 days. Because of the way I named that, I know that's a streaming segment right there. But I also want to capture people if they just did it. And if I scroll down here, I have one. Because it say they the Lightroom Product View today, so I'm going to check that one as well. With both of those checked, I can click this Combine audiences button over here on the right. Target's going to say, great. You want to add these audiences together, you can add in other audiences and so on, and so forth, create containers and make it, you know, extra deep if you wanted to. But for this use case, we just need a simple or condition to say they did it today or in the last 30 days. I don't care which one, I just want to be able to consider both of them. And now we're just going to name this. We'll call this Viewed Lightroom in the last 30 days.

And we've now created an uber-audience that considers both of those RT-CDP audiences, the next-hit one, which we'll cover today, and the streaming audience, which will give me a much longer look back window and be a much more powerful option for me in this scenario. So I wanted to show that tip to you.

All right, coming back over here, there's one more that I wanted to share with you, and this is using and taking advantage of the Auto-Target plus A4T capability. This is something that's been around for a while. Some of you are probably familiar with. It's very popular. But I feel like a lot of people might be missing out on some of the key capabilities that this opens up to you. So if you're just looking at the A4T panel, there's probably more you can get from your analytics for Target, reporting on Auto-Target. The A4T panel is really designed for A/B testing experiences where you're trying to find the best experience that wins overall, and it gives you a lift in confidence on that. But an Auto-Target activity isn't doing that. It's a paradigm shift away from that. It's a different kind of experience. It's trying to find the best experience for a certain, kind of, visitor, and that may be different than a different, kind of, visitor. And so you may be using, there may be three or four or five winning experiences, so to speak. And so looking at it in the same, kind of, frame that we have in the A4T panel isn't always going to give you the most insight. So I like to use the free-form table in analytics to do my analysis on my Auto-Target activities. The key dimension to keep in mind is the Control versus Targeted dimension. So that's one of the dimensions available to you when you have it set up. You're probably familiar with that, but bring that in, you know, I filter to my activity that I care about, and I bring in my custom or my Control versus Targeted, and then add in your metrics. This is the evaluation that is probably the most applicable. How is it targeted doing overall over the control? And you can get a comparison here. And if you want to get fancy, since a free-form table doesn't have the automatic lift calculation that is available for A/B activities in the A4T panel, you can actually create one. So I created one here as an example. This is looking at my Targeted conversion rate over my Control conversion rate and generating that, hey, this activity's Targeted group is performing 177% better than the Control group, which is great. But then you may want to find another level deep, and so you can drag on top of the Targeted and the Control your experiences and then see how the experiences replying at that level. But doing it in the free-form table is, I think, a key way to get the value out of there. If I'm just looking at an A4T panel, you can't do the breakdowns as easily, and you may not be getting all the insights that you can get. So feel free to move into the free-form table and drag those dimensions in and evaluate them this way. You can see them the way I set up my metrics, and that might be helpful for you as well. I hope that's been helpful. Thank you for watching, and be participating with us in our session today. [Music]