[MUSIC] [Christopher Young] For those that don't - know me, my name is Christopher Young. This mic is a little echoey. I'm the Senior Director of Industry, Strategy, - and Marketing for Financial Services. And welcome to today's - session of the "How" of Personalization at Scale in Financial Services. Once again, we find ourselves blessed with incredible customer speakers - today. Joining me is Robyn Linter - from Bread Financial. She's going to talk about how to orchestrate an enterprise workflow - like a symphony. I'm not kidding around. That's what she's going to cover. Then Niel Seideman, and he's going to talk - about scaling an enterprise experimentation initiative and function - across an organization the size of Chase. So a lot of the discussions - that we've been having over the last few days is how do we take - the things that we do and scale them.

I think when you think about what - this industry has been doing, you know, in terms of digitization, - removing friction from the process, being customer-centric, - and the culmination of that is, you know, $42 billion leaving a single institution - in a single afternoon. 17 trillion of deposits - that are just a few clicks away from moving - from one institution to another. So I would argue that things - that we're going to talk about today are super important, - and the stakes are extremely high in terms of engaging customers and - personalization and building relationships and communicating trust and confidence - in your respective institutions.

So let's take it from the top. I'll spend the first few minutes looking at some of our most recent - survey data. This is from the Adobe - Digital Trends Report. And the idea is like, what are we all - trying to do as an industry? What is our top priority? And that is to improve the financial - health and well-being of our customers. And the reason why it says - meaningful digital interactions to do that is simply if that's where more - and more individuals will interact and you provide more higher-end advice - and guidance in 1 to 1 interactions, it's not going to help them - make smarter decisions with their money in their channel of frequent choice - and preference. So I just want to start at least by saying - this is the end game for all of us.

The challenge that - we have and what we've been talking about is that type of meaningful interaction - that operates at a scale, in many cases for those of you - in the room, in the millions. So a lot of this isn't necessarily - about me identifying someone. It's more about where they are - in the journey. This could represent different individuals - in different journeys. This could represent different individuals - in the same journey, but in different parts - or different phases of that journey. So how do you understand context - and then ultimately progress then? And the way that we think about - this is personalization at scale.

Personalization at scale for us, - and it's an afternoon on Wednesday. You're in Vegas. I'm going to keep it - nice and light. Okay? Nice and light. It's the inextricable connectivity - of three things. Data: your understanding of the institution - or consumer that you're interacting with. Content: your ability to express that understanding - with relevant experiences, and then journeys: where data and content - come together for purpose - in that moment of need for the customer. So no matter what happens, - these three things get together. They have to be connected together. And a lot of the discussions - that we've been having around technology and how our applications solve for this, - the session is a little different. This session is going to focus - on the giant red bar below that doesn't - necessarily get talked about all the time, which is people and process, - elements of change management. This is the stuff that I hear - a lot from you is what holds a lot of our customers back. Not the what, not the why, but the how.

How do we do this with the applications - and the things that we own today? How do I bring along all of my different partners - across different business units to the same cause? - How? That's what we're going to explore today - with our great customer speakers. The other thing I want to pivot into - is some of our survey data. So I said, Okay, the how has been something - that I've been thinking about quite a lot. We've done research around this, and what you're going to look at - is just some of these findings. And just keep in mind, - a lot of the findings that we have is going to get into - where do you start, which is initial, where you are today, - and then ideally where you want to be. So the starting point is, Do you agree or disagree - with the following statements? And I want to get a show of hands - here, 53% said we are happy - with our personalization performance. So by show of hands, - how many of you are happy with your personalization performance? This is like the second time - I get no hands. Like I question this entire survey now. It's like 53%. Who are these happy people? No hands.

And you're our customers. There's a lot of soul searching - we need to do. But either way, - okay, let's go with 53%. And now let's start - peeling back why this might be, - why there are these elusive 53%. Okay, so where things start - getting even lower is, We are staffed adequately to meet our personalization goals.

Even lower. We are providing ongoing training - to our personalization teams. Our personalization teams - have the right expertise. And this is the one: We have aligned our employees' incentives to enable personalization. That's a big one. If you have individual teams - that can be successful in their roles by doing things individually, - those cross-channel journeys that we're talking - about will be extremely difficult.

So let's move into the barriers of personalization - and why. I mean, this is the way - I explain what the general issue is. It's not technical in many cases. It's how you are organized - and how you collaborate. So if I'm responsible for small business - marketing, I send emails, I do paid media, - I'm merchandizing the website, they're individually focused. And the things that we're architecting - require all of these different individuals to work together - to common outcomes and objectives. Secondly, you know, how do you get things - done within your organization? Emails back and forth, marking up PDFs? Like, you know, a lot of the things - that I've been hearing is it takes us so long to do things because of this, - because of this.

So let's talk about execution challenges, - going back to the survey. So what were the top - three personalization challenges when you first started? So when we asked, initially, - what were your challenges, these are pretty significant. No organizational buy-in, no use cases, - inability to prove ROI. We had poor data quality, the data was unusable, - and we had limited budgets to do it. Okay, so that's where we started. Today, those have gone down - pretty significantly, meaning that these starting point - challenges were addressed and reduced. Now, going back to the challenges - that are most prevalent today, this is the initial numbers of inability - to integrate multiple sources of data, siloed data, lack of single executive - ownership of these types of initiatives, and we had limited skill sets doing that. These have gone up. So these, we've migrated as an industry together from, - Why are we doing this? We're not funding it. We don't have any data to use, to now, - it's like, now we've got to integrate data. And frankly, we're getting so evolved at the size of our companies, - we need someone to lead this. And then further, now we need more skill - sets to do this. So these are the things - that are challenges now.

Who owns personalization? That's a million-dollar question. But let's say, where people started was - no one was responsible for personalization, 1% had a committee - and then multiple executives, probably customer-facing, retail-oriented, were responsible for it. Today, you know, still a lot of nobody owns it, but increasing at 11% of single owner - responsible for personalization. Where do the responders think they need to be in an ideal state? Much higher in terms of having someone responsible for personalization.

Who's responsible for the customer? Who owns the customer? And I'm not saying - someone has to actually own that, but someone has to be responsible - for the creation of segments that may span different business units - targeting the same individual or business. And I get how difficult this is, you know, - because this single executive has to have the authority or even, frankly, - the P&L responsibility to drive a lot of this change more broadly, How are you organized for personalization? I mean, the starting point - was really technology and individual departments, and, - you know, my friends here in technology understand how difficult that is. The world that you're in to - not only implement this technology but then enable business owners - to use that. That's very difficult, - but that's kind of where it started.

Today, we're seeing still high - individual department IT/technology with the rise of other -- and it's SMEs. I really wish I caught that typo - before I got on stage. But either way, centers of excellence, cross-functional, agile squads - and specialized SMEs. The rise of these functions, - and in an ideal scenario, the reality is it's going to be a mix. It's going to be a mix of individual - departments, technology - that's aligned to centers of excellence that are associated with more tech, - that really bridge the gap between technology - and the business stakeholder, backed by cross-functional, agile - teams that move quickly, and then subject matter expertise - and specific areas to back them.

How were your programs funded? And, you know, the tough one is shared - where you basically have to go around with the hat to different stakeholders - and try to get these things. Oh, it's painful. It's painful. I know, I've been there. Could you please fund - my personalization work? Okay, so central, shared, prioritized, - so incremental based on business cases and added tech prioritization and - strategic funded at the corporate level. So sometimes marketing might have - it centrally funded, as an example, but today, you know, a bit of a reduction - more into the prioritized, and the ideal state - is definitely closer to prioritized. And I get that, it should be - something strategically funded, but at the same time, you should be able - to prove going into these initiatives what you believe the business - and consumer outcomes should be. So that makes sense to me.

This is going to be a tricky slide - to animate, but we're going to try this. This is experimental, test and learn. So what's the way - your organization is orchestrating end-to-end journeys? And primarily it's segment based. Segment based. So no personalization, - basic personalization, segment based, microsegment, 1 to 1.

But the ideal state, actually, - interestingly enough, when we go from visitor, no segmentation, - customer, basic segmentation, investing, could be a - microsegment of a new mom. And then actually down to Jane herself, - really it's between the two because I think there are certain areas - where 1 to 1 makes a lot of sense, and then microsegments - could be just as effective when you talk about the iterations - that you create. So when we think about Jane and - we think about these end-to-end journeys, let's talk about where, - from a channel perspective, the 1 to 1 microsegment prioritization - is going to take place, and it's mobile app for sure, - call center, interestingly enough. Okay. Authenticated web, email, and obviously 1 to 1 in direct - mail is super expensive. So I kind of get that too.

Last but not least is, - What do we believe we will get in terms of conversion - or lift in conversion by going through this journey of no - to basic to segment to microsegment? So from microsegment, - 27 or almost 28% lift in conversion, again, this is what they believe - by going through this exercise, and then as high as almost 44% on a 1 to 1 basis. So that's the survey data - that we have in terms of thinking about the how, - structure, funding. And then ultimately we want to get to - what does that mean in terms of the channels - that you're going to execute this on and what you believe - ultimately the impact will be. So this is a report that we are going - to publish with a lot more detail and anecdotes, and this is something - to look out for in the very near future. But I wanted to share these - initial results with all of you today. So without further ado, a warm round of - applause for our first customer speaker, Robyn from Bread Financial. [APPLAUSE] [Robyn Linter] Thank you, Chris. - I'm Robyn Linter. I'm the Senior Manager of our Marketing Workflow - and Tools team at Bread Financial. Bread Financial, you might ask, we are a tech-forward - financial services company. We provide payment, lending, - and savings solutions that are simple - and personalized for our customers. So today what I want to walk you through, - and Chris alluded a little bit to it, is that all of these strategies that we've been hearing about - as we go through our, you know, experiences here, there is this underlying current of, well, - how am I going to do that? Right? So you really need the - right tools and you need -- sorry, you need - a well-tuned operations team and you need the right tools - to be able to deliver that. So what I wanted to kind of walk - you through today is our Adobe Workfront journey. Okay, - so you've heard about Adobe Workfront. We launched - Adobe Workfront in June of 2020, and I want to talk you - through a little bit about how we leveraged Adobe Workfront for - us to go from an operational one-man band where you got - someone playing an accordion, someone playing a harmonica, - someone beating drums, to a full-fledged orchestra.

Oops. Yeah. So Adobe Workfront, right? Adobe Workfront supports the end-to-end - marketing workflow lifecycle workflow. And they do everything from planning all the way to analysis - and everything in between. In June of 2020, - we launched Adobe Workfront to support our marketing campaign workflows - and our marketing services workflows. We had intake forms. We have queues where, you know, - requests come in for those services. We leveraged the Workfront project templates - that have tasks, and those are assigned to very specific teams - and/or very specific individuals. They have due dates and deliverables. In addition to that, - we leveraged Workfront Proof to route, approve, and retain - our proofing workflows and collateral that's going to our customers. And we also leveraged Workfront reporting not only to show us - the KPIs of our workflow, but to actually report out - on those key metrics that you're capturing as that metadata - kind of throughout that workflow.

This is a string orchestra seating plan. And so the analogy here, right, is like you can see there - that you have a conductor, right? And as you go out on your journey - of trying to implement Workfront, you need to identify your conductor. The conductor is the glue that holds - that orchestra together. They know every section, - from their purview they can hear when someone's coming in - a little too soon, maybe timpani, or a little bit too late. Right? And so what you need to find in - your organization is your conductor. So leaders, who in your organization - thrives in workflow and has an uncanny knack and ability - at seeing that end-to-end? Who has that influence across and up your organization - and is a strong change agent? Who is that person - that when it is in their book of business, you actually sleep at night? That's the process person - you want to identify as your conductor for your Workfront implementation. Maybe some of you sitting in the audience - say, I think I am the conductor. I do all of those said things. I have a vision for how things should work - and how things should go. Well, then go and ask your leader - and tell your leader, "Let me do this. Let me lead this activity." In our business, our conductor resided - within marketing management. I might have been the conductor, and, you know, crazy here, not required, but they sat - within marketing management, right? Because your marketing manager - and your campaign manager, they see that end-to-end workflow, they're the hub that is pulling in - all of those other stakeholders. So once we identified that marketing management - should be within our conductor seat, we then needed to figure out, well, - who's sitting in all the other seats? So the way that you do that - is that you want to identify who are those key - stakeholder participants and teams that are participating in those workflows - that you are looking to implement? And in our situation, we identified that - it might have been, oh, the strategy team, the content review team, - our quality team, our delivery team, etc. This is just a demonstration, - when we had our launch, we actually had over 17 teams that had a seat at the table and we had representation - from each of those teams. What does representation look like? Well, I identified your conductor - and you know your stakeholder. You got to go into each of those teams - and you want to identify your first chair. The first chair - within each of those organizations that has a passion for improving - that process in that workflow for their team members, for their peers, those folks that are also really hungry, - that are good, strong change agents. And you want to identify that first chair, - because you know what they turn into? Your champion for that department, - because this is hard and this is massive. But if I have an embedded champion - who is along on this journey the entire way, it makes that change, - if you will, a lot more manageable. So we identified our first chairs - throughout that process. Now that we have our conductor - and you have your first chairs, you got to get into mapping the workflow, - right? And everybody says, I got to workflow. They all say that - until you get collaborative and get everybody in the same room and you start going through it, - and you say, Who are my players? What are the steps? What is the input that is required - for that step to actually be accomplished? When that step is done, well, - then what's the outcome? Is there an output? Does that output go somewhere? What's my SLA - for delivering that particular step? And then what are the current tools - being leveraged for each of those particular tasks? At times, when we sat doing this work, - it kind of felt like we were all sitting on this huge pile of sheet music - that is spread all over the place. In some instances you had this section playing that note, - this section playing that note. There's a whole pile of sheet music - that no one wants to do, right? You have those tasks within your workflow. That's when you're working - through those tough measures and you want to keep hammering that out - in that collaborative space so that eventually you have assigned out - that this sheet music and that task goes to this group and this group, and they all agree to that, - and they're all aligned to that.

What comes out of all of that work is that you are then starting to identify - cross-functionally what your pain points are, right? What is it - that you're solving for with the tool? So for us, and these are just a full - few of our thematic pain points, our tools were not nimbly adjustable. What does that mean? Well, let's say I have a dropdown - on an intake form in that dropdown gives me green and blue - and now I want to go and add yellow. Oh my God, get in line. You can maybe get that done and - prioritized in the next six to nine months. Meanwhile, by the time that's done, - I also needed orange, purple, and quite frankly - I don't need the field at all. So we needed tools - that were more nimbly adjustable.

We had seek and find project details. So that's really awesome - when you're doing a scavenger hunt and it's really terrible when you're - trying to get marketing out the door. So we knew that we had an opportunity - to kind of solution that. One of our largest - areas of opportunity was the lack of scheduling and project management - kind of tool capability. We had project plans, - we had schedules. They're done in, I don't know, Excel, Word, some tool - that doesn't automatically update. People aren't getting notified. I pulled down a wrong schedule and - I sent it to my partner, whoopsie-daisy. We also had minuscule automation - and dynamic and user experiences on top of that, - and then multiple fragmented tools. So once you've done the process mapping, - you've identified your pain points. Those two things - combine into your tool requirements. What are tool requirements? They're the very specific things that you, - for your instance, need for a tool to deliver. My favorite sentence is, "I need the ability to..." - and then you finish it. "I need the ability to provision end user - groups and I have five end-user groups." "I need the ability - to evolve my workflow over time." We had over 90 requirements - that we wrote out in great detail. Workfront did not solution - all 90 of those requirements by a, you know, landslide. But we prioritized them and it - certainly solutioned all of the big ones, and we've made sure that with our - implementation, and I'll talk a little bit about the fast follows and enhancements - we've done since then, that we continue to resolve and solution - on all of our business requirements.

So you have your requirements, now you need to find the right partner. Good news you guys, - I've already crowdsourced this for you, and you're at the Adobe Conference. So Adobe Workfront - might be a really good tool, but, in the event that it's not, just kidding. You crowdsource, - and what does crowdsourcing mean, right? You go in, you find people who are in - your industry who have similar workflows, and you take down the walls - and the barriers and you partner together. I happen to have the luxury of sitting - in a roundtable environment with other financial service - operations people. We talk about this stuff. What's the workflow tool - that you're using? Oh yeah, does that work for you? What's the good, the bad, and the ugly? And you want to hear it all. In our conversations, Adobe Workfront - continues to come up at the top. We did further research, we did demos, we did lots of engagements, - and we selected Adobe Workfront. As we went through that process, we also said, well, I would really like - to have a solid implementation partner. I'm going to be learning a tool while I'm implementing a tool - and that sure seems really challenging. So again, we crowdsourced - and actually came up with a fantastic partnership - that we have to this day with LeapPoint. They are a premiere partner - of Adobe as well, if you'd like to check them out, - they're in the community pavilion, you can tell them Robyn sent you, - I do not get a discount, unless I do. Yeah, that's what I'm saying.

But I highly encourage that you get partners to help you out - on this particular journey. So we got all together, - we have our partners, we got together, we built out collaboratively - the balance of the implementation roadmap. We're going to fast forward - to like two of the most critical components of your implementation roadmap - to ensure your success. The first one is practice. Really? Practice is also training, and promotion. So if I'm going to have a symphony, - I need to promote it so people show up. But promotion in our situation - equals communication.

All right. Training. So your conductor - and your first chairs are navigating through with the partners - all through the implementation process. Now there are lots of training tools - that are available. Adobe does a wonderful job - on Adobe Experience League. You can also do plug-ins that will help - you along with your training journey. But I want to just - throw some caution out there that when you are launching something - that is this big of a change, you need to provide your end users with role-based training - that is specific to your instance. So you know what we did? We had it all documented. We said, Hey, marketing management, - you're going to receive a request from the strategy side. Here's what that request looks like. - You're going to convert that into a project - applying the right project template. Here's how you do that - using our project templates. So I encourage you to go - that path, supplement that with the training that's - available on AEL and for your system admins and whatnot, - but focus on that role-based training. Then you schedule, - you facilitate your training sessions. You're going to have overviews - of Workfront because it's a new tool, and then you're going to get into - that real specific role-based training. Pro-tip, record the training. So my team does a fantastic job - at building out training and process guides - and facilitating those training sessions, but we are not learning and development, - right? So every time we've launched something, - we'll host these training sessions. But then it is on each of those - stakeholder groups to train their new people - that come on board, right? Like, it's not my job to train - the entire organization. I'm going to train you on the new stuff - and then you need to train the trainer. As we've gone through - and highlighted those key stakeholders in those first chairs, remember, they've gone along - on this journey with you. You empower them to host their team's - role-based training because they're learning from - one of their own, way less challenging, way more championing, if you will, - and they seem to respond better to it. So those are just some training tips. Then we get into communication. Again, this is a massive change. If you don't tell anyone - that you are having an orchestra symphony, if you will, then A, - they're not going to show up, or B, they're going to see that - all of a sudden they have a symphony on their calendar for tomorrow and - it lasts 4 hours and nobody told them. So you need to make sure - that you are developing a very robust, - audience-based communication strategy that will include audience-based - communication templates. So my executives might need to have - a different message along this journey - then my future end users. I want to also develop - that communication cadence. So if I'm going to launch in June, - when am I starting my communication cadence? Everybody's business is different. We started really early, - and we had communicated even before we started our engagement.

Leaders, what do you get to do? You get to promote - the work that these people are doing. A lot of folks have asked me, How come your Workfront implementation - and your tools have been so successful? Because I had top-down support. I have a chief marketing officer - that believes in the power of operations and believes in what the power a tool like this can do - that's providing you with that visibility. And he then was a champion - to all of his peer groups as well. So we have had top-down support, - that is your job, your team, - your conductors, your first chairs. They can even develop an author, - to do communication for you and give it to you weekly, like "Beep, that's what you need - to send out to your peer group. Looks like it's from you. Looks great. It's just wonderful, - if I do say so myself." So your job is to promote that work.

Ta-da, you made it to opening night, - big orchestra. Everybody loves it. A little bit off-key sometimes. It's okay. It's a work in progress, but let's go back - and kind of look at what we did. So when we first launched, - we had about 300 end users. We now have over 400 paid end users, - and we have thousands of free reviewers who have the ability - to request our services and they have the ability - to review, comment and approve on all of the collateral - that is routed with them. We also route approximately 15,000 creative assets - that are routed and reviewed annually that go through the tool. If you'll remember, - we said that, you know, our tools were not nimbly adjustable. So it took me months to figure out - how to add a field. Well, Adobe Workfront is business - configurable. You do not have to know - how to be a developer to be able to configure this tool, - and that is the benefit. It sits in our tower, in our organization under the marketing umbrella.

We also talked - about how we didn't really have good schedule visibility, - right, or good scheduling tools. Well, Adobe Workfront - enabled us to have great scheduling. We know who is responsible for what - and when and they know it too. You then have visibility. If maybe something's late, - you can have an alert to that as well. So that provided us visibility. Visibility equals high accountability. Side note, - high accountability is sometimes where you can find your biggest challenge - in change management. When it is visible to the world, - everybody starts to feel a little like, "Oh, I thought they were always late," so just be aware of that and navigate - through that with your communication - and your training plan.

So where do we go from here? Well, lots of orchestras - these days are doing like fantastic collaborations. Bob Weir - just recently played with an orchestra. If we happen to have any Grateful Dead - fans in the audience.

But we also have, in our city, - they're doing what they call movie mashups. So there's like Harry Potter - is being played on the screen as a movie that you get to watch - and the orchestra is playing Harry Potter. Well, that's amazing, - you know what that does? They've got new activation, right? Maybe people have never wanted - to see the symphony before coming, and then they've also really engaged - in a unique way their existing audience members. Well, how does that relate - to an Adobe Workfront implementation? Well, because for us, - our mashup is our next. It's what's on the horizon. So we're going to be leveraging - the native integrations to be able to take Workfront - and integrate with AEM assets. We're going to be able - to go Creative Cloud into Workfront. We're actually going to be able to go - from Workfront into a few other tools. Your options are really limitless, but that is what is on our horizon - because we know we have new end users, we've got great activation, - we've got, additional key metrics that Chris was talking about to deliver - on a lot of that great digital journey. And we need to be able - to have our implementations to do that.

Okay, It did not happen overnight, y'all. So this was our progressive journey. We started in our crawl phase in 2019 where we were gathering - all of those requirements. We were identifying - what our toolset was going to be. We started mapping our process. We had all of that documented, - also in our crawl stage we launched our MVP in June of 2020. Why is that important? Because one week before our nation - shut down due to COVID was the only time that I had an in-person - meeting with LeapPoint. They flew out, and the next day - everybody was on lockdown. We were not a work from home business. My target to implement was June of 2020 - and we did it in June of 2020, figuring out how to navigate - work from home and navigate a pandemic. From there, we go into the walk phase. Well, what's great about that walk phase - is that we actually stood up the Marketing Workflow and Tools team. So I had a pocket of folks - who were doing this work along with managing campaigns, - but we had a long-term vision. We were very good at what we delivered, - and we were able to stand up a team that is just focused - on our marketing workflow and tools. We also, through that process, - launched an enhancement prioritization principles and process. So that has enabled us, I think last year we delivered - maybe 60 enhancements to our Workfront instance, building upon - that foundation that we set up. And then we started dabbling - and leveraging some fusion - and some other incremental automations. So I'll give you just a brief example - where you come in and you've submitted that request, - and now you need to take that request and convert it into a project applying a project template. - We're automating that. So the request comes in, it gets triaged - by the team who receives it to make sure it's all there. And then based on the questions - that were answered at intake, they click a button and it applies - the appropriate project template. When we get into our run phase, - we are projected to deliver over 90 enhancements just to - Workfront alone from my team this year. We're on task to resolve approximately 800+ help and support - tickets from our end user base. And then we have that tool integration - that I was speaking about. So it can be overwhelming to launch a new Workflow - platform or tool, right? You're touching a lot of people, - you're touching a lot of end users. While people might be working on - an Excel spreadsheet, and they're married - to that Excel spreadsheet, am I right? So I just suggest these key items: identify the workflow lead - in your cross-functional stakeholders, map out your process - and make sure you're all aligned to it, do not map that process in a silo. You're mapping it cross-functionally. You want to select - your tools and partners, and then you want to develop and implement your communication - and training strategies. And finally, you want to develop - a prioritized enhancement roadmap that will help you for that - long-range plan. That's how we leveraged Workfront, - it took us from a little one-man band to a great big orchestra, and I - highly recommend that you all do the same. All right, Niel, yep, yep. [APPLAUSE] [Niel Seideman] Okay. Hi, everyone. I'm Niel Seideman from JPMorgan Chase. I lead our digital channels organization, - and I'm here to talk to you about scaling experimentation - in the enterprise. I love what Chris and Robyn - were talking about, before we talk about tools, before we talk about MarTech, - before we talk about personalization, we have to talk about people - influencing people. Look at all the people in this room. It starts with a message. It starts with a story. Someone needs to move - from their current state, and that's someone who works next - to you, works with you, your leader, your stakeholder. They need to understand what you're doing - and be convinced that it's valuable. So a lot of what I'm going to talk about - today really dials into the how and a lot of the how - isn't about integrating technology, it's about bringing people together.

So let's get started, - by the end of this presentation. My goal is that you understand - what we mean by experimentation, it means a lot - of things to a lot of people. Why do we experiment? And then you can use us. And I see my great Chase team - sitting here. You can use us as your personal R&D lab. We've seen it all, - we've gotten all the pushback. We've been through all the pain, we've been through - the organizational inertia, the matrix kind of nonsense - that it takes to work at a big company. And so what I hope that you take from this - conversation is that you can accelerate because it took us a couple of years - to get us to where we are today. So I'll tell you some hopefully - entertaining war stories and challenges. But before we start, who am I? Why am I here talking to you - about experimentation? We have a fun thing at Chase - we call Humans of Chase. For those of you on social media, - you've seen Humans of New York. It's a way to get to know the people - of New York to bring you closer together. This is my Humans of Chase. So top left is my origin story. I'm the descendant of immigrants - from Eastern Europe. My family settled down in Philadelphia, - and you can see there in the top left, my great grandfather, Henry Seidemann, - had a really fun store that sold jewelry, musical instruments, - guns and sporting goods. So fun. A lot of good stories there. Bottom left, that's my crew, my wife Janna, - my two daughters, Sonia and Nina. We love to travel. We practice gratitude. Top right - is a little bit of my career journey, so I started many years ago working - in the media and entertainment field. I worked at HBO, MTV, - and a bunch of other fun sundry jobs made my way to Chase in 2008. I got a call in 2015, - went to Goldman Sachs, was one of the founding members of Marcus. I left them on top. Everything was fine when I left. I don't know what happened, - came back to Chase, got my career started in marketing, but now I'm a recovering performance - marketer and work in product. And then bottom right is some of the books - that I'm reading and work philosophy.

And then what do we do? Digital channels at Chase. So a lot of you are probably, - I hope, Chase customers. A lot of you have provided me a lot of good feedback about our digital experience - throughout this Summit. Please keep it coming. But essentially the Digital Channels - group at Chase is a quad comprised of product, technology, design, and analytics, and we're proud to deliver - best-in-class scaled digital experience in the mobile app and the website. And we do that actually via - a lot of platforms, including AEM, Target, and Analytics, and a lot of our - practitioners are here today.

Why do we do what we do? At JPMorgan Chase, we help customers make the most of their money - to make the most of their lives. We are the front door - and daily interaction that most of our customers - have with the firm. And so we spend a lot of our time making - sure that our experiences are performant. They're available, they're always on, - and our customers are safe. And then we get to interesting stuff - like personalization and experimentation.

Our OKRs are centered on four things: growth, satisfaction, engagement, - and modernization. And our one sentence - kind of reason for being is we want our customers - to choose to serve with us digitally. When our customers choose to serve with us - digitally, we want our most engaged customers to be our most satisfied. Okay, so now - we're kind of diving into experimentation. Why do we do it? And so those are not real customers. The woman in the pink jumpsuit, - maybe she is, don't know. But this is why we experiment. The reason we experiment is - we want to ensure that all the great work, you may have read - we have a $12 billion tech budget, all the great work, - all the great ideas that that quad that I mentioned comes up with, - that they express themselves and deliver on the impact - that was intended. What we know is from firms like Microsoft, Google, Amazon that are on the other end - of an experimentation and scale journey. They know when they run - 10,000 tests a year that a third lift and deliver - the impact that was intended, a third are flat, - and a third actually suppress. So the reason we actually got started - on our experimentation journey at Chase was we didn't know which third was which, we were celebrating delivery all day long. We had town halls talking about martyrs - who had stayed up all night and somehow delivered this - incredible experience for our customers. And we talked about how many customers we were serving and all the great software - and experience we were delivering, but we didn't know that it was delivering - the impact it intended. We tried a DRS, we did QA, - we did user research, and we'll and we'll talk about that - a little more later. But we didn't actually know. And so we've set out on a journey to know - and understand. Okay, - so just a couple of grounding basics. So what is experimentation? Controlled experimentation is the best way to understand - whether you're delivering the impact that was intended. So the causal relationship.

I talked about satisfaction, - engagement and growth. We have a ton of numbers - that we would never share publicly around that, - but everyone on our team and everyone on our quad knows - exactly what those numbers are and what we need to hit. And so we rally thousands of - technology, design, and product professionals - towards those goals. And so experimentation - helps us to understand when we ship all these thousands of features, - lines of code, whether they are delivering the impact that was intended. What is an experimentation? It's not like a separate thing. It's not an innovation lab - that you kind of break off with a lot of like cool people with asymmetrical haircuts - and like cool clothes and stuff. It's like core to - the product development lifecycle. It's what we do and how we ship product.

How does it work? This is like super basic. I stole Christmas clip art here. So on the left, randomized and controlled. So it's not -- So for every thousand customers coming in, we split them randomly and we de-risk by taking any kind of confounding factors, what channel do they come from, - time of day, do they have blue hair, etc.? Target randomizes our traffic for us, - it's controlled. Like think back to 10th grade - chemistry class. You need to isolate your variable - and control for it. So that you understand if that is truly - the thing that is delivering the impact, and it's measurable. And so we split the traffic 50/50, - and we measure the impact. I'll share more details later, - but these basics are super important as we go through the talk.

Okay, experimentation is also complementary and additive to other pieces - of the product development lifecycle. None of these pieces - can exist on their own, and in fact they're highly interactive. And when they are, when you do user research - and you come up with a great insight, then you want to test that insight - and make sure real customers actually either validate or challenge - what happened in a lab. Great news. Five or six times out of ten, - the thing that your researcher came out of the lab - with as the best possible idea was actually the thing - that customers gravitated to. But guess what? Half of the time it's not. And I'll show you an example of that. So these things work together. It's not an either/or. It's an and.

Okay, so this is the Chase mobile app. There's a little button there - that says Open an Account.

That's actually not just a button, that's like a magic button. Why is it magic? It's magic for our customers - because it is a gateway into a really important decision - in their financial lives. It's the way that they enter - into a journey to buy their first home, to invest for their retirement, - plan for their future, book a trip and many other life moments. So this is the beginning of a journey, - and it's important for our business because 30% of our customer acquisitions across the Chase consumer business actually come from this button, - plus another one on the mobile app. Super important. So we focus on that button - and the rest of the app so that we can understand whether we can - deliver the impact that we intended, which is to match customers to the right - solution for their financial lives and also underwrite them for a product - that will help our P&L.

So we set out, remember, those kind of three pieces of the product - development lifecycle. There's research, there's controlled - rollout and experimentation. We started with research, - and we talked to customers, and we said, What might you want that button to say - if you were starting your new financial relationship - with Chase? And they ranked them, a little MaxDiff qualitative study - came up with our top-ranked variant, Open a New Account, - and four others. And so now we had - a couple of great ingredients. We had an output from research, - We had a clear hypothesis. If we update the copy on the CTA - to focus on products, we'll see an increase - in account production. Now we're ready to experiment. So what did we do? First Test. Top variant. Remember this one? What did customers tell us? What was it? - Open a New Account. Great. Let's do it. Ran the test. What happened? Flat. Why? - For every 100 people that came in were the results the same? - No. There was this really interesting node of - customers that were suppressing results. Who were they? They were people who had opened - a new account in the last 90 days. Okay, great. That kind of makes sense. But you wouldn't know that from research. Rolled that one back. Did a little more research, - talked to customers, thought about - what might an alternative be. Let's play with the word "products." Okay. Minor change. Kind of interesting. Oh, no. Suppressed by 5%. It was -- You have a little bit of real estate, it was confusing customers. What's a product? - Is a product a new iPad? Is it a new car? What's a product? And so we recognized that we needed - a little more real estate, a little more vivid kind of detail in that module. And so we combined - kind of the greatest hits, the pieces that were working - from each of the various elements. And we came up with this placement that still exists today - that we call the PLU, or, I actually don't know what it is, - PLU or something, anyway, super cool. What's great about it? - A couple of things. When we talked to customers, - they actually didn't understand a credit card or an auto loan - or a home loan to be an account, that was a product for them, whereas an account was something like - a checking account or a savings account. So this placement has Open an Account, - Explore Products, and it's really specific - and unambiguous about the products. You're going to open - app productions up 13%, which at our scale, you saw, you know, - 8 billion sessions a year in the Chase mobile app, 50 million users, - 13% times NPV equals a lot of dollars. So great.

Okay, shifting gears a little bit, why is experimentation important? Who thinks it's important? I'm sure you recognize everybody - on this slide. What I like about kind of - what these leaders are saying is not that they believe - that innovation is important. Shipping shiny new projects - or products is important. It's really about driving impact - and improving decision making. So, Jeff Bezos, "Our success is a function - of how many experiments we do." There's another one over there, it's "how many experiments - we do per month, per week, per day." So we really rally around this - and think about we have an OKR around - volume of experiments. Mark Zuckerberg, - "One of the things I'm most proud of and I think what is the key to our success - is the testing framework we've built." And then the guy on the right, whereas this is not lifted, he's not like peeping into our business - all day. This is not in response - to anything that we've done. But I think it speaks to experimentation is actually resonant - with how the firm does business. It just wasn't localized - in our product function. So Jamie said, "Great - management is disciplined, rigorous. Facts, analysis, detail, repeat. You can never do enough... complex activity - requires hard work and not guessing. Test, test, test, and learn, learn, learn, accept failure - as a normal recurring outcome." Awesome.

So I loved what Robyn and Chris - were saying about kind of getting outside of your four walls - and learning from others. That can be a real accelerant. And so we got our hands - on an academic paper that was written by experimentation - leaders from Booking.com and Microsoft. These are two firms - that are on the right-hand side of this maturity curve, - and they actually did an academic study. They went into over 50 organizations - and they looked at what are the aspects - that are common across those organizations and what are they doing, - what's their playbook, what are they implementing - in order to scale? And so that was our cheat sheet. And we looked at that - and we said, we want to be on the right. We want to be up and to the right - and growing logarithmically. And so we started over at zero in 2018. It was me and a couple of people - in a room for 15 minutes, hired an experimentation expert - and then seated a great team. And I'm proud to say we're two years in, - and we're running hundreds of tests a year, and I'll show you how, and we have - a clear line of sight into thousands. And then some of the how, I see some of - you saying, "Pictures," are on the bottom. I can share, for anyone who's interested, - a link to the paper after. Okay. But the how actually, again, doesn't - start with a technology conversation. It doesn't start with an RFP. It doesn't start - with a finance business case for us. We, and I'm sure a lot of you - can empathize, in a large organization you need to distill things - and you need to distill things to stories that everyone can understand, - no matter where they're coming from. And those stories are best - told in a very visceral way. They get people. I like to get people agitated, - either excited or fearful. And so this is our story. Our story is basically this Jeep - kind of riding along this curvy road somewhere - that looks pretty awesome to visit. And what we're saying here is, you know, - and this is why I said the idea of experimentation, you saw Jamie's quote earlier, - vibrant at Chase. It's not like we don't use data, - not like risk management isn't a thing, not like we don't understand numbers. It just wasn't localized. And so to the left of the funnel - and kind of the demand side, we have an incredible paid media team. We have a great direct mail team all day - long, crunching numbers, doing tests, etc. Then there was this massive blind spot. That's the car - kind of driving around that curvy road. And then at the bottom of the funnel, - a certain number of people kind of spit out - the bottom of the funnel. And so lots of emojis, lots of excitement. What's happening there? - We don't know. All right. So first step is a strong story - that's easily understood. Second step is don't try and do this - with your whole organization all at once. You're going to collapse - under that weight, especially - if you haven't built the capability yet. And so we distilled our energy - into kind of the perfect intersection of a willing partner with high ROI - with high likelihood to succeed, that was the credit card - acquisitions team. And then we kind of just started along - the journey a little bit blindly, like we didn't have Adobe Target, - we didn't have people. We just said, The next step - is to start creating hypotheses, and we saw where that led us.

So this is where the story takes - a little bit of a turn.

Going back, don't get nauseous. The story takes a little bit of a turn. You are going to meet resistance. It's going to come in the form, I swear, - 95% certainty, in these four areas. One, I don't have enough capacity, - two, I'm already doing this, pre/post analysis. Three, and we talked about this earlier, - research already supported this, like roll this thing out, and four, - I don't have the tools. So I'm going to move quickly - through these.

One... Sorry I'm going to do - the Summit money slide. So technology, a couple of things. One, the Adobe suite of products makes it incredibly easy to lower the bar - for technology integration. It's a great stack from AEM. Our content management team - and our experimentation team are fully integrated and as much as - possible we try and make new launches a content launch decoupled from Dev. Second, the analytics - for Target integration makes it so we can have out-of-the-box - generic workspaces that spit out Analytics and Target dashboards where anyone across the organization - can read the results of experiments, and then data, analytics and insights, - great tools, and I'm really interested in - Adobe Product Analytics which should help this accelerate - even further. I'm going to fast forward through this - because this wonky, and I probably wouldn't - do a good job there. I want to leave you - with a couple of things.

I'm going to fast forward, - I got one minute. Chris is giving me the hi sign. Two. Great. Awesome. So I'll leave you with a couple of things. One, I hope that you can -- The biggest message here is - to get proximate and get started. Experimentation feels big. It feels a little weird. - It feels like something really smart, either quantitative people - or product and design people do. But the fact of the matter is, you know more than you think you do, - and the material is out there. Second, ideas are everywhere.

Delivery is everywhere. Anyone can deliver at scale. We talk about delivering with impact, - and that impact needs to be measurable, and experimentation is the way that - you can do that, and then just get started. Just start doing this. Start moving along the journey. You'll be surprised - at what you're able to accomplish. So with that, I'll turn it back over to Chris. Thank you. [APPLAUSE] Thank you. That was awesome. Well, I hope you all appreciate - the perspectives that we got from both Robyn and Niel and literally - what's involved in making these things happen. Let me leave you with three things - before we conclude. First and foremost is - when we talked about the very beginning, the lowest thing was aligned incentives - to get people to work together. Changing those incentives - will change behavior. The second thing from Robyn's - presentation, these are my takeaways, the importance of the first chair. Many of you in this room are conductors. It's to find the people that will support - your efforts, that can influence all the different business units - and teams. And then last but not least, you know, Niel covered this, - but you should do this. Celebrate not just the business impact, - but the things that we do, the changes that we make - for our customers, whomever they may be.

So with that, - I wanted to thank everyone for attending. I appreciated this really great turnout. I hope you enjoyed the session - and the different perspectives and I hope you enjoy the rest of Summit, - and I'll see you all later at the bash. Thanks again. [APPLAUSE] [MUSIC] -