[Music] [Dheepak Ramnathan] Hello. Good afternoon. Welcome, welcome.

In today's session, we're going to talk about data-driven customer experience, and also hear about Vanguard's journey towards learning out the data-driven personalization. Quick introduction, I'm Dheepak Ramnathan, I lead the Adobe Practice for Capgemini Financial Services in North America, and I have two great leaders joining me.

[Jake Bennet] I'm Jake Bennet, head of MarTech Product at Vanguard. [Theron Hofsetz] And I'm Theron Hofsetz, I'm head of MarTech Engineering at Vanguard.

Thank you, Jake and Theron.

A quick introduction about Capgemini. We are a global consulting firm. We do a lot of turnkey customer experience transformation, starting from design strategy all the way to agency services. We have been proud to be a partner of Vanguard the entire time during this journey.

And our Capgemini Research Institute releases a number of industry reports, which are highly sought after and widely read. I encourage you to take a look at it on our website or by scanning the QR code.

Shifting gears, I wantto talk quickly about the power of data.

Did you know that there was a humongous loss to the tune of $10 billion due to a single feature released by Apple to change the way data is collected and enacting personal privacy.

And there is in every day, there is almost 0.33 zettabytes of data generated around the world.

And 90% of the world's data was generated in the last two years. And with the advent of GenAI, this will quadruple in the next three to five years.

All this means is we are drowning in data, and there is a 80-20 rule of data science, which is currently seen in all the data teams, wherein 80% of the time is spent by the data teams in preparing data, cleaning it up, normalizing it, and only 20% of the time is spent on deriving insights and doing meaningful work.

So there are a number of complexities in using this data towards achieving better customer experience.

Notably, stitching the data between known and unknown data sources.

The issue of the data lakes having humongous amount of data, but still presenting an incomplete picture, data usage is being limited by the privacy and consent and regulations. Nearly 50% of the financial services form that we have pulled have inadequate data governance or data management practices.

And finally, the channel performance data itself is heavily siloed, resulting in very tough to derive journey actionable insights, making a marketer's life very tough.

So if we look beyond the tactics, the marketing MarTech teams are heavily bogged down by the tactical items, but the question in front of them to activate the customer experience is very well known. It starts with how do I know what my customer wants at any point in time? How do I deliver a better personalized experience for unknown visitors to my web channel? How do I measure the ROI for some of the marketing activities that we do? How do we stitch the data between the disparate data sources and how do we define an attribution model? To address all of this, Capgemini has a data-driven customer experience framework, which helps you get to the actual meaningful work of using the data for powering activation with 200 plus industry use cases, starting from marketing, all the way to sales, services and commerce. And there are a number of prebuilt models and connectors, which we help take the customers take advantage of, in getting quick time to value. With that, I invite Theron and Jake to share some of their insights in rolling this out for Vanguard. Thank, you. Thanks, Deepak. So let me give you guys a little context on Vanguard. So we're one of the largest asset managers on Earth. We manage $8.7 trillion. I've said that twice wrong today. Trillion dollars We have over 50 million clients, we have 20,000 crew members worldwide, and we have over 400 funds. We're the first largest fund producer in the world and the second largest ETFs. The company was founded by John Bogle back in 1975 with a principle of giving every investor the best chance for investment for success by treating them fairly and helping them achieve their goals. That's something that you'll hear as we talk about data-driven customer experience today in the way we're approaching use of data, our personalization journey and our activation journey.

We also, sorry, go back one slide, Jake. We also manage a large number of, beyond our mutual funds and ETFs, we offer full service brokerage, we have a number of other financial products including 401K, other investment vehicles like 529s or just other after-tax savings products.

So before we started this journey, what did our data-driven customer experience look like? Well, it was in a pretty sad state. We were very, very, very rudimentary, basic is probably generous at the level that we were at. So we had a set of legacy tools that were built or acquired back in the late 1990s or early 2000s. We weren't really doing personalization, we were actually building multiple web pages to deliver different experiences for different audiences, even though 90% of the content was the same.

Our stack was largely disparate, we had Salesforce, was probably one of the core anchors of our stack. Everything else was very on-prem legacy technology, required heavy IT involvement to do content deployments, personalization deployments. We had a number of risky, at least in the financial services industry, risky business practices that were being employed. We had a lot of on-prem technologies, where data had to round trip from the browser all the way into our data center and back out to do things like simple A/B testing.

So simple personalization was not effective. Most of the data that we had available was in data lakes. It wasn't easy to activate, and our data teams were trying to encourage our marketers to activate out of things like S3 buckets. So pretty sad state of affairs when Jake and I arrived about three years ago.

Yeah, and so, when I started at Vanguard, I actually moved my family across the country. I was in Seattle, moved to Pennsylvania, and I think Theron and I knew it was going to be a challenge going through this transformation. And, but I don't think it was until our feet hit the ground, we looked under the cover and realized, "Oh my gosh, what have we gotten ourselves into?" And as bad as that may have been, what actually kept us up at night were four trends that we saw going across the industry. The first was integration. And so we saw businesses increasingly focus on using data to connect their disparate MarTech products to create a more unified experience. Then there was the emergence of the Customer Data Platform, the CDP. And this is a tool that gave marketers the ability to take data from multiple sources, put that in a unified customer profile, give marketer self-service tools, and now they could do segmentation on their own for personalization. Now, in 2021, Gartner looked at their research and realized from their respondents the year prior, that 87% of companies had either implemented a CDP or were in process of doing so. And trend number three, for those that had gained much more maturity than us, they were now eyeing orchestration. So this is the ability to do multi-stage, multi-channel conditional journeys, to have an ongoing conversation with clients and using next best action or next best offer as a driver of personalization. And then, of course, we had the backdrop of the deprecation of the third-party cookie. So, needless to say, in those early days, Theron and I slept very little. Fortunately, we were so far behind that leadership realized we needed to make a seismic shift. It wasn't enough for us to just incrementally improve where we were. We needed to make a huge investment, a big focus in order to leapfrog others in the industry. So a business case was put together, it was presented and approved, and this gave us a three year roadmap to allow us to re-platform the systems that underpin all of the marketing across Vanguard worldwide. And this vision gave us a path for the first time, for an enterprise-wide marketing platform of leading technologies and capabilities. And in addition to those resources, another critical thing that we did at that time was restructure our organization, to create a single MarTech department that spanned the enterprise, charged with rationalizing our toolkit, retiring old products, and moving forward with the new stack and the new vision. So this is some of the products in our journey but not all of them. And so when we look back, there were seven capabilities, MarTech capabilities that we modernized. So these were old crusty legacy products, many of which ran on-prem in Vanguard data centers. And we moved those to modernized capabilities to the cloud, either in AWS or as SaaS products. And in addition to that, we rolled out 11 new capabilities. So things like Customer Data Platform, things like MobilePush. So now marketers had a much bigger toolkit at their disposal. But as important as this technology rollout was, that wasn't the real transformation. The real transformation was organizational. And there was a lot of work that we did to make sure that the enterprise was ready to accept the vision that we'd put forth. And I'll just give you a few examples. When we rolled out CDP, we had to work with our very large data organization across Vanguard. Now, here's the thing about traditional data engineers, and there might be many of you in the room, so I apologize if I fend. But, for many of them, the data lake is the solution to all problems. So they were very skeptical of a marketing technology organization bringing in a CDP because it looked a lot like a data lake. So we wrote a white paper, we talked about the gaps in capability that Vanguard has. We talked about where the rest of the industry was, the outcomes that we could achieve by having a modern tech stack. We put together blueprints and diagrams that talked about the pattern of how we take their data lakes, we use that to fuel CDP, we bring in digital data, we bring in third-party data, and now we put that in the hands of marketers. And once we have that paper, we had a lot of conversations, dozens and dozens over the course of many years. Now, the CDP is at the centerpiece of our stack. On the other hand, we had marketers who had long complained about IT being in the way of doing their work. However, it was a change for them as well, because now they had tools in their toolkit that gave themselves service that also mean that they needed to learn how to use them. So big organizational shifts need to upskill their teams and augment with staff, additional staff in some cases. All the while, we're on a three-year journey, we have funding for those three years, but guess what, that's a long time. So in order to maintain momentum, continue support, it was critical for us to show the incremental value that we provided along the way in terms that the business understood. And then finally, we are a highly regulated industry, so we can't do any of this without talking to our risk and compliance and security partners. And we worked with them very early on. We said, "Hey, we're going to do this thing that's going to sound really scary. We're going to take all this customer data, we're going to put it all in one place that's going to be in a third party, and then we're going to give it to the hands of marketers." Now, needless to say, there were some eyebrows raised. But, because we brought them into the conversation, because we walked them through the process, now they are supporters of the program.

Stage two. Where did we end up after we completed the roadmap? So we ended up in a place where we consider ourselves advanced. What does that mean? It didn't happen overnight. And you heard Jake talk a lot about CDP. So as the technology part of our dual, like you might think, "Hey, I'm the guy that says go buy a bunch of tools." The reality is you have to learn and effectively use those tools in combination to deliver that customer experience, right? A lot of you in the audience might even be thinking, "Hey, data-driven customer experience might equal web." Well, what about other channels, right? So along the way, we added other tools like Marketo. We activated marketing data on our websites through things like web data layers, cookies, first-party consent, and brought together personalization that we didn't have the capability of before. We replatformed the web using Adobe Experience Manager. So they gave our teams faster time to market and much more flexibility with integrations like Adobe Launch and Target. But then we did start to step into that unified customer profile, and we brought CDP into the forefront. So you see that we became more of an Adobe shop. A couple of those tools, Workfront and Marketo, we actually had on the roadmap and we were planning to implement before Adobe acquired them. So there's probably a little more Adobe here than we may be anticipated. You still see Salesforce Marketing Cloud up there. We advanced capabilities like delivering in-app messages or push notifications using that platform. We've connected our CDP to Salesforce to trigger journeys. We have much more comprehensive and impactful journeys with decisioning logic that are driven out of Salesforce today. So it wasn't all just buy tools and implement them. As Jake talked about, we had to do a lot of organizational change to drive this customer-driven data experience.

So one of the early campaigns where we saw success was a multi-channel campaign in our personal investor business. And the business found, So Vanguard's been in operations for about 50 years. We have a lot of information and data on how people save money and how they think of investing in retirement or investing for goals. So one of the pieces of data that we had, again, through great analytics and tools like a data lake. We find that people who change jobs are very receptive to wanting to learn more about how to make impacts in their financial future. So things like, I might have had an income increase, should I allocate more money to my 401K? Should I put more money into my 529 plan? What if I'm thinking about paying down my mortgage? So it also gave us the opportunity to talk to more of our clients about our advice offerings. So that we could, again, Vanguard is very mission-driven. We want to help people better their financial outcome. We're not trying to sell you the next pair of sneakers or, you know, soda water. We're trying to help people achieve their financial goals. So a lot of our personalization ties to that. And what you see on the screen is really our first scalable multi-channel personalization campaign. So this was for those people who were job changers, and we did this in social media, email, and then on the website.

And then, things started to really get cool once we started having those technologies together. And what you're looking at here is the balances page, account balances page. So you can, and by the way, this is not real data, this is Jane Doe. This is her account that we use to test and show. But you can imagine if you're a Vanguard client, you're logging in occasionally, maybe once a month, maybe once a quarter. You want to see how are my ETFs performing, how are my index funds performing? What does my account balance look like? Now, understand that that becomes a critical moment for us to create a touch with that customer because they may not come back for a while. And so what you're looking at here is a banner. And what this banner is saying is, "Hey, listen, we know you specifically, Jane, you had 1,801 losses in your portfolio that we could actually use to reduce your taxes. We can do something fancy called tax loss harvesting. If you work with one of our advisors, either with a relationship over the phone or a digital-only robo-Advisor, we can help you implement tax saving based on the information that we know about you. And the reason why this is so powerful is because now, what we we looking at before, which was exciting for us, who had not been able to do personalization at scale, were broad-based based, more targeted segments. Now we're having more of a individual conversation with our clients And so we saw tremendous success with this. This was a 20% plus increase in conversion when based on a control. And so a lot of our strategy right now is to say, "Okay, this is great. How do we do more? And then how do we do that in a more scalable way? And that's not the only outcome we received. So at the end of the three-year roadmap, we took stock of the outcomes that we'd achieved along the way, and there were some really good ones. The first really big one was, moving from a legacy content management tool to AEM. We were able to see a six to tenfold improvement in speed for marketers. And you can imagine back in the day, even to preview a page before you hit Publish, we had to deploy it, take a look at that. Right now there are point-and-click tools, pre-built workflows, People could do that in a snap. We also saw increased engagement. And so we track engagement through a quality engagement score, And we've seen engagement scores increase 20, even 50% after rolling out the MarTech roadmap. And even simple things like AdTech, adding a pixel or a tag for a new ad partner on the site, it took three months before. now with Adobe Launch, Adobe Tags, It's a matter of minutes in order to do that. But the cost savings were tremendous. And the cost savings that we were able to calculate, I would say, are a very small fraction of the overall efficiency. But they were the ones that were easy to measure. We're saving 4.5 million on an annualized basis simply because various teams had to spend money on IT, to do things like pull lists or deploy sites. Now they didn't have to do that. Self-Service tools eliminated that. So these teams could now repurpose that budget for more important high-value activities. And conversion, we saw big impacts in conversion. So the Adobe Stack working in conjunction allowed us to see 20, even as much as a 200% increase in conversion. So, huge numbers. Very big for a company that didn't have a super mature stack And just simply increasing marketing speed. You know, we had marketing teams that would do, you know, what you might consider fairly rudimentary customer journeys, It would take months, 10 teams to put together one simple campaign. Now using Marketo, combining that with CDP, combining that with Workfront, people are able to take that from months or weeks to hours. So huge game-changing changes for the organization. But that's where we are. That's not where we're ending, though. We have an even bigger vision for the future.

Thanks, Jake. So one of the tricks at Vanguard was trying to convince our marketing partners who sponsored the three-year roadmap that the work wasn't done just because we'd finished the three years.

They still want to do micro-segmentation, they want to do one-to-one personalization. That required additional investments in our stack. Jake mentioned Marketo. We actually brought Marketo in because two of our B2B businesses said, "We see this as so impactful that we need to do it. And they were going to fund it outside of the MarTech budget,. And then when connected to CDP, as Jake mentioned a minute ago, the previous year, they took 10 teams three months to launch a single integrated campaign. In the back half of last year, once they activated data out of CDP and used Marketo for the channel engagement, they were able to do 25 campaigns. They identified 1000 prospect advisors that they'd never had a relationship previously, because they were getting much more segmented communication going out to those advisors in a very personalized and specific way. So we weren't batch and blast anymore, but we're still heavy, big segments. Where do we want to go? We want to go to this microsegmentation concept. We're bringing in additional tools. So you've heard a lot today about Adobe Journey Optimizer, Customer Journey Analytics being built on top of AEP. I think three years ago, Jake and I had other tools in mind when we were thinking about Journey orchestration or Journey Analytics. But last year after seeing what Adobe had done with AEP, so it was my first Summit in five years, and if you saw what they talked about in '18, coming back in 2023 and seeing what they built, it was pretty exciting, right? You started to see, oh, there's this actual data platform that brings unified profiles of your client for both B2C and B2B marketers together. so that I can use that data in a consistent way, I can do real-time activation in many channels, and I can get direct feedback back into AEP. I don't have to pipe the data in from a different marketing platform. So I think a couple of those items that we saw on a roadmap quickly became Adobe-centric. But you also see things like identity resolution. We had a vendor in mind for that. We're very privacy-centric company. We pulled back last year and said we were going to rethink that. We don't know that putting our client data in the cloud for a smaller vendor was the right choice. So we're still rethinking our identity resolution strategy. Of course, generative AI changed the landscape a year and a half ago.

So now everybody's talking about, how do I get essentially a one-to-one customer experience through a prompt? So generative AI and some of the advancements you saw today, like announcements that you saw with Adobe, are keys to our future for this micro-segmentation-based personalization as well. But there is a dark side to all this technology, and that is adoption. And as we add more and more products to the MarTech stack, it is harder and harder for marketers to use them all. And Gartner tracks a MarTech stack utilization. They've been tracking it for years. And what you see is, as new MarTech products are introduced to the space, we see a steady decline in company's ability to utilize that. And this is something that Theron and I knew was going to happen if we weren't careful and methodical about how we rolled out MarTech. And I certainly saw this as a consultant. Before I came to Vanguard, I ran a MarTech practice, and I had clients often come to me, because they purchased all the things, they didn't really have a vision for how to put them together, and they needed help. And I remember one CMO worked at a tech company, came to me and said, "Hey, listen, Jake, we just spent a huge amount of money with Adobe. We have this Ferrari sitting in the garage. We can't get it out of first gear." So we didn't want to do that. So Vanguard, as the industry has been declining in utilization and adoption, we've actually been increasing. And here's how we do it. We have a systematic way of measuring, quantifying, and driving adoption. And so what you're looking at here is an adoption dashboard. And you can see we have all of the products in the MarTech stack. And for each product, we break it down into a set of capabilities. And for each capability, we look at crawl, walk, run. we call it foundational, optimizing, investing class. So this gives us a framework to measure adoption or utilization of the MarTech stack. But we don't stop there. Now, we work with our divisional partners, our regional partners, our marketing teams, and together we self-score where they are from a utilization perspective. And this is great. This is the next step, right? We can say, "Hey, here's where we are today." But the thing that really drives utilization on a go-forward basis is sitting down and saying, and this is part of our operating model on yearly basis, sitting down with those same marketing teams and saying, "What are your strategic objectives for the year? What are the OKRs that you're trying to hit?" And then we can look together to say, "Hey, here's the technologies and products that will enable that." We can then say which specific capabilities in each of those products will help you drive those OKRs? Then we put the data in, that gives us a target, as, so now we can look at not only where we are today, but also forecast where we're going tomorrow. And in addition to that, as we have now enthusiastic marketers that are saying, "Great, I've got a target in terms of where I need to be to hit my OKRs, but how do I put together a cross-channel journey nudge campaign that takes somebody out of it, once they've achieved that action. How do I use progressive profiling to do anonymous personalization as people continue to use the site? And so in order to help them, we also created a series of playbooks. And so these playbooks give them tactics and techniques that they can use to increase their marketing, and ultimately have a better data-driven CX experience.

So with that, if you only take away three things from this session, I would suggest it is these three. So number one, establish a technical blueprint early. You saw Theron and the architectural blueprints that his team put together. These were things that we had put together early on when we first started at Vanguard. And it required persistence in order to keep staying true to that vision. And as he mentioned, the technology has changed. So we had the right vision, We thought we knew who those technology partners were going to be, and those changed as the technology involved. So, that's the first thing. The second is build a co-relation. So a lot of times, when we're talking to marketing leaders and MarTech leaders, they're one little piece of the puzzle. and MarTech technology is kind of distributed across the organization. So you may not own it all, but you absolutely have the ability to work with other teams, align on goals, come up with common architecture, and make sure that's not only your external partners, you've got internal teams that are going to take you there as well. We are very lucky, our employees came along with this journey. They've embraced the technology and now they own it and are experts in it. So build a coalition externally and make sure that your own employees are part of that coalition. And then the last piece, make sure you're focusing on business outcomes, always. We, in MarTech, we speak this weird dialect, right? We're using technology in terms that people don't understand even within IT, even in some cases, within marketing. So make sure that you are quantifying your outcomes, you're letting your stakeholders know you're speaking in terms that the business understands, and make sure you're using those outcomes or potential outcomes to ruthlessly prioritize what you focus on.

And that's the end of our talk portion of the day. You should probably sit down for the questions. All right, all right. Thank you, thank you.

The first question again comes from me.

Theron, as diverse as a business as Vanguard is, from institutional investors to individuals, can you talk about, how you approach CDP for B2B and B2B2C use cases? Sure. So we have three large US-based businesses. Two of them are B2B, one of them is business to consumer. We also have three regional offerings, which have a mix of B2B and B2C. So we weren't going to be able to do this without doing both. So a large part of that is we now use both the B2B side of the CDP as well as the B2C side. We work with our data platforms teams to help bring the data together that we need to build those unified profiles, ingest that into CDP, and then in some cases, like our Australia partner are using web SDK to bring that data in from the digital experience in real-time, and see personalization working across multiple devices. So we've had a thoughtful approach. One of the other things that we do that's probably a little unique, Why are there two of us up here talking? Jake's organization is a product organization within MarTech that is intentionally built to prioritize capabilities that are needed for all of our line of business partners. So his team has the hard job of ruthlessly prioritizing which capabilities we build so that we can meet the most impact for our business. My team's job is easy, right? They're gonna laugh at me later. But they get to enact the vision, right? So we get to be the ones that tinker with the tools and help deliver those experiences for our clients. Thank you. Jake, Theron touched upon the identity resolution. That's on top of the mind for anybody rolling out CDP with the advent of, or the demise of third-party cookie. Can you talk a little bit about what your strategy is for identity resolution? Yeah. So identity resolution was a capability that we had early on in the MarTech stack. And we know that our ability to truly do one-to-one personalized messaging required the ability to stitch together what we knew about our own clients and customers across different divisions to take a look at what we know about semi-known users, what we know about people from third-party data. So, as much as we love Adobe Realtime CDP as a centerpiece of our data-driven customer experience, it doesn't, in all honesty, have the best identity resolution possible. So for us right now, and Theron mentioned that, we were going down one path, we're now going down another. But we believe, especially now, that third-party cookie is in the process of being deprecated, that in order to have a fully functioning MarTech, modern MarTech stack, it's critical to have really, should be probabilistic, ideally, machine learning-driven identity resolution that enables you to stitch together known data, semi-known data like device data, and then third-party data to have a really complete profile of our customers.

I'll be looking forward to it in 2025, hopefully, or '24.

We'll see. Hopefully '24, but maybe '25. Good question.

Theron, AIML GenAI is in everybody's mind, or it's saturating the airwaves. Adobe themselves in AEP have launched a number of AI modules like Customer AI or Attribution AI, and there are a ton of features even Salesforce is releasing. Can you talk about your AIML approach for data-driven marketing? Yeah, so I think in MarTech have kind of a combination type of approach.

Obviously many of the tools we have, are SaaS-based platforms, so we're leveraging capabilities that are built in natively. Sensei's been around for a long time. Einstein's been around for a long time in the Salesforce side, but GenAI kind of changed all of that about a year ago, right? The ability for anybody to interact with AI through a prompt and through a very simple question and response really changed the way that we made AI available to the individual. So we're still leveraging a large portion of platform capabilities. We've rolled out a really comprehensive search solution with one of our partners. We're looking at things like Firefly more extensively. We're looking at simple tools like writer for content generation. But then if you layer into that, I think we're also looking at some of the advances you saw today in terms of audience generation, segmentation. I mean, the journey creation one was really, really interesting to me. And then I think a big one for our industry, financial services is highly regulated and what we say, we have to back up. We can't make a claim that we're going to make you wealthy and then not fulfill that by just kind of promoting a product. So a lot of our requirements are heavily compliance-oriented in terms of how do we document what we told people, how do we capture that? How do we make sure that it's true and correct? So I think there's a huge opportunity for us in the financial services industry to look at AI capabilities in GenAI to automate and speed up that compliance review of content. And you heard today, like five times the amount of content being produced every year. Well, we can't scale the humans fast enough in our compliance departments to solve that. So we need to look at AI and other technologies. Thank you. You talked about compliance and regulation.

There was a very interesting use case, you talked about tax law harvesting, Jake.

The amount of data and the amount of sensitive data it needs. For example, the portfolio data for you to make a very advanced personalization use case like that. How do you make these kind of data available at the hands of marketers without compromising the privacy and security needs of Vanguard? Yeah, it's a good question and one that our privacy, security, and risk people wanted to know the answer to as well. And I think, our data strategy, especially when it comes to AI and ML and one-to-one personalization, is, we don't necessarily want to put all of the source data into something like a customer data platform. In many cases, we want to actually build our own machine learning models. And part of that is, because we want to own that data. We want that in our data systems. What we want is the output of that for the end user. So, part of the reason that we're able to do that type of one-to-one personalization is, as we are very limited in what data goes in the CDP. And we can be much more, sort of, greedy about the data that we use in our own data lakes. But we go through a very rigorous process. Any data point that we're putting in the CDP, we need to go through regulation or go through compliance. Even just showing something like we showed up there, we need to have all that content go through compliance. And so a big part of what we're looking at now, and I think, as Theron mentioned, some of the innovations that we saw today is, how do you scale that content supply chain? That's a big thing for us. But our strategy right now is to put as little in CDP put much of that in our backend systems that our data engineers have more control over.

Thank you. Thank you. Jake, one bonus question before we open to the audience. The real-time in the Real-Time CDP. Can you talk about any of the plans for really doing a real-time marketing use cases that you're thinking of in Vanguard? I think for us, and it's interesting, my mind has changed a lot since I went from the consulting side to the in-house. And when I was a consultant, I would ask this question to my clients because, they all said, "I need real time, I need real time." And then you would actually ask them and you're like, well, you just want to do an email campaign based on a product launch. I'm like that doesn't need to be real-time. I think the real power in real-time personalization is same page, same session personalization. And so the problem is that, if I am batching things up on a 24-hour basis, that I'm missing a huge opportunity, because I might see someone and I might understand who they are at some point in their journey on that page. But I want to serve up the second they touch the page in 100 milliseconds or less. I want to make sure that they've got personalization, and if they take action on something, I don't want to nudge them, I want to move on to something else that's going to be more interesting. And I wanna do it in that moment, because, in the example, for example, you know what I showed, they may not come back for another quarter. So I need to make sure that we're serving up real-time recommendations. And for us, I think those are really critical real-time use cases. Thus the reason why we're with Real-Time CDP as opposed to maybe some of the other competitors out there. Excellent, thank you. We now open up to any questions from the audience.

So the ground rules are unfortunately have to come up to the front, speak to this microphone. Not to make it... Or you can shout and we'll try to repeat your question.

But if you would like to, please come up and ask. And it helps get your steps in for the day.

Hi thanks so much for the session. How are you guys measuring success from your campaigns, from both an attribution perspective, but also like a revenue in the B2B space? That's a great question. And I would say for us, we have two B2B businesses. So one is working directly with advisors. So you may not invest in Vanguard funds directly. You might have a financial advisor that you work with. So we want to make sure that Vanguard is the advisor to those advisors. So that's one scenario. Our other B2B business is really around many retirement plans, so 401Ks, running those for companies. So it's a really good question in terms of measuring success. I think in many cases, we can, as Theron mentioned, look at, there are advisors that we had not been in touch with, that we hadn't contacted, now we are, so that's one. We're looking at things like engagement scores as number two. It's a little harder in our business though, to say, "Hey, this advisor made a recommendation for this fund and put it in their client's portfolio, and therefore tracking that end to end. So, some of that information we can look at, a lot of it we can't. But I think for us, what's important is, we don't wanna be paralyzed by not being able to get down to the penny in terms of what those impacts are. But I think to your point, we want to get as far down as we can. And so we use the metrics that are available for us. And for our 401K institutional business, these are big deals, they are very sales-driven. Interestingly, we certainly have better information in terms of how many assets under management that influence, but we'll never know the mind of that buying group, if it was specifically because of that. But we do track for those businesses, at least the data that we produce, and marketing's contribution to that. That's a really good question. Yeah, I mean, just as a side follow up, you have a really interesting question. and the way Vanguard approaches this is very unique, We're not trying to get traditional conversion of units of sales or conversion dollars. So in some ways, we look at customer lifetime value, especially in our B2C space. Like, how long have I had a relationship with a client? How high has their asset base grown? Did they diversify into multiple financial products with us, so that they're meeting college savings needs, retirement needs, their personal, like after-tax savings or investing, right? So some of it's kind of like a lifetime value. The other one that we, this is very unique to Vanguard, but we have been talking about this a lot more, is we want to generate alpha, which is outcome for our client. We want to see their investment base grow. Part of the way we do that is getting them to take the advice we're giving them, stop leaving money in cash, put it in in your funds, right? Part of it is also helping them realize when they need to make decisions about what, how should they mix their portfolio? How do I think about the change that I'm going through as I'm getting closer to retirement? You know, things like that. So we have a goal for, I think it's 2025 or 2026, I don't remember, but to create $5 billion in client value. What does that mean? It doesn't mean anything to Vanguard from a bottom-line perspective. It means our clients have $5 billion more in assets in their accounts than they had today. So very different sort of metrics that way. I mean, we still have some of the traditional ones, like Jake mentioned, qualified lead is is a big thing. Qualified marketing touchpoint and sales touchpoints is a big thing in our B2B side.

Other questions? Come on up. I have a bad knee so...

So in terms of adoption, so that you had, that kind of inverse trend and you did talk to it a little bit in your presentation, but can you double-click on something that you've seen that may have been an aha with the marketing group in terms of their adoption of some of this tech that you've... Yeah, I mean, the big aha unlock for us was, and this may seem simple, but when we first rolled it out, we flubbed it. And I'll tell you how. I remember sitting down with one of our marketing teams and we had this great idea, and we're like, we're gonna measure adoption, it's gonna be the best thing ever. And they're like,"Wait a second, you're giving me a scorecard?" This is going to my boss. And needless to say, there was some skepticism. So honestly, for us, the big "aha" moment was in year two when we really closely aligned the adoption with those OKRs.

And the reason why, and it had been socialized, and I remember talking to one of our marketing partners, and this is when I knew it stuck. They were talking about their vision and they were actually presenting in their vision, for the website, for this specific division. If we do this thing, our adoption will increase by 8%. If we do this thing, our adoption will increase by 5%. So the real "aha" moment is just getting on board with the marketers themselves, so they are driving that adoption. But, on a product by product basis, it just really differs across the lines of business. Each one of them had specific areas they wanted to focus on and that made them pretty aggressive at hitting those goals. So the other thing I'll add, and this is all Jake's organization that built this and then got it implemented. But, it's a Trojan horse, right? So adoption is a metric that is about enabling somebody to use the technology, which then helps them achieve their business outcome. But the Trojan horse part is, it also shows them where they need to be upskilling their crew, hiring new people, how they're going to change their business process. So it became this change management catalyst for us, that instead of us fighting with them to try to get them to send people to a Summit, or attend an Adobe class or whatever it had to be, they were seeing, "Oh, to unlock this potential to generate this OKR or meet this OKR for our business, I need to build people on my team that have these skills, right? We need to build teams that support this. We need to understand the technology better. And so they became much more empowered to do that change management on their own behalf instead of us just having to drag them to that. And then the last thing that it did, it really unlocked this potential around value creation. So again, when we recognized, not everybody needs to adopt Target at the same pace. Like one of our B2B business was like, "We're not ready for Target. We don't need personalization on the website. It's not important." We're going after Marketo and CDP this year, and we think that's going to unlock X, and it did. And by unlocking X, so this is the team that spent 10 teams, three months, to launch one campaign in 2022. In the back half of 2023, through CDP and Marketo, they launched 25 campaigns that were highly data driven and personalized to their audiences. And it generated 1000 engagements with advisors that they didn't know. So all of the sudden, they're seeing massive amounts of efficiency, which is cost savings to the organization that we get to take credit for. And it was simply because we had a framework for them to think about, "How do I get better at using the tools that we have?" I just want to ask from your perspective, what was the biggest mentality shift you had to do? You know, I think for me, personally, the transformation was very marketing sponsored. And I think one of our big misses, especially in year one, it kind of smacked this in the forehead in year two, was we weren't engaging our IT peers enough. And a lot of our IT peers had gone from some bad legacy technologies to very empowered build things in AWS. And our data partner is the same thing. So we had a lot of build mentality in the company, and then you get these two yahoos coming in and say, "No, no, no, let's rent software, right? Let's do SaaS." And people are like, "Why would we do that when we can go build stuff in Amazon, right?" So we didn't engage our IT partners as early as we should have, and they weren't part of the decision-making process to do this either. So we've spent probably the better part of the last year and a half really deepening those relationships and helping them understand how to unlock the power of these tools and how it benefits them as well, right? We're not trying to say, you need less of these kinds of people or more of that. We're trying to say, these are the outcomes we can achieve if we've worked together. And so I think that outcome orientation that Jake talked about in the takeaway has really helped that conversation because now it's not us and them, it's, we collectively 2are working with our business to solve problems and deliver outcomes.

Thanks, Theron and Jake, that was a great session.

Any more questions? Well, thank you guys all for coming. Oh, one more question. Come on up.

So, did you consider any other than the Adobe tech suite? So we did, so a lot of... Oh, sorry, to repeat the question. When we were looking at where we were headed, did we consider things other than the Adobe Tech suite? And yeah, most of our, some of our original investment ideas were outside of the Adobe stack. Jake had worked with Amplitude, for example, which is a journey analytics tool and was a big fan of it. I'd worked with some things like Thunderhead and Kitewheel in the past which are journey orchestration tools. So some of our pivot in the last couple years has been because we saw the built-in integration that AEP and AJO, CJA, and Real-Time CDP provide. So one of the things I always think about as like the head of engineering is, it's expensive to integrate tools, right? There's a lot of overhead, you create a lot of technical debt, you create a lot of ownership and maintenance, that sometimes, isn't worth it. So if I have to create a team that costs $1 million or $1.5 million a year to maintain an integration between two platforms, or I could buy a solution and get 80% of the way to what we need, and have it natively integrated, I'm probably going to go that route. So we definitely pivoted a couple of the, Marketo and Workfront, they were on our map, but they were not tools that were owned by Adobe at the time. And so, then they became kind of, by happenstance, they became part of the Adobe Stack. We're still big Salesforce users, we do many of our journeys in Salesforce. We activate from CDP into Salesforce pretty regularly. We just renewed with them for another multiple year. So, there are big parts of our stack, that were not going to be Adobe and we never really set out to be purely an Adobe shop. We were definitely a best-of-breed when he and I arrived three years ago.

Anybody else? I'm just going to climb instead of climbing all over everyone.

You talked a lot about the kind of sales aspect of this, from the business partners, the influence aspects, kind of the fact that there's people in politics behind every box and why in a marketing tech stack.

You kind of hinted on this a lot, but are there any personas that you haven't really mentioned yet, or like tactics that you use to really break the ice or break through? So the question is really beyond some of the stakeholder groups that we talked about before, or there other stakeholder groups or personas that we needed to collaborate with in order to kind of break through those organizational barriers? Or even the ones that you did mention "aha" moments.

Yeah, I mean, there were, honestly, it was probably less about the one "aha" moment and more about the ongoing day-to-day, relentless pursuit of finding anybody that had an opinion about the MarTech stack. And they were all over the woodwork. So it was, it was probably less about that eureka moment, but there's risk and then there's third-party risk, and then there's Firefly's ad compliance and other compliance. There's different engineering teams even within... One of our largest divisions has 200 different pods that are touching the user experience. So I would say, as much as I would love to say what's that "aha" moment, it's about just relentless focus, never tiring of evangelizing and talking to anybody who has an opinion, whether they're aligned with you or not. And if they're not aligned with you, at least trying to get crisp on the points of difference and then being relentless of beating them down until they agree with you. Yeah, I mean, the one thing I would add is I think we undervalue the relationship we have with some of our partners.

Probably like many of you guys that are large companies, Vanguard uses many different implementation partners. And I would say sometimes that gets in our own way, right? We might have MarTech doing something with one partner and a line of business doing something with another, and that creates unnecessary friction. So I think you wat to look at your partner relationships and use them strategically. It's helpful to understand like some of our line of business partners are very much more aligned to some partners than others. It's good to understand those relationships so that you can capitalize on them, right, Instead of generating friction. So, I think that's something. I think the other thing, Capt did a really good presentation when we were here last year, and it totally validated our thinking on CDP. I mean, Jake brought a lot of, I came from more of the web and digital world, Jake came from more of the data world, right? But it validated a lot of the thinking we had around CDP. They basically laid it out and said, there are three core things you need in a CDP. You need an activation layer, which is what Adobe does for us. You need an identity resolution layer, which is what we were looking at with a different vendor. And you need a data collection or ingestion tool. So whether it was mParticle or Telium or stuff like that.

So there are many different layers to the way you think about using and collecting and generating data. And we knew about most of that. Two out of three of those turned out to be Adobe for us, where we're using web SDK. Again, because it's more effective for us to integrate that data real-time than it is to try to go bring Telium into our environment. But, again, use your partners. Don't just think they're here to try to sell you services. They do have, many of these large integration partners have great strategic consulting, data-driven information, like Dheepak shared at the beginning, hundreds of templates and use cases to like stand things up quickly. So leverage that is definitely an "aha" for me that I think, probably didn't see before.

Yes, sir.

Well my question is about, you talked about Vanguard, obviously you talked about scale, talked about influencing over 50 teams to adapt to this new strategy.

That also means failing comes in many different teams...

many different teams offering content in AEM.

And with that, probably comes some governance model to keep that data sanitized in CDP.

And you talked about in your, one of your answers that you allow very little data to go into CDP. So you have some governance product in place. Can you talk a little bit about how you balance having that governance, but at the same time noT affecting type market for you? Yeah, I mean for CDP specifically, I mean the good news is we have hundreds of teams using web technology. We have a limited few using data tech. I mean, they're big teams, but we have less of them to work with. So I would say that makes our lives a little bit easier. But one of the techniques that we used in order to both enable scale with self-service segmentation, but also quality. So number one, working with our divisional data teams to scrub in, in many cases, annoyingly. So check data quality going into CDP, so it's solid. That's number one. Number two is they build basic building blocks that are in essence segments that we can use for broad-based segments that are commonly used. And then marketers can go in and use those. So it requires a lot less hands in the cookie jar in terms of putting data in, but gives them a lot of power when they take that data out. And so as a result of that, we haven't necessarily had the limitation of either too many people touching it and causing trouble. But we have been able to see a lot of that scale in terms of self-service giving marketers access to that data.

Yeah, I might say more generically, 'cause you touched on many different parts of our tech stack.

Governance is a scary word to many people. Most people hear a giant stop, right? Nothing moves fast anymore. We have to get in committees and talk and think things through. And Vanguard certainly has its share of that.

I wouldn't say we've been successful in many of the ways, right? I came in with a model of an AEM. 70% of the components needed to be generally available to all web teams, 20% by line of business and 10% to regional. Nobody at Vanguard wants to follow that, right? Which is fine, right? So we've gone down a different path. We have a little bit of components sprawl, we're gonna come back to a reconciliation at some point because it gets expensive to maintain that. So I think there has to be a give-and-take in our approach. We followed more of a community of practice type of a governance model instead of a center of excellence or even a formal governance. Again, we are enablers of technology and then our line of business partners are the users or the builders on that technology. So we try to guide, but we have a very small voice and a small vote in any of those governance discussions. But they are things that come up from time to time, right? And we've taken more, maybe a bit more of an open-source software mentality of like a contributor model is better. More people contributing, helps the organization generally. But again, it gets into very specific nuances around like, are you talking segmentation, governance around who gets to create segments in the CDP? Are you talking about Web SDK and who gets to define the XDM model? Are you talking about AEM and template and component governance? Are you talking about Target, and do I allow people to just do freeform JavaScript on the site or do I force them to use something like an experience fragment? So there's a lot of layers to sort of the governance thing, but I think we've tried to do much more community of practice and then also a lot of office hours to help people come and ask questions and collaborate with each other, not just us, right? We're not the only experts in Target. We now have five teams that use Target in each of the lines of business. And they share ideas with each other much more effectively than we can try to broker that. So I think that's been a kind of a more general way that we've thought of governance.

All right, I think we're at time. Thank you all for attending.