ASU+GSV Summit: Our Increasing Expectations of Online Learning

Speakers:

• Josh Francia: Chief Growth Officer, Blueshift
• Kenny Kim: Chief Marketing Officer, Udacity

Welcome and Introduction

Josh Francia: All right, welcome everybody. We're really excited to talk to you today. My name is Josh Francia, I'm the Chief Growth Officer here at Blueshift. Blueshift is the world's leading Smart Hub customer data platform. Today I'm joined by Kenny Kim, Chief Marketing Officer of Udacity. So Kenny, welcome. Really excited to talk to you today.

Kenny Kim: Thanks. It's a pleasure to be here. It's been amazing, and really thank you for the time.

Josh Francia: Absolutely. Our main topic today is the increasing expectations of online learning consumers. Kenny, no one better to answer that question than you, as you are at the forefront of this. Udacity is a premier, leading brand of online learning globally, across many different countries and demographics.

So, talk to us a little bit about how students' expectations have changed for online learning during these last 18 months to two years.

Evolving Student Expectations in Online Learning

Kenny Kim: Well, certainly. I'll start, Josh, by saying, not just being the CMO of Udacity, but also being the parent of two school-aged kids and having nieces all the way up through college. What you learn, not just from my side, but also as a parent with kids learning from home, is:

• General Amenability: People are now generally amenable to online learning. You don't need to explain it anymore. Virtual webinars, for example, are second nature. My kids even use "Zoom" as a verb.
• Degree Relevance: From a degree perspective, many people are having a "come to Jesus moment" about whether their traditional degree will be enough for future jobs. For instance, my psychology degree came at an unprecedented time of unemployment. Will that liberal arts degree be sufficient for future job markets?
• Post-COVID Learning: If there's anything any parent has learned at home, it's that passively consuming content, one-way through a screen, is a terrible way to learn, let alone job-related digital skills, which is our focus. You can't just convert syllabi and lectures onto a laptop and expect people to learn by taking quizzes. It needs to be more than that.
• Proliferation of Options: There's a huge increase in available online learning options.
• Human Interaction: People are seeking different ways for human interaction. How do you incorporate human aspects to help people feel connected or get "unblocked" when they hit a problem they can't solve? When I took a math course in college, it wasn't a multiple-choice exam; you had to fill out problems. The best way to learn was going to office hours and having a TA review the work. Getting that human connection in online learning is critical.

Josh Francia: It's so interesting because I also have school-aged kids, and I definitely agree: you can't just put a lesson plan in digital format on a slide and say, "Hey, go to it and figure it out." Learning doesn't happen that way. It's very different.

The objective of teachers and institutions is to provide learning. When learning occurs, they're successful. How do they translate that for the digital medium? It's very different. You brought up good points: people still want and need some layer of interactivity and human interaction. They don't want to be lost to their own devices; it's not a "study alone" or "learn by yourself" thing. It's "learn with someone."

So, how is Udacity thinking about this? What are some ways you're providing that level of interaction and meeting these expectations?

Udacity's Approach to Interactive and Mentored Learning

Kenny Kim: Completely. We were fortunate enough to have a founder, Sebastian, who realized this 10 years ago. If you're going to leverage the online space as the medium to learn something new, especially a digital, employable skill, you have to make it more interactive.

Our learning platform combines:

• Short projects or lectures.
• Interactive quizzes.
• Project-based learning (what we call "experiential learning"), where you learn by doing something that replicates a real-world environment.

For example:

• In our self-driving car course, the final project involves programming a car we have in a parking lot.
• For our digital marketing course, you're actually in AdWords and Facebook Ads, developing real campaigns.

This tangibility is critical.

From a human interaction aspect, we've partnered with and developed a network of 1,400 mentors. These mentors are qualified as experts in their field, having worked for companies like Google, Uber, and Facebook. They are practitioners who do this for a living.

Just like my TA example, when there's no single way to technically answer a question or prove understanding, mentors do two critical things:

1. Unblock you: When you hit a roadblock (e.g., a compiler error in your code), they can help.
2. Verify skills: They review your project submissions to ensure you can apply those skillsets and have truly learned the material.

This is super critical in online learning because people learn and apply material in different ways. A multiple-choice exam or a "paint-by-numbers" project (where they tell you to fill in circle one with blue) isn't necessarily an effective way to train users or help them grasp material.

We also maintain a high bar for our mentors:

• 40% have a Master's degree or PhD.
• The admissions rate for mentor applicants is less than 10%, on par with Ivy League schools.
• Most importantly in the online space, people seek quick feedback. We guarantee a response time of less than an hour when you submit a project.

Josh Francia: I love that. I love it particularly because when anyone wants to learn a new skill, like programming a car or digital marketing, there's obviously a knowledge gap. You have to learn the fundamentals, read, take quizzes, and do assessments. But then there's the applicability concept: Have I learned enough to actually execute and do the thing I'm trying to do?

When you pair that with real-world practitioners, you get confirmation that your applicable skills are real. That is so valuable. If you're taking a course to up-level yourself, perhaps a psychology major wanting to become a programmer, you can't just say, "Hey, I took this course." You need the ability and confidence to know how to do it and interact in that arena. Those mentors you talk about provide that layer of confirmation and guidance, saying, "Not only do you have the skills, you also know how to apply them." I think that's amazing and so valuable.

Kenny Kim: Completely. I was a psychology major, and as a web developer, I took one CS course in undergrad. I learned on the job, by doing, and my mentor was my boss reviewing my work. The same when I transitioned to marketing: I was hired by a marketer who felt he could teach me the requisite materials. I learned on the job. I look at the world we're in today, and I hope folks are lucky enough to have that opportunity. But more importantly, if you don't, that platforms like ours exist to replicate that environment, allowing for a more material impact globally.

Josh Francia: Yeah, absolutely. My journey was somewhat similar to yours. I was a journalism major, took a couple of CS courses, started in web development, and then found myself in marketing somehow. Funny how we both kind of arrived at the same endpoint from different paths.

Udacity's MarTech Stack and Technology Strategy

Josh Francia: Talk to me about some of the technology that you guys use at Udacity. Your MarTech stack, your other parts of your technology, to actually enable this. This is not easy. This is a big challenge to provide this level of interaction to all your students at scale, at the millions level. So walk me through some of that and maybe provide some advice to people who aren't at the same stage you guys are with their technology, but want to get there.

Kenny Kim: Yeah. Messaging and interaction are critical to our platform. We are a Blueshift client. Between that and our chat tool, Intercom, you quickly realize you want to reach users through the channel that engages them. It's easy when they're online and learning. At the point of purchase, people are super excited. They provide their onboarding learning, interests, and information. But then, what you do with that information is critical.

Understanding their learning objectives and goals is key. We leverage chat with our "education specialists" to help people browsing understand the distinctions of our programs and find the right one for them, as there's a proliferation of options. From a personalization perspective, regarding recommendations, once they complete an introductory course, we recommend different paths or what would be relevant to them, or what people like them (who have graduated) found important.

We also do a lot of analytics, leveraging our data science team to identify critical "moments of truth" that ensure people stay on track. For example, getting that first project submission is critical to driving engagement. Graduation will have a critical impact on whether they'll see a positive career outcome from being on our platform.

It's a combination of:

• Capturing data upfront.
• Leveraging tools like Blueshift and Intercom to communicate with users through SMS, email, and chat.
• Utilizing our analytics platform to glean insights and democratize the information we have.

Josh Francia: You talk about these different platforms, from the messaging component (Blueshift and Intercom), and how you can talk to customers whether they're in or outside the platform, and the best way of doing that. How important do you think it is, as Blueshift is a CDP, capturing the full profile of the customer? How valuable is that when you're starting to do these things to make sure that you can continue to engage with customers along the way?

Kenny Kim: Critical. As a marketer, you want to democratize communication channels or access to that customer for your marketers. The more they rely on engineering or other tools or platforms to get to that point, the more difficult it becomes for access and insights. Having a CDP accelerates the process of identifying segments or types of individuals you want to reach.

Finally, if you're just relying on email, even at industry-best open rates, you're only touching 30-40% of your population. So, for re-engagement, leveraging a platform like Blueshift to re-market to users who aren't opening your emails, or extracting that information to target them in direct mail, is critical.

Josh Francia: Exactly. I think about the journey of a student taking a course at Udacity. They're excited at the beginning, go through a couple of lessons, but then some might get sidetracked. Maybe they don't do the third or fourth lesson, things get in the way, or they find it too hard. They might start to waver and potentially not complete the course.

What are some ways you target those people who seem to be "churning" (not as in leaving as a customer, but not completing the course)? What tactics do you use to bring them back, re-engage them?

Kenny Kim: We have inactivity re-engagement emails where we reach out to users to get them back into the classroom. With the help of Blueshift, we've optimized to identify the right intervals to reach them before they hit a point of no return. So we'll send messages at day 4, day 8, day 11, and day 20, with the message varying and constantly being tested and optimized. We don't want to be a blunt instrument. We're not just saying, "Hey, you haven't logged in in four days. How's it going?" There's a lot of care and testing the team put into those seemingly arbitrary intervals. We also optimize for send time based on the individual and their likelihood to open emails. For some people, they're morning folks; for others, afternoon. It's critical.

Josh Francia: How important is that? A lot of the old thinking in marketing is "batch and blast." We still hear that from time to time: "Just load everyone up and send them something." You mentioned earlier, you only get 30% at best with that approach, so it's flawed. But what about the nuances you just talked about: ensuring the right intervals, the right personalization, the right timing? How should someone not at Udacity's level think about that, and the value those types of insights and signals have on your ultimate outcomes?

Kenny Kim: You have to understand your business model and your user. What's important to them? I have a big background in e-commerce, travel, and other areas. There are always those "rules of thumb": "Monday morning's best," "don't ever send after Thursday," "mobile's greater for that," and so forth.

But you realize that people are taking our courses on their own time. Evenings are probably a high point of engagement for them. If you're hitting them in the middle of a workday when they're thinking about paying bills, it's probably not going to be very effective. So there is some nuance. There's a general rule of thumb and best practices for batch and blast, particularly for commerce. But for usage engagement, you have to put yourself in the shoes of your user. When is it going to be top of mind for them? Just like any CPG product, or if you're Instacart and thinking about shopping, that typically happens on weekends and late afternoons. For e-commerce, early Monday morning is surprisingly popular, as most people do their shopping in the middle of a workday. For us, the context is different.

Josh Francia: And it's interesting because if you miss it, you miss the window. It's not like they remember you. If you send them a message at the wrong interval, when their mind is focused on work or shopping or taking care of the kids, they're not focused on completing the course or logging back into Udacity. It's not like they remember eight hours later to go back. You kind of missed that interval. Getting that timing right is so crucial.

Kenny Kim: If you look at open rates as the only example, they might get drowned out in a few percentage points. But the click-through rates, that material difference in engagement, will show up much larger if you're thoughtful about it and don't carve up your test into too small segments.

Josh Francia: Okay. So we talked about re-engaging students who are falling behind. They've already put money out there for the course, so there's some motivation not to let the money go to waste and finish the nanodegree. What about people who have completed their first course, and now you're trying to get them onto a second course? Walk me through some of the things you do to get them to do that second course in short order. Have you found any magic windows, like, "Hey, people who do a second course within a certain period of time are going to be very good, long-life Udacity customers," or "If they don't, they'll never come back"? Walk me through some of those engagement ideas.

Kenny Kim: The course completion. One thing we try to be very mindful of is giving them some time for that to marinate. It also depends on where they are. If they're developing a new skillset to see a positive career outcome, you have to wait. We do a graduation survey to our graduates six months after they left. We have to see if we've had a material impact. For individual learners, 70% of them report back that they're seeing a favorable income outcome (e.g., a pay raise, a new project, a new job, or starting a new company). So there's a certain amount of time, particularly if that nanodegree was meant to launch them (e.g., a cab driver becoming a web developer, or a waiter becoming a full-stack developer).

For up-scalers—individuals who are intellectually curious and building out their existing pathway—recommendations are key. We leverage our data science team to build models that recommend courses based on what they've consumed before. We talked about older trends, like the proliferation of choice, not just in learning providers but in content. Even in data science, there are new courses, specializations (AI, RPA). There's no shortage of choice. It's just a matter of condensing that down.

Josh Francia: That is the problem a lot of companies struggle with, right? They almost have too many choices, and consumers struggle when they have too many choices. It's almost worse than having no choices. So how do you sift through all that? Because all consumers want some choice, some variability, but they want a finite number of choices that they don't have to struggle with.

Kenny Kim: That's a great suggestion. One thing we really focus on is that we've been very mindful of curating our content for the workplace and making sure we have the right skill sets. If you come to us wanting to learn JavaScript, there's one course for intro to JavaScript, intermediate JavaScript. It's not a marketplace with 50 different courses, hoping it's the right one. This allows us to focus not on sifting through subject matter, but on their outcomes. Do you want to be a full-stack developer, or are you just writing scripts for a side website?

Josh Francia: And that's great because you reduce the paradox of choice by focusing on the career pathing.

Kenny Kim: And then we limit the amount of choices. We don't need to show you the whole category of nanodegrees. It's a matter of, "Okay, if you've taken this course, here are two or three other logical choices or recommendations we can provide."

Josh Francia: Very cool. So, going back to, we know that Udacity has done a lot to bring that human interaction to a digital presence. We also know during this pandemic that not doing that is a very poor way of learning. It's hard to learn all on your own. We've all experienced it as parents numerous times. Udacity has figured out how to bridge that gap, provide that value.

What are some of the risks, do you think, of other educational companies not adjusting or not being willing to adjust to these new expectations of this online learning environment with tight interactivity?

Risks of Not Adapting in Online Education

Kenny Kim: I think the two logical ones are churn (if you're a subscription company, that's the four-letter word of death) and low graduation rates. When we talked about expectations, there are outcomes. What is this education going to give to me? Even if it's a sourdough making class, you want to know how reliable it is, what materials you need, and what to expect.

For us, it's a matter of giving confidence day one. Whether you are a current developer learning a new skill set that you can apply in the workplace, or if these aren't trivial costs that you're shelling out, we're dealing with people's livelihoods and their long-term careers. So, churn, low graduation rates, and then just expensive traffic or increasing cost of traffic for you. There's no shortage of options. People will be more inclined to try a free option if they're not driving the outcome for you.

With the proliferation of free options and access to free content, you quickly realize that not all content is created equal, just like not all online learning providers are created equal. So you need to be very mindful of not just making the marketing claim, but demonstrating how your platform is materially different and will help you learn. For me, that analogy of having a TA in an Uber-like, on-demand model was super critical.

Josh Francia: That's a great point you bring up. There's no shortage of options. You do a Google search, you can find hundreds of courses or ways to learn JavaScript. Some are free, some aren't. Some are someone's YouTube channel. There's just a lot of content out there. But ultimately, there's a variety of quality. If you're serious and truly want to learn and use it as a stepping stone or a career pivot, how do you guys make sure you stand out in that noise? What does Udacity provide to ensure new customers looking for what you offer are aware of your unique approach and glean that you're different? How do you do that?

Kenny Kim: That's a great question. There are about five key questions I would ask people:

1. How do you ensure your program/platform is high-quality and relevant to the workplace? Our mission is to train the world's workforce for the careers of the future. This means we're getting people job-ready for roles we think will be in demand. This gives you the confidence that you're investing in something with lasting impact for your livelihood. We've spent 10 years focusing on this singular mission: developing and teaching job-ready digital skills online. We've perfected and iterated; we didn't just convert syllabi and put them out there. We continuously tinker with our learning methodology and technologies to engage users.
2. Content Curation: We curate our content. We develop one course for self-driving, one for introductory JavaScript. It's not a marketplace where we put the onus on the user to figure out what's right for them. We start with the resume and the role in the professional workplace as our starting point and work backward.
3. Practitioner-Led Instruction: We work with practitioners and experts in the field. We launched our first Android nanodegree with Google. We work with automotive and self-driving companies for the self-driving car course. Our programs are rooted in people who do this for a living and excel in it, as opposed to pure theory. For example, my CS course didn't teach me how to program a webpage; I learned how to program blackjack.
4. Skills Verification: Because we're rooted in digital technology, we leverage mentors to review code. These practitioners and experts say, "Yeah, that's a creative way to approach that. That may not be the right answer, but this person gets what they're talking about." Building a network of that size allows for 24/7 access and responses in less than an hour, which are consumer expectations today. You can order a bottle of Gatorade and get it in 30 minutes; why wouldn't they expect the same for human feedback on code review?
5. Impact Measurement: How are you making an impact? 70% of our individual learners report a positive income outcome. For our corporate clients, we provide business impact reports quarterly to show how learning directly applies to their business.
6. Unblocking Users: How do you unblock users when they're stuck? This goes back to our mentors and helping address problems, and developing a peer network.

Josh Francia: I love it. At the end of the day, when someone needs to learn and is willing to commit the time, the biggest risk they have is that they don't learn or acquire the skill they're seeking. The way you've presented this mitigates that risk considerably. You worry about whether what you're learning is applicable in the real world. With Udacity, it is, and here's someone in the real world doing the job you want to do, telling you it's real. Content is co-created by leaders in the field, specialized for the workplace, and experts applying those skills help you along the way. I can't think of a more pragmatic combination of interaction and verification that's replicated in the marketplace today.

Essentially, it's like the old apprenticeship model, where people learned from masters of a craft, tied with traditional literature, all married in one digital interactive platform. You can work at your own pace, on your own schedule. It seems like you've combined many needs for true learning and provided it simply, enabling success. No doubt that's why 70% of graduates actually see a positive impact. That's an astonishing statistic.

The Future of Online Learning and Advice for Brands

Josh Francia: We only have time for a couple more questions. You talked about how if companies don't adapt, they'll be left behind, obsolete. If you're the company that just says, "Well, we're not going to pivot, we're not going to embrace this," you may not be around much longer. So what's your advice for someone who says, "Hey, I want to embrace this. I want to change. I don't know where to start. My company isn't Udacity. We didn't think about this. We're different than you guys." What is your advice for them if they have that desire to change but aren't sure where to start?

Kenny Kim: I think first, work backwards. Look at it from two different perspectives. For other education companies, focus on driving outcomes and hold yourselves and your colleagues accountable. Graduation rates and making an impact have always been paramount here and non-negotiable.

Think about the applicability of the content you're creating. Monitor your engagement data closely and keep yourself honest on what's relevant. Technology is changing over and over on a regular basis. Even our digital marketing course, for example, has nuances in Google and Facebook platforms that might have been relevant a year ago, but any digital marketing practitioner knows those platforms change quarterly. So you have to refresh your content. Core basics in accounting might not change in five years, but in the professional environment, especially digital, technologies and platforms can change dramatically in 24 months. You have to reinvest in and upgrade that content.

Help connect the dots on how your courses build and lead to outcomes. We ran that survey to learn whether students were achieving outcomes, and we were pleasantly surprised to see that 70% stat around positive outcomes. For data points that aren't positive, dig deeper. Do that first wave, then iterate, talk to those customers, and try to understand their expectations and what drives them. We have certain learners who are very advanced and intellectually curious; they want to be left alone and will complete courses on their own. Others are starting from ground zero and have no idea where to begin. How do you capture that information about the user and then align their expectations to that outcome to best help them succeed?

Josh Francia: Okay. So last question for you, Kenny. Thank you so much. This has been super helpful and interesting to hear your thoughts and how you approach these challenges. What does the future look like? We saw the last two years have been a major step change in expectations. What does the future look like from your perspective?

Kenny Kim: It's tough to say what the future looks like because I don't have a crystal ball. I think the way I would answer is: What does the future need to be? I feel like this space needs to mature. There needs to be broader acceptance from Capitol Hill and across the globe in terms of usage and engagement. For the people doing it, I think they're seeing the material outcomes.

If you look at the increasing demand for job creation due to automation, AI, and ML, the amount of jobs created from those technologies will dwarf the jobs lost. The onus is on all of us, our platform and others, to give people the means to learn these new technologies so they can prosper in their careers and livelihoods. The university system alone isn't going to close the gap on that increased demand. Thankfully, they're investing in ways to make those offerings more available.

My predictions:

1. Online learning will probably be one of many choices for my son's generation to develop skills.
2. Developing skills versus degrees will be critical.
3. The need for continuous and ongoing learning will be recognized for careers.
4. My broader challenge and mandate is to accelerate funding from federal and foundation perspectives into these programs to get more people into roles that align with future demand.

Josh Francia: Awesome. I love it. I love those predictions, and I guess we'll wait and see, but I agree. I think it'll be an exciting time to see a lot of those things manifest. Well, Kenny, thanks again for the time and for all of your insights. This has been a fun conversation, and hopefully everyone watching gleaned a lot from this. Hopefully, you figure out the next steps to get your company and organization in the right spot to capture new demand and new expectations as customers' expectations continue to change with this new world and new technology. Thanks again, Kenny. Thanks so much for joining me and thanks so much for watching.

Kenny Kim: Thanks so much.