Real-time Concepts on YouTube
Oct 2022 - Nov 2023
Overview
Have you ever paused a video while you were learning something because you wanted to look up something the creator mentioned?
Have you ever then gotten distracted enough not to return to the video?
This project started because I found myself wondering how often students pause because they don’t understand something, and how many dropped off after that moment. I pitched this idea to an engineer
on my team, who helped me dig up relevant data to help me validate my hypothesis. What started as a simple hypothesis ultimately led to a high priority
feature in my team's roadmap, and ultimately launched in YouTube less than 2 years later.
Validating My Hypothesis with Data
The hypothesis was clear and simple: While attempting to learn by video, users may get stuck when they don't understand a term or concept, and we risk losing them, whether they simply get lost and give up, or if they look up the information elsewhere and decide not to return to the video.
Screenshot of a Geometry Video
Questions can also come in many forms. Most simply, they can be basic definitions and explanations. But given the multi-modality of video,
I thought there were also opportunities to explore visual questions within a video frame. Moreover, I thought about how often one does not
even know yet what questions they have, in which case, could we intelligently extract data from the video and proactively offer answers to
questions that we anticipate the user to have?
This is how Real-time Concepts on YouTube was born.
And the patterns in the data analysis were significant:
We found that not only were users pausing significantly more in videos classified as "learning" videos than non-learning videos...
...they were pausing for significantly longer.
AND the rewind and fast-forward behaviors were vastly different. While there were significantly more rewind behaviors for learning videos, there were also significantly less fast-forward behaviors. Everything we found in the data seemed to suggest that my initial hypothesis was on to something.
Live now: RTC across all Learning Videos on YouTube
Within 1 year of the initial conception of the idea, we completed rigorous data analyses and research, experimented with successful metrics,
and completed a full launch.
This project gained momentum to build a new team, which continues to work on scaling this feature. Teams from other verticals have also reached out
because they want to scale it to their teams. This is an overview of the MVP that we launched:
Beyond MVP
In January ‘23, there was a top-down ask to integrate LLMs / AI into our feature roadmap. At the time, a lot of the focus was on integrating a chat experience across products. The team was looking for a quick win and considered updating the concept definition with an explanation from Gemini (fka Bard). However, I was not convinced that it added any value to the existing feature as it was designed. Plus it’s a lot of text! The focus at the time was primarily text-based and I thought it was important to incorporate visuals into the content. I tried generating diagrams with Imagen (internal tool) but found them to be incomprehensible.
Text explanation
Diagrammatic explanation
However, it gave me another idea... What if we used LLMs to generate examples, analogies, and metaphors? Similar to how a good teacher might explain something complex.
Incorporating LLMs into Real-Time Concepts
LLMs have certainly come a long way since these designs were explored in early 2023, but at the time, we had to be extra creative in our prompting because they weren't nearly as powerful as they are today. Months later, we launched contextual explanations, which explained the concept within the larger context of the video, and had a list of ideas for how to evolve this feature further, such as enriching the explanation with examples and fun facts that learning more accessible.
Soon after our initial launch, we built an agentic AI into the experience, so that not only could a user follow up with questions, we could anticipate their questions based on the content from both the video and the page. Below are some principles I established as part of our core learning journeys, as we built this feature in partnership with the YouTube Main App team. Because the PM was on leave at the time, I drove this product vision and owned the presentation of this work to senior leadership.
Proactive Answers
Multimodal Explanations
Video-First Explanations
Persistent Experience
This deeply collaborative project ultimately launched in the YouTube Mobile app after many months of testing. Due to shifting team priorities, we no longer had the bandwith to further invest in RTC and integrate it more closely with this agentic experience as I had proposed. Nonetheless, I am still incredibly proud of the experience we were able to ship after a very grassroots grounds-up approach starting from problem/hypothesisic identification, going through qualitative research and design iteration, pitching to stakeholders and incorporating it into the roadmap across organizations, and ultimately building and shipping it across all of YouTube.
