UXR Case Study:

Duolingo User Research Study

Role: Primary Researcher & Secondary Writer

Timeframe: October - December 2023 

Problem context

Our team project revolves around researching and conducting an extensive user research report and testing phase for Duolingo, a prominent mobile language learning application. Duolingo was selected due to its widespread popularity and its mission to democratize language learning across various languages such as French, Spanish, Italian, Japanese, and more. The app uniquely combines technology with a playful approach, offering bite-sized, free lessons to make language education accessible to all.

However, a notable concern among users is the apprehension about the app's efficacy in enabling fluency and the skepticism surrounding whether users can attain full proficiency in a language solely through the app's methods. Addressing this skepticism and exploring the perceived limitations of Duolingo's efficacy in achieving fluency is a key focus of our research. 

the team

Holly Liban

Primary Researcher & Group Facilitator

Emma Thomas

Primary Researcher & Primary Writer

Brandon Turman

Primary Researcher & Primary Writer

Gael Mateo Cruz

Primary Researcher & Secondary Writer

Sinit Tesfamariam

Primary Researcher & Secondary Writer

Oct 1, 2023

Participant Recruiting

Data Analysis

  • quantitative findings:  

    • Italian Pre-test vs. Post-test:

      • Analyzed 9 questions, observed 17.9% improvement in correct responses.

    • Fixation Duration & Time of Interest (eye-tracking):

      • Consistent durations imply no major UI issues.

      • Focus on content over navigation, except for Participant 012's abnormal setup.

  • qualitative findings:

    • Positive Items Users’ Liked:

      • repetitiveness, retention, user interface, & game-like feel

    • Negative Items Users’ Disliked:

      • lack of variation, ads/pop-ups, elementary feel, & heart feature and instructions

  • main usability issues: lack of variation, ads/pop-ups, & lack of learning

Planning Phase

  • project title: “Assessing the Effectiveness of Duolingo’s User Interface in Teaching Italian to Novice Language Learners”

  • incentive budget: $75 in total; 15 participants, with $5 Starbucks gift cards

  • usability goal: We aim to assess Duolingo’s effectiveness in teaching Italian to new college users and identify app enhancements

  • testing schedule: Our testing period was over three days, mixing lab and remote environments

Nov 14 -22, 2023

Usability Testing

  • quantitative metric: Tobii Pro Glasses II, Italian Pre & Post Test

  • qualitative metric: Exit User Interview

  • tasks: Seven tasks were given to participants over three days.

    • Days 1 & 3 were done in the User Experience Lab at UTK campus

    • Day 2 was done remotely by participants on their own time and environment

Dec 12,  2023

Presentation

  • link: https://docs.google.com/presentation/d/1EGdxyEG7x6my95DF3t60oUEnYtDWml_anuoYZIrr7z4/edit?usp=sharing

  • Dr. Dania Bilal (our professor) commented: 

    • “Very interesting!”

    • “Presentation and report went above my expectations!”

    • “Almost at the level of Graduate students.”

Timeline

Oct 30 - Nov 13 , 2023

  • participant requirements: Participants made of college undergraduates (freshmen to seniors) without prior exposure to Duolingo or Italian

  • inclusivity measures: An open invitation to all eligible undergraduate students at UTK with no discrimination based on race, gender, or disabilities

  • recruitment process: Each team member recruited three participants (total of 15), recruited UTK’s students, and provided consent forms

Dec 5 - 11, 2023

Usability test goals

When conducting this usability test, our team decided to focus on answering two questions based on quantitative (eye-tracking) data and qualitative (user interviews) data. Our usability test evaluates the effectiveness of Duolingo in helping college students new to the application learn a new language, specifically Italian.

  1. How effective was Duolingo in helping college students new to the application learn a new language, specifically Italian?

  2. What could be improved in the Duolingo app to help college students learn a new language? 

    • (Examples include a more accessible user interface, easier navigation, and more intuitive lessons, etc.) 

Participant recruitment

Recruitment Approach

We recruited through campus engagements (with classmates, student union) and targeted emails, ensuring a diverse participant pool from the University of Tennessee-Knoxville. This approach created a comprehensive base for our study.

Participant  Recruitment

We aimed to recruit University of Tennessee-Knoxville college students unfamiliar with Duolingo or Italian, seeking fresh perspectives on the app's usability for newcomers to both the platform and the language.

Reason for Selection

College students offered crucial insights into Duolingo's usability for newcomers due to their distinct tech inclinations. Their representation enriched our study by highlighting pertinent usage patterns among those starting language learning journeys.

Screening Criteria

We implemented specific screening criteria, confirming University of Tennessee-Knoxville enrollment, assessing prior Duolingo usage, and evaluating Italian language familiarity. These criteria were vital in selecting participants aligned with our study's objectives.

Usability test Methodology

Overview:

Methods Used:

The usability test methodology comprised a structured series of tasks conducted over multiple days to assess participants' interaction with Duolingo, a language learning application. The test aimed to evaluate usability aspects, language learning performance, and user experiences through a sequential set of tasks.

Tools Utilized:

Usability Tasks

DAY 1

DAY 2

DAY 3

  • Questionnaires: Pre-testing questionnaires containing open-ended questions were administered to gather participant demographics and prior experience with Duolingo and language learning.

  • Online Surveys (Google Forms): Utilized for pre-tests and post-tests to evaluate participants' existing and improved language knowledge after completing Duolingo lessons.

  • Eye-Tracking Technology (Tobii Pro Glasses 2): Employed to observe participant behavior and gaze patterns during specific tasks, such as signing up for Duolingo and completing lessons.

  • Unmonitored Environment (Day 2 Tasks): Allowed participants to complete tasks independently without direct monitoring to simulate real-life usage scenarios.

  • Exit Interviews: Conducted to gather qualitative insights and user experiences regarding the usability testing process.

  • Pre-testing and Post-testing Surveys: Google Forms were used to administer Italian pre-tests and post-tests, assessing participants' language proficiency before and after completing Duolingo lessons.

  • Tobii Pro Glasses 2: Eye-tracking technology employed during certain tasks to monitor participant gaze and interaction patterns while using Duolingo on mobile devices.

  • Duolingo Application: Used as the platform for language learning tasks. The application's features, including lessons and heart mechanics, were integral to task completion and assessment.

task 1

Users completed a screening pre-test questionnaire which determined usability test eligibility.

task 5

Users completed the final three lessons while using the eye-tracking glasses.

task 2

Completion of Italian pre-test via Google Forms to assess language background.determining usability test eligibility.

task 4

Users independently completed three Duolingo lessons on their own time and environment (no monitoring).

task 6

Users took an Italian post-test, via Google Forms, aiming for improved scores.

task 3

Users signed up for Duolingo by making an account and then, completed three initial lessons.

task 7

Users completed exit interview to share their Duolingo usability experiences.

Results  &  findings

Quantitative Findings:

Italian Pre & Post-test Results

After comparing post-test results to pre-tests, we found a 17.9% improvement in correct response averages across 9 identical questions. This indicates positive progress in language learning post-Duolingo.

Qualitative Themes:

Total time interest duration, reflecting completion time, didn't significantly vary between Days 1 and 3. Factors like lesson difficulty impacted completion time, and technical issues excluded one participant's eye-tracking data. Overall, participants' completion times varied, hinting at different language-learning levels and reading speeds, concluding that Duolingo's interface and navigation were user-friendly. 

Fixation Duration Results

Fixation durations remained consistent between Day 1 and Day 3 for all participants, suggesting an effective, non-distracting interface. While one participant had notably longer setup time skewing their data, others showed consistent fixation durations conducive to learning.

Total Time Duration Results

Positive Themes

Participants praised Duolingo for its repetitiveness aiding retention, game-like feel, varied lesson formats, and user-friendly interface. Specific positive feedback included:

"The repetitiveness really helped me retain vocabulary."

"The game-like feel made learning fun and engaging."

"The varied lesson formats kept it interesting."

"The user-friendly interface made navigation easy."

Negative Themes

However, concerns were raised regarding lesson variation, desire for more conversational content, issues with disruptive ads, perceptions of simplicity, pronunciation accuracy, vague instructions, and dislike for limiting features. Negative feedback included:

"I wish there was more variation in lessons."

"Ads disrupt the learning experience."

"The app feels too elementary and lacks complexity."

"Pronunciation accuracy needs improvement."

analysis & Solutions

Usability Issues

lack of variation

Participants sought varied structures, desired conversational learning over mere vocabulary, and craved additional cultural context in their language-learning experience. (Figures A, B)

lack of learning challenge

Participants noted the absence of a learning challenge, expressing Duolingo's focus on vocabulary rather than conversational or practical learning, perceiving the app as too elementary; additionally, the heart’s feature halts learning once depleted. (Figures G, H)

Key Solutions

ads/pop-ups

 Participants felt there were too many pop-ups and ads. (Figures C, D)

Figure A

Figure C

Figure D

Figure G

Figure H

Increase variation of language content within Duolingo lessons.

Decrease ads (if possible) in free mode.

Choose specific topics within the language to enhance learning.

Figure B

conclusions

Implications of Findings

  • Usability Assessment and Interface: The absence of significant usability issues in Duolingo's interface suggests a fundamentally sound platform. However, the study's insights highlight the importance of continuous improvement in specific areas for enhanced user engagement.

  • Content Variability and Personalization: The study underscores the necessity of diverse content to cater to personalized learning experiences. Implementing adaptive content strategies could positively impact user engagement and retention rates.

  • User Empowerment through Topic Selection: Providing users with the autonomy to select specific topics aligns with the study's findings. Enabling users to personalize their learning paths can significantly enhance their engagement and satisfaction with the platform.

Summary of Findings

The study conducted an in-depth analysis of Duolingo's user experience, combining quantitative and qualitative data. While some concerns were identified, such as the need for varied content and user-driven topic selection, the overall assessment didn't reveal significant usability issues with Duolingo's interface.

Future Considerations

  • Continuous Iterative Improvements: Encouraging ongoing assessments and iterative improvements in specific aspects of the platform, despite the absence of significant usability issues, will contribute to maintaining a high-quality user experience.

  • Content Development Strategies: Emphasizing diverse content creation and implementation strategies, including adaptive learning approaches, could solidify Duolingo's position as a personalized and effective language-learning platform.

  • User-Centric Features: Implementing features that empower users by offering more control over their learning experiences could be a strategic direction for future platform enhancements.