Project Overview

In 2019 and again in 2022, I led comprehensive web and mobile quality studies for Chile's major pension fund administrators (AFPs), including subsidiaries of global financial leaders:

• MetLife (AFP Provida)
• Principal Financial Group (AFP Cuprum)
• Generali Group (AFP PlanVital)
• AFP Habitat
• AFP Capital
• AFP Modelo
• AFP UNO

The project aimed to evaluate and improve the digital experience for millions of Chilean citizens who rely on these platforms to manage their retirement savings.

The study established standardized quality indicators that became industry benchmarks, allowing AFPs to assess their competitive position and identify critical areas for improvement. By combining expert evaluation techniques with extensive user testing, we delivered actionable insights that helped transform the digital experience across the industry.

Challenges

The Chilean pension system is mandatory for all workers, making AFP platforms essential services that must cater to users of all ages, technical abilities, and socioeconomic backgrounds. Through our initial research, we identified several critical challenges:

1. Inconsistent User Experience: Each AFP had developed its digital platforms independently, creating wildly varying experiences with no industry standards for quality or usability.
2. Technical Complexity: Pension management involves complex financial concepts and transactions that needed to be simplified for average users.
3. Critical User Tasks: Certain operations like fund switching, password recovery, and pension application had particularly poor success rates across platforms.
4. Authentication Issues: Multiple security credentials (access passwords and security keys) caused significant user confusion.
5. Accessibility Concerns: Most platforms fell well below WCAG standards, creating barriers for users with disabilities.
6. Mobile Experience Gaps: The growing number of mobile users faced particularly challenging experiences.

Methodology

We developed a comprehensive evaluation framework that combined quantitative measurements with qualitative insights:

Expert Evaluation Component

• Heuristic Analysis: Assessment of 41 standardized usability principles
• Cognitive Walkthrough: Expert simulation of user behavior on critical tasks
• Accessibility Evaluation: Analysis of compliance with international standards

User Testing Component

• Task-Based Usability Testing: 12 users per AFP testing seven critical tasks:
  • Access password recovery
  • Security key recovery
  • Security key request
  • Fund switching
  • Fund distribution
  • Viewing account balance
  • Initiating pension application
• Metrics Collection: Success rate, task completion time, perceived ease of use
• Post-Task Surveys: Satisfaction and recommendation likelihood

Quality Indicators

We developed a proprietary Web Quality Service Indicator that combined:

• Technical quality (from expert evaluations)
• Perceived quality (from user testing)
• User efficiency (time to complete tasks)
• User effectiveness (task success rates)
• Satisfaction (post-task rating)

Process

The study was conducted during the two distinct periods, allowing us to track improvements and changing user expectations:

Key Findings

Our research revealed several patterns across the industry:

• Authentication Issues: Users consistently struggled with differentiating between access passwords and security keys, with recovery tasks showing success rates below 40% for most platforms.
• Fund Management Complexity: Processes for switching and distributing retirement funds used inconsistent interaction models, with some platforms showing success rates as low as 33%.
• Pension Application Barriers: Starting the pension application was the most problematic task industry-wide, with over half of users unable to complete it successfully.
• Mobile Experience Gaps: Tasks performed on mobile devices showed 15-20% lower success rates compared to desktop, particularly for complex operations.
• Consistency Problems: Many AFPs had different visual and interaction designs for different sections of their platforms, confusing users when navigation patterns suddenly changed.
• Accessibility Shortcomings: The industry average for accessibility compliance was below 45%, with contrast issues and screen reader compatibility being particular problems.

Solutions and Impact

Based on our findings, we developed tailored improvement roadmaps for each AFP:

1. Authentication Solutions
   • Simplifying credential management
   • Implementing contextual help systems
   • Creating clearer recovery paths
2. Task Redesign Recommendations
   • Developing standardized interaction patterns for complex tasks
   • Simplifying multi-step processes
   • Redesigning fund management interfaces
3. Information Architecture Improvements
   • Reorganizing navigation to match user mental models
   • Creating consistent terminology and labeling
   • Implementing progressive disclosure for complex information
4. Mobile Optimization Strategies
   • Adapting critical flows for small screens
   • Designing touch-friendly interfaces
   • Maintaining consistency across devices
5. Accessibility Enhancements
   • Implementing higher contrast ratios
   • Improving screen reader compatibility
   • Adding alternative navigation options

Measurable Impact

By the 2022 study, we observed the following industry-wide improvements (averages across all AFPs, not specific to any individual pension administrator):

• Increased Task Success: Overall success rates improved from 67% to 78% across the industry
• Improved Efficiency: Average task completion time decreased by 18%
• Higher Satisfaction: User-reported satisfaction increased from 5.9 to 7.3 out of 10
• Accessibility Progress: Average accessibility compliance rose from 43% to 57%
• Competitive Improvement: The gap between the highest and lowest performing AFPs narrowed significantly

Lessons Learned

This industry-wide study provided valuable insights beyond the specific findings:

1. Benchmark Power: Establishing industry benchmarks created healthy competition and motivated improvements more effectively than individual evaluations.
2. Mixed Methodology Value: Combining expert evaluation with user testing provided a comprehensive view that neither approach could deliver alone.
3. Standardized Tasks: Using the same task set across platforms enabled meaningful comparisons and identification of best practices.
4. Longitudinal Insights: Comparing results between the two studies revealed how improvements in one area sometimes created unexpected issues in others.
5. Implementation Challenges: Some technically simple fixes (like improving contrast) were implemented quickly, while more complex architectural changes took longer despite their importance.

Conclusion

The Chilean Pension Fund Administrators Web Quality Study established a new standard for evaluating essential financial services. By developing industry-specific quality indicators and conducting rigorous evaluations, we helped transform digital pension management in Chile.

The methodology we developed has since been adapted for other financial sectors and government services, demonstrating its value beyond the original context. Most importantly, the improvements implemented as a result of our work have made managing retirement savings more accessible to millions of Chilean citizens, regardless of their technical expertise or abilities.