August 2018 - Oct 2019
Experimentation…
is the fastest, most insightful method to test a hypothesis for a low risk and innovative product development that responds to real user problems
Role: UX Strategist (Experimentation Initiative Lead)
Company: Adevinta Central UX Team working across Global Marketplaces
Project timeline: Indefinite - continued efforts to improve our craft
Problem to solve
Adevinta consists of many Marketplaces with different levels of experimentation maturity (product development and testing). There were varying perceptions of what is regarded as ‘being good in your craft’. This project was aimed at creating an awareness and a platform to drive experimentation, making it an integral process of any product development, while always striving to learn and improve the craft to have a greater impact on business KPIs.
Top Level Objective:
Adevinta is a User Centric Organization showing value through experimentation
What are we doing?
Empower teams in Adevinta with the right experimentation framework and tools to build business cases based upon user centric problems and learnings
Why are we doing it?
OKR: Adevinta believes that if we deliver via experimentation the right context, via the right channel at the right time, we will increase user satisfaction and liquidity.
How we measure success?
KR1: Deliver value faster by improving our experimentation processes/knowledge across 70% MPs.
KR2: Connecting MPs User Problem with SubMetrics to measure performance on Liquidity
How was it solved
Process, method, team structure, examples of phases
Experimentation is the way of building successful products, and teams needs to be adopted it naturally, and continuously use it to help with making data-informed decisions, with an ability to test many ideas quickly, with good team velocity, with alignment, with managing up, thereby improving time to market and minimizing risk.
Set up an team structure
Working with the Global Markets across all of Adevinta, I was the Experimentation Lead for this initiative My initial outset was to establish if there was a “experimentation culture”, and then raise the bar within the company over time through different phases. The core Central team consisted of multi-disciplines from UX, Strategy, and Data & Insights, working with some local teams from several Global Marketplaces. Later in the project I hired a A/B Testing expert who played a key role in raising the bar.
Project phases:
Phase 1: Is there culture for experimentation?
Phase 2: Experimentation, Research, UX Best Practices Toolkit
Phase 3: Experimentation & Research Repository
Phase 4: Research & Experimentation Guild of experts
Phase 5: Experimentation Maturity assessment
Phase 6: Awareness training workshops
Phase 1: Is there culture for experimentation?
Solution
We needed to see if there was for starters, a culture of experimentation. Keeping it simple and trying not to make too much work for the local teams, we tried to understand if there was any experiments taking place, and if any what was the frequency. Working with 14x marketplaces we asked them to ‘document their experiments’, and using a simple excel sheet we collected the minimal metrics that will allow us to track the progress of all teams, mature and immature. Metrics:
Vanity metrics
- Number of experiments
- Number of valid experiments (following a common criteria)
- Number of users reached (interviews, surveys, AB tests)Qualitative Data
- Number of learnings (including experiments that failed)
- Number of achievements (new features, usability errors fixed, customer support tickets resolved)KPI: Number of Marketplaces who have completed the sheet
Results and learnings
The result was a clear indication of the state across the company, some with experimentation as part of their everyday sprint while others only a few a year. There were also some mixed feelings, as some companies felt that the time it took to documenting their experiments was a duplication and a waste of time - more like reporting. Others thought it was very useful as they could learn and gain valuable insights. At the time we were also building a Research/Experimentation repository where all global markets could upload their work (see Research Portal project below).
Phase 2: Experimentation, Research, UX Best Practices Toolkit
We provided guidelines and best practices to improve the way we work, to have impactful product development in the Marketplaces.
Below is a list with popular and most used processes and best practices.
Experimentation & Research Toolkit
• Usability lab testing guide and template
• Experiment design session
• How to Write a Solid A/B Test Hypothesis
• Hypothesis guide template for A/B test
• Best Practices doing AB tests
• Template Experiment Design
UX Best Practice Toolkit
• Advanced research - a learning lab course
• Icebreakers & Energizers
• Information Visualisation
• PIL Framework
• Ideation techniques
• Personas methodology
• Self Service framework
Phase 3: Experimentation & Research Repository
Problem
All research and product experimentation remained at local level, meaning that many Marketplaces were solving the same user problems, thereby duplicating effort and resources. There was a need and willingness to learn from others before tackling a problem, but it was extremely difficult to track down insights and learnings.
Solution
It was clear we needed a common repository to document all research. This would enable us to quickly share learnings for others to build on, or bing others up to speed. Taking into account that we were a new company (off-spin from Schibsted Product & Tech), who had already solved for this before the split. We already knew of the successes achievable, so we set out to build an improved version. Following common product design methodology we designed, built and tested a Beta version for 3 months with some hero companies to get feedback. We had a soft launch to test adoption first, before launching 100% in Q3 2019.
Results and learnings
Qualitative research definitely indicated a need for this product. Once a few Beta testers started to use the tool and give feedback, news soon spread around. Some were very skeptical, as it still seemed like reporting and a duplication of effort - a good thing we did a ‘soft launch’ first.
The next problem was the need to gain adoption and for each Marketplace to see the value this tool could bring longterm. But, the tool was only ever going to be as effective depending on how much research was on the site. We tackled this by getting 5-6 eager Marketplaces to upload all their work at the end of every week, and by the end of summer 2019 we had well over 140 bits of research content. Word soon spread and all Product Managers globally saw and understood the benefits. Contributing to the content of the repository soon become part of each Marketplaces’ KPIs.
Phase 4: Research & Experimentation Guild of experts
Goal, purpose and need for the Guild
• Provide the best support working together on cross-functional projects
• Be the GO-TO for consultation, while building trust & confidence with the Marketplaces
• Information sharing between teams in a transparent and timely manner on projects, results and process development
• Avoid duplication of efforts, plus ambitions are channeled in the same direction
• Alignment to speak with one voice as Adevinta to avoid confusion (from a central team of experts)
UX Strategy: helping the Marketplaces improve their craft to scale experimentation, thereby minimising business risks.
• Upskilling intro pack ‘self-help resources’: glossary, onboarding intro, best practices
• Experimentation Programs: training agendas depending on the audience
• Marketplace’s 1:1 learn-by-doing consultations: Q&A, identify user problems, liquidity tree, ideas prioritization, usability testing set up, A/B testing, insights & tech, etc
• Sharing impact: build a Research & Experimentation repository, meet-up sessions to learn from one another
• Adevinta Experimentation Maturity: UX Self Assessment, Experimentation Toolkit & UX best practices
Below a vision and eco-system of connectivity for learning
Phase 4: Experimentation maturity assessment
Analysis to understand the experimentation maturity within the marketplaces.
By having a view of where each team is positioned, we were best placed to advise on providing the right support and at the right time (e.g. self-help materials, or 1:1 training by doing).
We used a 5-scale phase model:
Initial, Adopted, Managed, Integrated, Driven
Initial (how this level looks)
Experimentation is driven by front-line staff, who start to understand some pain points through a informal process (or no process).
Might not have the confidence to implement anything concrete, and roadmap is a list of test ideas (or "gut-feel").
Analytics not used.
Adopted (how this level looks)
Management is aware of experimentation and it's impact, but it is primarily driven by specialised resources (PM/UX).
Roadmap includes hypotheses and prioritization criteria, and experimentation is part of backlog planning
Is using limited data analytics or past experimentation results for inputs.
There is understanding of some basic tools and processes.
Using standardized inputs from qual/quant research on a regular basis.
Managed (how this level looks)
Management actively drives experimentation and views result prior to decision making.
Roadmap includes previous criteria and is directly linked to organizational strategic goals.
There is a standardised process across several teams and/or departments.
Integrated (how this level looks)
Experimentation is at the VP-level and results make it to the executive level.
The entire company is actively involved in experimentation and publicly discuss the value of experimentation.
Driven (how this level looks)
Experimentation is part of their DNA
Stage 5: Awareness & training
1:1 training ‘Learn-by-doing’
Using ‘learning-by-doing’ methodologies, we worked with individual companies that were trying to improve their craft of experimentation. The time-frame was based on a need-by-need basis, ranging from a few short consultations to a 1-2 month intensive collaboration. We always tried to work with a cross-functional local team that consisted of UX, Product, Data and Engineering, as experimentation cannot happen properly if one of those is missing.
For alignment we created a working agreement - Goals (what & why) - UX, PO, Insights, MPC- How do we measure success?- What we can expect and how we collaborate- What are we going to do- draft plan in weeks
To kick off the engagement we would do a Experimentation Maturity Assessment to understand the level of training needed. It was also very important to work with a real user problem connected to real product metrics, and added value by assisting and overviewing quantitative (user research) and qualitative activities (A/B testing). The process included planning, executing, analysis, and iterations upon success.
How do we measure impact
Improved process should lead to:
increased #experiments: faster delivery of results / improve agile rhythm
increased successful #experiments: less insignificant experiments / improved work quality
better strategic decisions: more accurate insights / less assumptions and greater confidence
Working on a ‘real user problem’ from concept to implementation leads to:
measure real impact on the North Star metric (matchmaking sellers to buyers)
Improved user satisfaction: decrease in customer support questions
Kirkpatrick model to understand the impact of the training
Awareness workshops
Firstly we sent out at survey to all those in charge of experimentation in each Marketplace to ask what areas would they like to know more about or receive some training. We then created some awareness workshops, and tailored the content depending on the audience (designers vs data scientists).
UX Offsites: UX Morning Show + Experimentation workshops
Bringing experimentation awareness - workshops and the exciting NEW ‘UX Morning Show’
I tried to bring a little awareness with a bit of interactive fun. In 2019 we had 2x UX Global offsites, which was attended by all Head of UX from around the globe. The 1st was held in Morocco (Marrakesh), the 2nd in Italy (Milan). As Global Experimentation lead I created a 45min chat-host ‘UX Morning Show’, hosted by the ‘one-and-only’, Mr Henry Redman - which was fun and sparked a lot of interest. The content was broken into 3 parts, first a discussion to understand a Marketplaces’ experimentation methods, practices, successes and learnings, secondly, some insight sharing through a real live product demo, and what areas they would like to improve on.Facilitated an Experimentation workshop, dividing all the Head of UX into 2 groups. The goal for the 2 hours exercise was to pick and top user problem and define the experiment, from creating an hypothesis to coming up with the implementation plan.
