Leveraging AI to gain insight from in-the-moment user feedback to inform decision making
How we helped an executive leadership team within an IT and Operations division elicit meaningful user feedback that would deliver the actionable data and insights they sought.
About our client
Our client is the IT and Operations division of a leading global financial services company. The Group services 14,000 customers spread across six global locations and is responsible for all systems that underpin the bank’s daily operations.
A bi-annual survey to 1,400 customers yielded feedback that was hard to digest and action. The executive leadership team needed access to insight consistently, enabling them to prioritise effectively, make informed decisions to serve their users better, and reduce risk and cost to the business.
- The feedback survey had a low response rate of 15%, highly qualitative responses, and poor insights
- Existing survey results was scattered across various channels restricting accessibility
Based on previous hypotheses, the divisions executive leadership team identified that a digital solution that enabled continuous conversations with users to elicit meaningful feedback would deliver the actionable data and the insights they were seeking.
Hypothesis was engaged to design and build the end-to-end solution within a tight timeframe of seven weeks.
A Hypothesis team including a Product Designer (UX / UI), two Full Stack Engineers, and a Product Manager was formed, and the engagement was initiated with a five–day Design Sprint with key stakeholders to create rapid alignment on:
- Product Vision
- Lean Canvas
- Product Roadmap
- The first set of experiments / hypotheses to be tested
- Guiding principles to base product decisions
- Software architecture i.e., how we will build the digital platform
With the earlier defined hypotheses, we were able to develop an end-to-end working prototype ready for user testing within 1.5 weeks. In parallel, our Engineers built a base platform to enable further experimentation.
One of the experiments included introducing key measures, which soon became the north star of the product, i.e. if we cannot measure it is difficult to demonstrate improvement. As no detailed brief was given other than wanting meaningful feedback, collecting and showing insights though interdependent, were separate things. And unlike traditional feedback surveys, we needed the ability to measure the insights given.
By following Human Centred Design and Lean Start-up methodologies, our Product Designers delivered a productionised digital product with active users within 5 weeks that showed measurable progress through continuous user feedback, in the form of insights for the executive leadership team.
The product captured feedback in a seamless and frictionless way. At times it was even assisting people in having structured conversations. With being a first of its kind in the Bank, the solution quickly gained attention, and the team was extended to five months to fulfil the next set of objectives.
Objective: Demonstrate the desirability of the product and surface meaningful insights.
a) Key result: There are at least 30 active users in 6-7 weeks post launch.
b) Key result: At least 3-5 key stakeholders find the insights meaningful in 6-7 weeks post launch.
Since the mechanism to capture data was in place (with early adopters), we fast-tracked the next set of experiments, one of which was to use the feedback obtained to deliver insights at scale using AI. Using a mix of existing cloud services and our own bespoke AI models, we could start to determine sentiment, identify themes, and uncover new opportunities in real-time. These insights were added to the reporting and were made easily accessible to the executive leadership team as they lived in one place.
The cross-functional Hypothesis team worked collaboratively to deliver a set of experiments every fortnight while promoting the product to a global audience through demonstrations and online content, driving further adoption.
- Executive Leader
The Executive Leadership team now have access to real-time insights and visual reporting gained from continuous user feedback, enabling better prioritisation and decision making to reduce operational risks and cost.
The team managed to deliver the end to end working product within 5 weeks with a vision of creating a data led culture in the organisation through a self-serve platform.
What started as a 7-week engagement to test the desirability of the product, was extended to a further five months to build out the roadmap and stickiness. Our test group of 5 teams quickly grew to 100+ teams using the product, with a response rate of 60-70% within 3 months.
Key success measures
By embedding measurement in the ways of working, we enabled the capture of meaningful data and demonstrated how AI can surface further insights and sentiment that was otherwise lost in emails and spreadsheets.
The base platform Hypothesis built to support the survey component was leveraged six months later, when creating two other bespoke apps for our client, adding further insights to the reporting.