In recent articles we have discussed the role of Generative and Evaluative research in experience design. Whilst the focus of those articles has largely related to qualitative research, in this article I want to explore the use of quantitative data in both Generative and Evaluative research. This article covers:
- What is quantitative data?
- Quantitative data can remove subjectivity from decision-making
- Statistical significance carries the day
- Matching behaviour and attitudes to customer data
What is quantitative data?
When we talk about quantitative data, or research methods, we are typically referring to data that is numerical in nature that allows us to quantify attitudes, behaviours and opinions, and more. The number of respondents is large enough to allow statistical reliability and to relate the results of the research back to the general population, or the population that is of interest.
Taking the subjectivity out of decision-making
Quantitative data can deliver objectivity to decision-making processes that are more often subjective in nature. One of our clients, a major Australian university, was undergoing a digital transformation, which also tied in with a brand refresh. A key aspect of the transformation was a website redesign. They had been provided with three design treatments to choose from, and normally, the decision on which treatment to proceed with would sit with some key stakeholders based on internal discussion.
However, using an online survey with over 600 respondents, together we were able to identify which treatment best represented each of their key brand attributes. This process yielded a very clear “winner”, whilst also teasing out which elements of the alternative treatments should be incorporated within the final design to deliver the best representation of their brand.
Quantitative data can help to make decision-making more objective.
The use of quantitative data in this context removed individual preference or subjectivity from the discussion and provided an unambiguous path forward, something that was highly valued by the stakeholders. Through establishing key criteria that can be used to assess the suitability of a design treatment, the team was able to make a decision with ease, and had a frame of reference for any future changes to the design of their website. Any future changes can be assessed against these brand attributes before implementing any change.
Statistical significance tells the story
Applying statistical significance testing to different approaches to interaction design can provide an unequivocal understanding of which approach to implement. We collaborated on such a project with a client operating in the financial services sector who needed assistance in choosing between two different interaction schemes for completing an online banking transaction.
Having enough responses to test for statistical significance can help identify clear winners.
We designed an online usability test that allowed participants to experience both interaction design versions, with the order of presentation varied across participants to reduce potential for bias. With over two hundred participants it was possible to test for significant differences in task completion and attributes such as preference and perceived ease of use. A statistically significant difference was identified allowing the team to move forward with confidence and bypass any potential contention amongst internal stakeholders regarding the selected interaction design.
Matching behaviour and attitudes to customer data
Organisations regularly use customer segmentation to assist them in understanding and communicating with their customers. These segments are often created via the identification of “clusters” of similar people based on a combination of shared demographics and characteristics. The issue we often find with these segments is that they lack the strategic differentiation required for making experience design decisions. For instance, we are specifically interested in the variants in digital behaviour and preferences when designing digital solutions and clients are interested in closing this gap.
Using strategic segmentation can help with product design decisions.
We worked with a client to achieve this outcome by running a large-scale survey with their customers that explored behaviours and attitudes related to use of digital. Through statistical analysis of the responses we were able to create a model that allowed our client to relate specific behaviours and attitudes to product holdings. We created “strategic segments” to aid them in experience design decisions, with the added benefit of also being able to tie these “strategic segments” back to their existing customer data.
Summary
The article explores examples of using quantitative data to support generative research. It was not particularly clear what the outcomes or predictive model would actually look like until we conducted an open exploration of the data and relationships between variables to reach the final model. For some, the application of quantitative approaches to generative research is not intuitive but it is an important, and often overlooked, tool in the experience design process, achieving outcomes not possible with qualitative techniques.
The preceding examples represent evaluative research where weight of numbers and statistical significance provided an indisputable path forward, removing ambiguity and subjectivity from decision-making. Whether it is qualitative or quantitative research, making informed evidence-based decisions, rather than relying on instinct or the highest paid opinion provides a strong foundation for success of your experience design.