The Power of Remote Usability Testing
Instead of dwelling on it, embrace it.
So, I’ll be the first to admit that perhaps my transition to remote usability testing was somewhat smoother than most, solely due to my limited experience of in-person usability testing beforehand. However, I can empathise with both the unfamiliarity that it brings and initial feelings of uncertainty that comes with it, which were present during my most recent project. However, there are undeniable benefits to remote usability testing, and just because it might throw up some challenges at first, doesn’t mean its capabilities and potential should be underestimated. During this post, I will discuss my recent round of remote usability testing and how it was conducted, as well as reflect on the benefits of this method of testing!
The Project
The first module that I have studied as part of my MA in Experience Design centered around assessing and designing for accessibility and usability- fundamental aspects of both digital and physical design that are often overlooked. The report investigated the usability and accessibility of an online retailer’s website, with attention focused directly towards its service as an e-commerce retailer.
Amongst the series of evaluative methods used, including a psychological and Heuristic evaluation, the tests that required remote data gathering centered around performance and satisfaction.
Performance
Prior to the redesign, a performance evaluation needed to be carried out on the existing website in order to identify elements that required improvement. It was measured through a series of metrics, including Error rate, task success, and lostness. I then statistically tested the results against its four primary competitors in order to identify where the website is under-performing.
A/B tests were then used to test the redesigned website against the existing website, in order to evaluate if the changes had an impact on the user’s performance. It was important for the redesign to address the weakness, as poor design and increased lostness indicate the user might be confused, this could cause the user to abandon their website and instead purchase from elsewhere.
Method:
During each of the sessions, the participant was given three tasks to complete. The tasks were selected as they incorporate the Top Tasks that were identified previously, allowing performance metrics from the most important elements of the website to be gathered. A Kolmogorov-Smirnov test did not return a significant value (p.65753), and due to the groups not containing the same participants, an independent t-test was carried out. Due to the relatively small sample size (<20), the achieved power of each test was calculated to determine if there was a need for a non-parametric test to be conducted.
How did Remote Testing Impact this?
There is no denying that the tests took a little bit more planning and organization than their real-world counterparts would; setting up screen recording software and ironing out any technical issues takes a bit of forward-thinking and preparation. However, I believe that, for this scenario, it was the only disadvantage. Technical hurdles aside, the time efficiency and access to participants significantly out-weigh the required investment in planning for unforseen circumstances.
Satisfaction
Aside from being accessible, products also need to leave the user feeling satisfied. Satisfaction and Usability go hand-in-hand, with one being directly influenced by the other; emotions are directly linked to usability, and positive emotions can leave the user with a positive user experience. Not only do they affect the user’s experience whilst using it, but they also affect their reflective memory meaning a user’s satisfaction determines whether they would return to the product.
Methods:
Quantitative methods allow for definite conclusions to be made which can be tracked over time and directly compared to competitors. System Usability Scale (SUS) was selected as the quantitative method as the websites tested are live and working. However, they don’t offer insights into the user's thoughts and emotions, or the problems on the page. A ‘Think Aloud’ method was used to gather this information and identify usability issues.
Data Collection and Analysis Methods:
SUS-
So, this is where remote testing is a necessity. Recruiting 75 participants to complete an SUS survey for the website and its four competitors, in person, is somewhat unrealistic on a tight time-scale. Whilst far from impossible, it is not time efficient, nor is it at all necessary. What remote testing allows for, that in-person methods do not so easily, is the ability for participants to complete the activity within context, allowing for their behavior to be appropriately influenced by their environment.
Think-aloud-
This is where remote usability testing begins to have its downfalls, but not enough to outshine its merits.
Due to being a qualitative method, a far smaller sample size was required (just six in this scenario). The sessions were conducted using the rather unreliable Microsoft Teams, where the user was asked to share their screen and have their microphone and camera turned on. They were then asked to complete three tasks, and, most importantly, vocalise what they are feeling, thinking, and doing.
So, what are the downfalls? Well, I believe it comes to the difficultly of creating a meaningful connection to the participants. Empathy is an undeniably powerful tool, and vital when conducting any form of research and testing. Whilst video calls and collaborative workspaces attempt to address this issue, they are unable to replace it.
What have I learned for next time?
Preparation is Key-
Whilst all the necessary equipment was prepared, consent forms signed and process explained, I did overlook one key element of remote testing on two occasions… technical issues. Ah… the dreaded pixelated feed… not ideal when analysing performance data. Whilst seemingly a minor mistake, it certainly disrupted workflow and must be addressed.
Use a backup!
…or just don’t use the inbuilt record function on Teams. Whilst it appeared to work seamlessly on most occasions, it failed to save the recording of my second Think-Aloud activity during the A/B testing… something I didn’t realise until ending the session. I’d suggest using Camtasia or similar to have more control over the recording.
