The usability testing of our GUI mockup was conducted using a remote questionnaire.
Questions were based on previously designed mockups: Homepage, Player Page, Game Page, Community Page, Account Settings and Search Bar.
Each question presents a task that needs to be completed using the mockup. Details of questions are outlined in section 2.1 Methodology.
The questionnaire consisted of 19 questions in total. Types of questions are summarised below:
- 7 'difficulty rating' type questions (See section 2.1 for details)
- 9 'correct or incorrect answer' type questions (See section 2.1 for details)
- 1 rating question on the overall user experience
- 1 short-answer question asking participants to choose 3 adjectives to describe the overall experience
- 1 long-answer question on areas of improvement
Link to questionnaire: https://docs.google.com/forms/d/e/1FAIpQLSeK5sfJahLyPbnxdwrTYTYz19VKZjQsS4a3gs72k6OT7zJg-Q/viewform
Overall there was a limited deviation from the original usability test plan. The number of participants was less than expected which has the potential to affect the quantitative results of the questionnaire however it unlikely that this effect will be sufficiently detrimental to invalidate the amassed results.
The gender distribution of participants is believed to have remained consistent with that expected in the test plan, as the questionnaire was distributed in groups of approximately similar numbers of male and female users as was mentioned in the usability test plan.
Finally, the analysis performed on the results of the usability questionnaire was conducted as outlined in the usability test plan, specifically statistical methods were applied to quantitative data and qualitative data was interpreted via inspection and the application of a word cloud.
The questionnaire received 20 responses. Results were analysed either quantitatively or qualitatively depending on the type of question. 'Difficulty rating' and 'correct/incorrect answer' questions were analysed with quantitative approach, while the rest were evaluated qualitatively.
7 'difficulty rating' and 9 'correct/incorrect' questions were analysed based on the success and failure criteria outlined in section 2.1.1. For each question (16 questions in total), 20 responses were categorised as either 'success' or 'failure'. Subsequently, success and failure counts were summed and are presented as percentages below. On the whole, participants were able to complete tasks successfully 92% of the time. This high success rate indicates that our GUI mockup satisfies the three usability testing goals outlined in section 2.
At least one 'difficulty rating' question was asked on each GUI mockup. Similar to above, 20 responses on 7 questions were identified as either success or failure. The counts were summed and are presented as percentages in the pie chart below. The successful task completion rate was 95%, suggesting that almost all participants found our GUI mockup easy to use and navigate.
The responses were further broken down into individual ratings to obtain a distribution of responses. As seen in the bar chart below, most participants selected rating 0, which corresponds to 'very easy'. Majority of participants selected ratings 0, 1 and 2, with few selecting options 3, 4 and 5, indicating that most participants found tasks 'very easy' to complete with our GUI mockup.
At least one 'correct/incorrect answer' question was asked on each GUI mockup. Once again, 20 responses on 9 questions were grouped as either success or failure. These counts were summed and are presented as percentages in the pie chart below. Some of these questions were aimed at determining whether core functionalities were divided into logical units that are obvious to users. On the whole, 90% of participants were able to complete tasks successfully.
Since the success rate was lower here when compared to 'difficulty rating' questions, success and failure counts were broken down into individual questions. On the whole, success counts were much greater than failure counts in all 9 questions. However, the last two questions, Q14 and Q15, appear to have higher failure rate than other questions. Q14 and Q15 were questions aimed to test whether core functionalities were divided into logical units that are obvious to users. For example, Q14 was 'From Account Settings, if you wanted to see your collection of games, where would you click?'. Lower success rates on these questions suggest that the division of functionalities into units is not logical and obvious to some users. Perhaps, having separate options on navigation bar instead of organising into subunits using dropdown options is a better approach.
In total there were two qualitative questions in the usability testing questionnaire, Q18 and Q19.
These questions consisted of:
- Asking the participant to choose 3 adjectives to describe their overall experience with the system.
- Asking if the participant found something difficult or if they had any improvement suggestions.
In Q18 participants provided a number of responses which were collated into a word cloud as shown below. It can be seen that the most frequent responses were "Clear", "Organised" and "Simple" which fulfils the intentions of the mockup, as the system was designed to be above all straightforward and accessible, regardless of computer background.
The main aims of the usability testing were to evaluate three aspects of the system GUI, these include:
- Easy to navigate.
- Easy to complete tasks.
- Functionalities divided into logical units and sub-units.
Given the most frequent responses mentioned above, the aspects of the system GUI which were intended to be evaluated have been found to be fulfilled successfully.
Conversely, the responses which followed those mentioned above in terms of frequency were "Dull" and "Plain". This suggests that despite the spartan design of the system supporting organisation and simplicity it was perceived by participants to be monotone and largely unembellished as can be deduced from the frequency with which participants responded with the adjectives "Dull" and "Plain". However, this may have simply been dependant on the individual tastes of the participants as another prevalent adjective demonstrated in the word cloud below is "Neat", implying that for some the clean and minimal design proved orderly and precise.
Compared with Q18, Q19 was far less likely to be answered, with half of the 20 participants failing to enter a response. A possible explanation is that as the final question of the questionnaire, participants may have become fatigued as the questionnaire was admittedly rather involved. Additionally, participants may not have encountered difficulties they felt worth mentioning or had any improvement suggestions.
Those participants that responded to Q19 had a number of useful suggestions. A common issue experienced by participants was in navigating the system which does bring the success of the navigational aspect of the system into question.
Specifically, participants did not like the fact that they were required to click through the different aspects of the site via the navigational bar at the top of every page.
Another common issue experienced by the participants was in finding their games within the system, not immediately realising that game collections resided in the profile page within the system. This ambiguity in the location of game collections is something that would be addressed in future design decisions.
Finally, the most common complaint of participants was that the system GUI was unattractive and desolate, as discussed above this may have been due to differing tastes of the participants however a more colourful and aesthetic design is likely to have been produced with successive iterations of the system, given that the system displayed to the participants of the questionnaire was merely a mockup of the potential system.
Overall, results indicate that our GUI mockup successfully fulfils the testing goals outlined in section 2. However, some flaws in navigation aspect of the system were raised in qualitative analysis. While producing a system that satisfies all users is not possible due to individual preferences, we decided to place all navigation options on the navigation bar instead of grouping options with dropdown menus. We have also concluded that a help manual would be in place to help users with navigation around the web application.