Was in Jakarta from Nov 1 (Sun) till Nov 7 (Sat).
Had achieved a lot of thorough reading of:
Rubin, Jeffrey. Handbook of Usability Testing.
Somervell, Jacob. Developing Heuristic Evaluation Methods for Large Screen Information Exhibits Based on Critical Parameters. [Dissertation, PhD in Computer Science and Applications]
Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.]
Good benchmark for me to write my chapters 1 and 2.
Through Rubin's book, I have understood more deeply about who, what, when and how of usability testing.
Showing posts with label Baker. Show all posts
Showing posts with label Baker. Show all posts
Sunday, November 8, 2009
Thursday, November 5, 2009
Nov 5,6 - Baker, Evaluation Methodology (MSc thesis)
Heuristic Evaluation of Shared Workspace Groupware
Chapter 4
Evaluation Methodology
Having formulated two sets of groupware heuristics from two inter-related frameworks in Chapter 3, the next logical task in my research is to validate these heuristics. To that extent, this chapter describes the two-step methodology used to carry out this objective.
The first step was a pilot study whereby the groupware heuristics were reviewed and subsequently modified prior to conducting the main research study, the second step. The main research study was set up to mirror the methodology and terminology employed by Nielsen to validate his heuristics (Nielsen and Molich 1990, Nielsen 1992).
For our purposes, two groups of inspectors with varying degrees of expertise in HCI and CSCW evaluated two groupware systems, a toy groupware editor called GroupDraw (Roseman and Greenberg 1994), and a very substantial commercial groupware system called Groove (www.groove.net). Their goal was to record as many usability problems as possible that violated the groupware heuristics.
4.1 Objective
As per my research goals (see Section 1.4), the objective of validating the heuristics is to:
“Demonstrate that the adapted heuristic evaluation for groupware remains a ‘discount’ usability technique by analyzing the ability of inspectors to identify problems in collaborative applications”.
As a means to execute the stated objective, I revisit Nielsen’s motivations for traditional
heuristic evaluation where he designed it as a usability engineering methodology that can
be done cheaply, quickly, and produce useful results (Nielsen and Molich 1990, Mack
and Nielsen 1994).
To briefly elaborate the three key terms:
1. Cheaply.
Nielsen’s heuristic evaluation places a low demand on resources. Nonexperts can carry out inspections; therefore, it is not confined to using more costly experts (the caveat: experts will produce better results) (Nielsen 1992).
This is practical because heuristic evaluation is, in practice, relatively easy to learn and to apply (Mack and Nielsen 1994). Consequently, extensive resources are not required for training. Finally, heuristic evaluation requires only the evaluator, the interface, paper, and pencil. No special equipment or facilities are necessary.
2. Quickly.
Heuristic evaluations do not require advance planning nor do they require a large amount of time to conduct (Nielsen and Molich 1990). Typically, the evaluation of an interface can be done within an hour or two for a simple interface and somewhat longer for more complex interfaces.
3. Useful results.
Despite performing heuristic evaluations with less control and formality than would be entailed by formal user testing, this technique provably produces useful results (Nielsen and Molich 1990, Mack and Nielsen 1994). As discussed in Chapter 2, only a few evaluators (3-5) can discover a majority (~75%) of interface bugs with varying severity. Fixing these bugs can in turn improve the usability of products.
Within my main research study, I look to validate heuristics applied to groupware evaluation to see if it remains a discount usability technique that is quick and cheap to perform while producing useful results.
To gauge this cost-effectiveness, I conducted the evaluation in the following manner:
• I chose as inspectors people with knowledge of human computer interaction, but limited knowledge of Computer Supported Cooperative Work.
• Prior to performing their evaluations, inspectors received a basic one hour training lecture and a packet of written materials (see Appendix A) explaining the heuristics.
• Inspectors self-selected the length of time and amount of effort they would put into completing the evaluation.
4.2 Pilot study
The groupware heuristics used by the inspectors to perform their evaluation of the two systems are presented in Appendix A.5 and A.6. A quick glance over the mechanics of collaboration heuristics reveals that their explanation and format differ from the same heuristics presented in the previous chapter.
Due to time constraints and my secondary focus on the Locales Framework heuristics, the pilot study was conducted with only the mechanics of collaboration heuristics.
4.2.1 Participants
Three professional HCI and groupware practitioners with 5 to 17 years of relevant experience were recruited to review the mechanics of collaboration heuristics. All three were also currently engaged in a project that involved re-designing the user interface for a real-time, shared workspace collaborative application.
4.2.2 Method
Each participant received a copy of the mechanics of collaboration heuristics similar to
what is found in Chapter 3.
Each was asked to address the following two questions:
1. Do I understand the principles of the heuristic?
2. Would I be able to apply this heuristic as part of a heuristic evaluation?
The objective was to gain informal feedback regarding the comprehensibility of each heuristic and its suitability to be applied in the context of an evaluation. Feedback was gathered in the form of written comments on the original handouts and verbal comments recorded from interviews with each individual after their review.
4.2.3 Results
Overall, the reviewers’ feedback was positive regarding the content of each mechanics of collaboration heuristic. They expressed that beginning a heuristic with its supporting theory—derived from studies of face-to-face interactions—helped to establish its motivation. In addition, the reviewers considered the heuristics to be practical since they included examples of techniques employed by groupware systems to comply with each one.
Despite the positive feedback, two main areas of concerns surfaced in response to the reviewers answering the two aforementioned questions.
The first area of concern surrounds the heuristics’ comprehension. Each reviewer had to re-read each heuristic (in some cases, several times) in order to comfortably understand the concepts.
The second area of concern centered on the ability of the reviewers to apply the heuristics as part of a groupware evaluation. ...the reviewers concluded that it would be awkward to use the heuristics in their current format to effectively evaluate groupware systems.
4.2.4 Discussion
Prior to conducting the main research study, the mechanics of collaboration heuristics (and subsequently the Locales Framework heuristics) had to be revised to address the reviewers’ concerns. I did not want the bottleneck to ‘good’ results to be the inability of the inspectors to understand and apply the heuristics.
To address the first concern, all of the heuristics were re-written with the intent of making them an ‘easier read’. Domain specific terms were replaced with more common terms. Sentences were shortened. In addition, all new concepts introduced by the heuristics were clearly defined and spelt out.
The second concern regarding the practicality of the heuristics in their current format raised an interesting issue, one that I had not considered up until this point. It is one thing to have all the pertinent theory encapsulated in a set of heuristics, but it is another to ensure that the heuristics are packaged as a practitioner’s training tool to facilitate conducting a heuristic evaluation.
The naïve approach is to structure these heuristics as a checklist whereby a practitioner is presented with a series of items that can be systematically checked off during an inspection.
In response to the reviewers’ comments, the mechanics of collaboration heuristics were re-structured in an attempt to facilitate their role as a tool.
The first section of each heuristic was divided into two new sections “Theory” and “What this means for groupware”.
“Theory” provides the underlying principles behind each heuristic and is not critical to performing an evaluation.
The intent of the next section, “What this means for groupware”, is to provide all the pertinent information that an inspector should consult when performing a heuristic evaluation.
The final section “Techniques used in groupware” remained essentially intact with the “Typical groupware support” section from the early version of the heuristics.
4.2.5 Sanity check
To ensure that all comments had been adequately addressed, the same three professionals reviewed the revised mechanics of collaboration heuristics. All were comfortable that their comments had been addressed. They viewed the new heuristics as easier to read and understand. This set of heuristics and its training material is presented in Appendix A.5.
4.2.6 Locales Framework heuristics
Although the pilot study was conducted with the mechanics of collaboration heuristics, the findings were transferable to the Locales Framework heuristics. Consequently, the latter were re-written to address the issues in a similar manner to the mechanics of collaboration heuristics.
The only difference between the two sets of heuristics is the lack of a “Techniques used in groupware” section with the locales heuristics.
4.3 Main research study
Subsequent to revising both sets of heuristics in response to the pilot study findings, my next step was to see if these adapted heuristics could be used for “heuristic evaluation of groupware and remain a ‘discount’ usability technique”.
To do this, I analyze the ability of inspectors to identify problems in two collaborative applications.
4.3.1 Participants
To assess the practicality of the groupware heuristics, I looked at the ability of individuals with minimal training in HCI but with varying levels of knowledge in CSCW to apply them. We recruited several groups fitting these criteria and validated our demographics through a questionnaire.
Participants were categorized as two evaluator types: novice and regular.
• Novice evaluators were 16 students in their 3rd or 4th year of a University computer science program. All had completed one full course in HCI and were currently enrolled in a second senior-level advanced undergraduate HCI course. When asked, the majority indicated some experience with designing and evaluating graphical user interfaces. However, few had any substantive knowledge regarding CSCW interface design principles. Consequently, the group consisted of “novices” with respect to CSCW usability but not with respect to computers and HCI.
• Regular specialists were 2 professors and 9 students working on their graduate degrees in computer science. All had a history of research, applied work, and/or class work in groupware and CSCW, as well as conventional user interface design and evaluation. These individuals were labeled ‘regular specialists’ since in contrast to the former group; they were knowledgeable of groupware fundamentals.
Except for the professors, all participants were students. Due to their limited availability, professional HCI/CSCW practitioners from industry were not employed as part of our research.
4.3.2 Materials
To help inspectors conduct their heuristic evaluation, we gave them a training packet, workstations, and the two groupware systems.
Training packet.
The training packet (Appendix A) consisted of two sets of groupware heuristics; one based on the mechanics of collaboration (A.5) and the other on the Locales Framework (A.6). These were revised versions in accordance with the pilot study findings.
The packet also contains the following forms:
• a consent form outlining the inspectors’ participation in the research study (A.1);
• a background information questionnaire to help ascertain each inspector’s knowledge and experience in the areas of HCI and CSCW (A.2);
• pre- and post-training feedback forms containing questions assessing the ease with which the inspectors were able to comprehend the heuristics before and after their training session (A.4); and
• problem reports designed in accordance with Nielsen (1994a) and Cox (1998) and used by the inspectors to capture their usability problems (A.7).
Workstations.
The novice evaluators conducted the heuristic evaluations of the two groupware systems on four PC workstations located in a single row in the undergraduate computer lab.
...Installation instructions (Appendix A.3) were provided on how to set-up the software.
Groupware systems.
As part of the study, the participants evaluated two quite different
shared visual workspaces contained in two real-time groupware systems: GroupDraw
and Groove.
GroupDraw is an object-oriented ‘toy’ drawing program built to show people
how to program the GroupKit groupware toolkit (Roseman and Greenberg 1996).
Groove is a virtual space for real-time, small group interactions. Users create “shared spaces” to communicate and collaborate with one another. Changes made to a shared space by one participant are automatically synchronized with all other computers.
Its functionality includes:
1. Communication tools – live voice over the Internet, instant messaging, text-based chat, and threaded discussion.
2. Content sharing tools – shared files, pictures, and contacts.
3. Joint activity tools – co-Web browsing, multiple-user drawing and editing, group calendar.
4.3.3 Method
The heuristic evaluation of the GroupDraw and Groove interfaces followed Nielsen’s standard recommendations (refer to chapter 2 for details).
This involved administering an orientation session to each group of inspectors prior to the evaluation process.
However, I did not conduct a debriefing session due to the geographical separation between the
inspectors and the researcher (myself).
Orientation session.
Prior to the session, each participant was asked to:
• sign the consent form (A.1);
• fill-out the background information questionnaire (A.2);
• read the detailed written description of the groupware heuristics (A.5 and A.6); and
• complete the pre-training feedback form (A.4).
Given the number of inspectors (27 total inspectors) and their location (22 in Calgary and 5 in Saskatoon), I conducted three separate 90-minute orientation sessions to three audiences.
* Collected the signed consent form, the completed background questionnaire and pretraining
feedback form from each inspector.
* Inspectors were handed a blank post-training feedback form (this form was identical to the pre-training feedback form) and an ample supply of blank problem reports for their heuristic evaluation of the systems.
* Conducted a one-hour training session on the proposed groupware heuristics. This included a review of the theory supporting each heuristics, how to apply them during an evaluation, and real-time groupware examples that illustrated compliance and non-compliance with each heuristic.
* Participants then filled out a post-training feedback form so that I could gauge how well they comprehended the groupware heuristics upon receiving the training.
* Provided the inspectors with an overview of the two groupware systems under test, GroupDraw and Groove.
* For GroupDraw, the inspectors were asked to evaluate only the shared workspace (Figure 4.1 bottom) and the Notes functionality (Figure 4.2) as the means for communicating with one another. For Groove, I asked the inspectors to evaluate the Outliner tool (Figure 4.3) as well as the text chat and audio link.
* General instructions were given regarding the process for conducting the heuristic evaluation. The novices were to inspect both systems with only the mechanics of collaboration heuristics. The regular specialists were to assess the groupware interfaces with both the mechanics of collaboration and Locales Framework heuristics.
Evaluation process.
Each inspector dictated when, where, and how to perform the evaluation. As with traditional heuristic evaluation, they could use the heuristics to systematically review all the functionality. Alternatively, they could walk through an imaginary task of their own making and verify how each step of the task complied with the heuristics.
Inspectors had complete control over the length of time and amount of effort they for completing the evaluation.
In some instances, the evaluation was performed in pairs and other times it involved four or five inspectors working concurrently.
4.3.4 Data collection.
For each problem uncovered, the inspectors completed a separate problem report by recording a description of the problem, the violated heuristic, a severity rating, and an (optional) solution to the problem. They judged a ‘major’ severity rating as one that represented a significant obstacle to effective collaboration, while a ‘minor’ rating is one that could be worked around by the participant. A blank problem report is found in Appendix A.7.
With respect to formulating severity ratings for each usability problem, Nielsen (1994a) states that evaluators have difficulty performing this step during the evaluation process since they are more focused on finding new usability problems. In addition, each evaluator will not find all the usability problems in the system; therefore, the severity ratings will be incomplete since they only reflect those problems found by the evaluator.
These original problem reports form the raw data for my analysis in the next chapter (see Appendix B for all problem reports).
4.4 Conclusion
This chapter reviewed the methodology for the pilot study and the subsequent changes to the groupware heuristics in preparation for the main study. Next, the main research study was introduced via a detailed description of its methodology. Of primary importance is that we used people that we felt were reasonable approximations of the actual practitioners we would expect to do groupware evaluations, and that our methodology echoed Nielsen’s traditional heuristic evaluation methodology.
Source: Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Chapter 4
Evaluation Methodology
Having formulated two sets of groupware heuristics from two inter-related frameworks in Chapter 3, the next logical task in my research is to validate these heuristics. To that extent, this chapter describes the two-step methodology used to carry out this objective.
The first step was a pilot study whereby the groupware heuristics were reviewed and subsequently modified prior to conducting the main research study, the second step. The main research study was set up to mirror the methodology and terminology employed by Nielsen to validate his heuristics (Nielsen and Molich 1990, Nielsen 1992).
For our purposes, two groups of inspectors with varying degrees of expertise in HCI and CSCW evaluated two groupware systems, a toy groupware editor called GroupDraw (Roseman and Greenberg 1994), and a very substantial commercial groupware system called Groove (www.groove.net). Their goal was to record as many usability problems as possible that violated the groupware heuristics.
4.1 Objective
As per my research goals (see Section 1.4), the objective of validating the heuristics is to:
“Demonstrate that the adapted heuristic evaluation for groupware remains a ‘discount’ usability technique by analyzing the ability of inspectors to identify problems in collaborative applications”.
As a means to execute the stated objective, I revisit Nielsen’s motivations for traditional
heuristic evaluation where he designed it as a usability engineering methodology that can
be done cheaply, quickly, and produce useful results (Nielsen and Molich 1990, Mack
and Nielsen 1994).
To briefly elaborate the three key terms:
1. Cheaply.
Nielsen’s heuristic evaluation places a low demand on resources. Nonexperts can carry out inspections; therefore, it is not confined to using more costly experts (the caveat: experts will produce better results) (Nielsen 1992).
This is practical because heuristic evaluation is, in practice, relatively easy to learn and to apply (Mack and Nielsen 1994). Consequently, extensive resources are not required for training. Finally, heuristic evaluation requires only the evaluator, the interface, paper, and pencil. No special equipment or facilities are necessary.
2. Quickly.
Heuristic evaluations do not require advance planning nor do they require a large amount of time to conduct (Nielsen and Molich 1990). Typically, the evaluation of an interface can be done within an hour or two for a simple interface and somewhat longer for more complex interfaces.
3. Useful results.
Despite performing heuristic evaluations with less control and formality than would be entailed by formal user testing, this technique provably produces useful results (Nielsen and Molich 1990, Mack and Nielsen 1994). As discussed in Chapter 2, only a few evaluators (3-5) can discover a majority (~75%) of interface bugs with varying severity. Fixing these bugs can in turn improve the usability of products.
Within my main research study, I look to validate heuristics applied to groupware evaluation to see if it remains a discount usability technique that is quick and cheap to perform while producing useful results.
To gauge this cost-effectiveness, I conducted the evaluation in the following manner:
• I chose as inspectors people with knowledge of human computer interaction, but limited knowledge of Computer Supported Cooperative Work.
• Prior to performing their evaluations, inspectors received a basic one hour training lecture and a packet of written materials (see Appendix A) explaining the heuristics.
• Inspectors self-selected the length of time and amount of effort they would put into completing the evaluation.
4.2 Pilot study
The groupware heuristics used by the inspectors to perform their evaluation of the two systems are presented in Appendix A.5 and A.6. A quick glance over the mechanics of collaboration heuristics reveals that their explanation and format differ from the same heuristics presented in the previous chapter.
Due to time constraints and my secondary focus on the Locales Framework heuristics, the pilot study was conducted with only the mechanics of collaboration heuristics.
4.2.1 Participants
Three professional HCI and groupware practitioners with 5 to 17 years of relevant experience were recruited to review the mechanics of collaboration heuristics. All three were also currently engaged in a project that involved re-designing the user interface for a real-time, shared workspace collaborative application.
4.2.2 Method
Each participant received a copy of the mechanics of collaboration heuristics similar to
what is found in Chapter 3.
Each was asked to address the following two questions:
1. Do I understand the principles of the heuristic?
2. Would I be able to apply this heuristic as part of a heuristic evaluation?
The objective was to gain informal feedback regarding the comprehensibility of each heuristic and its suitability to be applied in the context of an evaluation. Feedback was gathered in the form of written comments on the original handouts and verbal comments recorded from interviews with each individual after their review.
4.2.3 Results
Overall, the reviewers’ feedback was positive regarding the content of each mechanics of collaboration heuristic. They expressed that beginning a heuristic with its supporting theory—derived from studies of face-to-face interactions—helped to establish its motivation. In addition, the reviewers considered the heuristics to be practical since they included examples of techniques employed by groupware systems to comply with each one.
Despite the positive feedback, two main areas of concerns surfaced in response to the reviewers answering the two aforementioned questions.
The first area of concern surrounds the heuristics’ comprehension. Each reviewer had to re-read each heuristic (in some cases, several times) in order to comfortably understand the concepts.
The second area of concern centered on the ability of the reviewers to apply the heuristics as part of a groupware evaluation. ...the reviewers concluded that it would be awkward to use the heuristics in their current format to effectively evaluate groupware systems.
4.2.4 Discussion
Prior to conducting the main research study, the mechanics of collaboration heuristics (and subsequently the Locales Framework heuristics) had to be revised to address the reviewers’ concerns. I did not want the bottleneck to ‘good’ results to be the inability of the inspectors to understand and apply the heuristics.
To address the first concern, all of the heuristics were re-written with the intent of making them an ‘easier read’. Domain specific terms were replaced with more common terms. Sentences were shortened. In addition, all new concepts introduced by the heuristics were clearly defined and spelt out.
The second concern regarding the practicality of the heuristics in their current format raised an interesting issue, one that I had not considered up until this point. It is one thing to have all the pertinent theory encapsulated in a set of heuristics, but it is another to ensure that the heuristics are packaged as a practitioner’s training tool to facilitate conducting a heuristic evaluation.
The naïve approach is to structure these heuristics as a checklist whereby a practitioner is presented with a series of items that can be systematically checked off during an inspection.
In response to the reviewers’ comments, the mechanics of collaboration heuristics were re-structured in an attempt to facilitate their role as a tool.
The first section of each heuristic was divided into two new sections “Theory” and “What this means for groupware”.
“Theory” provides the underlying principles behind each heuristic and is not critical to performing an evaluation.
The intent of the next section, “What this means for groupware”, is to provide all the pertinent information that an inspector should consult when performing a heuristic evaluation.
The final section “Techniques used in groupware” remained essentially intact with the “Typical groupware support” section from the early version of the heuristics.
4.2.5 Sanity check
To ensure that all comments had been adequately addressed, the same three professionals reviewed the revised mechanics of collaboration heuristics. All were comfortable that their comments had been addressed. They viewed the new heuristics as easier to read and understand. This set of heuristics and its training material is presented in Appendix A.5.
4.2.6 Locales Framework heuristics
Although the pilot study was conducted with the mechanics of collaboration heuristics, the findings were transferable to the Locales Framework heuristics. Consequently, the latter were re-written to address the issues in a similar manner to the mechanics of collaboration heuristics.
The only difference between the two sets of heuristics is the lack of a “Techniques used in groupware” section with the locales heuristics.
4.3 Main research study
Subsequent to revising both sets of heuristics in response to the pilot study findings, my next step was to see if these adapted heuristics could be used for “heuristic evaluation of groupware and remain a ‘discount’ usability technique”.
To do this, I analyze the ability of inspectors to identify problems in two collaborative applications.
4.3.1 Participants
To assess the practicality of the groupware heuristics, I looked at the ability of individuals with minimal training in HCI but with varying levels of knowledge in CSCW to apply them. We recruited several groups fitting these criteria and validated our demographics through a questionnaire.
Participants were categorized as two evaluator types: novice and regular.
• Novice evaluators were 16 students in their 3rd or 4th year of a University computer science program. All had completed one full course in HCI and were currently enrolled in a second senior-level advanced undergraduate HCI course. When asked, the majority indicated some experience with designing and evaluating graphical user interfaces. However, few had any substantive knowledge regarding CSCW interface design principles. Consequently, the group consisted of “novices” with respect to CSCW usability but not with respect to computers and HCI.
• Regular specialists were 2 professors and 9 students working on their graduate degrees in computer science. All had a history of research, applied work, and/or class work in groupware and CSCW, as well as conventional user interface design and evaluation. These individuals were labeled ‘regular specialists’ since in contrast to the former group; they were knowledgeable of groupware fundamentals.
Except for the professors, all participants were students. Due to their limited availability, professional HCI/CSCW practitioners from industry were not employed as part of our research.
4.3.2 Materials
To help inspectors conduct their heuristic evaluation, we gave them a training packet, workstations, and the two groupware systems.
Training packet.
The training packet (Appendix A) consisted of two sets of groupware heuristics; one based on the mechanics of collaboration (A.5) and the other on the Locales Framework (A.6). These were revised versions in accordance with the pilot study findings.
The packet also contains the following forms:
• a consent form outlining the inspectors’ participation in the research study (A.1);
• a background information questionnaire to help ascertain each inspector’s knowledge and experience in the areas of HCI and CSCW (A.2);
• pre- and post-training feedback forms containing questions assessing the ease with which the inspectors were able to comprehend the heuristics before and after their training session (A.4); and
• problem reports designed in accordance with Nielsen (1994a) and Cox (1998) and used by the inspectors to capture their usability problems (A.7).
Workstations.
The novice evaluators conducted the heuristic evaluations of the two groupware systems on four PC workstations located in a single row in the undergraduate computer lab.
...Installation instructions (Appendix A.3) were provided on how to set-up the software.
Groupware systems.
As part of the study, the participants evaluated two quite different
shared visual workspaces contained in two real-time groupware systems: GroupDraw
and Groove.
GroupDraw is an object-oriented ‘toy’ drawing program built to show people
how to program the GroupKit groupware toolkit (Roseman and Greenberg 1996).
Groove is a virtual space for real-time, small group interactions. Users create “shared spaces” to communicate and collaborate with one another. Changes made to a shared space by one participant are automatically synchronized with all other computers.
Its functionality includes:
1. Communication tools – live voice over the Internet, instant messaging, text-based chat, and threaded discussion.
2. Content sharing tools – shared files, pictures, and contacts.
3. Joint activity tools – co-Web browsing, multiple-user drawing and editing, group calendar.
4.3.3 Method
The heuristic evaluation of the GroupDraw and Groove interfaces followed Nielsen’s standard recommendations (refer to chapter 2 for details).
This involved administering an orientation session to each group of inspectors prior to the evaluation process.
However, I did not conduct a debriefing session due to the geographical separation between the
inspectors and the researcher (myself).
Orientation session.
Prior to the session, each participant was asked to:
• sign the consent form (A.1);
• fill-out the background information questionnaire (A.2);
• read the detailed written description of the groupware heuristics (A.5 and A.6); and
• complete the pre-training feedback form (A.4).
Given the number of inspectors (27 total inspectors) and their location (22 in Calgary and 5 in Saskatoon), I conducted three separate 90-minute orientation sessions to three audiences.
* Collected the signed consent form, the completed background questionnaire and pretraining
feedback form from each inspector.
* Inspectors were handed a blank post-training feedback form (this form was identical to the pre-training feedback form) and an ample supply of blank problem reports for their heuristic evaluation of the systems.
* Conducted a one-hour training session on the proposed groupware heuristics. This included a review of the theory supporting each heuristics, how to apply them during an evaluation, and real-time groupware examples that illustrated compliance and non-compliance with each heuristic.
* Participants then filled out a post-training feedback form so that I could gauge how well they comprehended the groupware heuristics upon receiving the training.
* Provided the inspectors with an overview of the two groupware systems under test, GroupDraw and Groove.
* For GroupDraw, the inspectors were asked to evaluate only the shared workspace (Figure 4.1 bottom) and the Notes functionality (Figure 4.2) as the means for communicating with one another. For Groove, I asked the inspectors to evaluate the Outliner tool (Figure 4.3) as well as the text chat and audio link.
* General instructions were given regarding the process for conducting the heuristic evaluation. The novices were to inspect both systems with only the mechanics of collaboration heuristics. The regular specialists were to assess the groupware interfaces with both the mechanics of collaboration and Locales Framework heuristics.
Evaluation process.
Each inspector dictated when, where, and how to perform the evaluation. As with traditional heuristic evaluation, they could use the heuristics to systematically review all the functionality. Alternatively, they could walk through an imaginary task of their own making and verify how each step of the task complied with the heuristics.
Inspectors had complete control over the length of time and amount of effort they for completing the evaluation.
In some instances, the evaluation was performed in pairs and other times it involved four or five inspectors working concurrently.
4.3.4 Data collection.
For each problem uncovered, the inspectors completed a separate problem report by recording a description of the problem, the violated heuristic, a severity rating, and an (optional) solution to the problem. They judged a ‘major’ severity rating as one that represented a significant obstacle to effective collaboration, while a ‘minor’ rating is one that could be worked around by the participant. A blank problem report is found in Appendix A.7.
With respect to formulating severity ratings for each usability problem, Nielsen (1994a) states that evaluators have difficulty performing this step during the evaluation process since they are more focused on finding new usability problems. In addition, each evaluator will not find all the usability problems in the system; therefore, the severity ratings will be incomplete since they only reflect those problems found by the evaluator.
These original problem reports form the raw data for my analysis in the next chapter (see Appendix B for all problem reports).
4.4 Conclusion
This chapter reviewed the methodology for the pilot study and the subsequent changes to the groupware heuristics in preparation for the main study. Next, the main research study was introduced via a detailed description of its methodology. Of primary importance is that we used people that we felt were reasonable approximations of the actual practitioners we would expect to do groupware evaluations, and that our methodology echoed Nielsen’s traditional heuristic evaluation methodology.
Source: Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Nov 5 - Baker, Groupware Heuristics (MSc thesis)
Chapter 3
Groupware Heuristics
Nielsen’s existing heuristics were derived from how well they explained usability problems resident within single-user systems. However, these heuristics are insufficient for groupware evaluation since they do not cater to groupware usability, which is distinctly different from single-user usability.
Single-user usability has been defined as the degree to which a system is effective, efficient, and pleasant to use, given a certain set of users and tasks (e.g., Shackel 1990).
Within this context, usability emphasizes ‘task work’: how a person performs the domain tasks and activities that result in the end products like drawings, documents, or models.
Groupware must also support task work to proceed effectively, efficiently, and pleasantly; however, these systems must go one step further and support teamwork—the work of working together—in order to be trulyusable.
Thus we can define groupware usability as the degree to which a system supports both single-user usability and teamwork. While this definition is a starting point, we need a better understanding of what we actually mean by ‘support for teamwork’.
Teamwork involves activities ranging from low level mechanical acts necessary for almost any type of real time collaboration, to those that are social and affective in nature. If we want to apply the heuristic evaluation methodology to groupware, we need new heuristics that identify those interface aspects necessary for effective teamwork.
Using these, the inspector can then examine the interface to see if adequate support is provided to a group. The problem is that developing new heuristics to evaluate teamwork is complicated since—unlike the single-user interface literature that helped Nielsen set up his heuristics—there is no broad corpus of design guidelines specific to teamwork issues.
As a starting point, I have instead adapted two sources as the basis for two potential sets of groupware heuristics.
1. Mechanics of Collaboration (Gutwin and Greenberg 2000)
2. Locales Framework (Fitzpatrick 1998)
These sources were chosen because they contain some of the few theories or frameworks dealing with teamwork that were specifically created with groupware in mind.
Gutwin and Greenberg’s (2000) mechanics of collaboration identifies those activities that support the mechanics of how people interact over a shared visual workspace. These activities include group members communicating, providing assistance, coordinating activity, dividing labour, and monitoring each other’s work.
I believe that heuristics derived from these mechanics can be applied to task-based groupware (e.g., shared whiteboards, brainstorming tools, etc.), which comprise the majority of existing groupware systems today.
In contrast, Fitzpatrick’s (1998) Locales Framework deals more with the social issues surrounding the use of groupware. Because they define social versus mechanical aspects of teamwork, I believe that heuristics derived from the Locales Framework are better suited for evaluating design subtleties of how social interaction is supported in general groupware environments that support a broad variety of tasks (such as Teamwave Workplace) rather than the mechanical aspects of task-specific groupware. While there is some overlap between heuristics suggested by the mechanics of collaboration and by the Locales Framework, they are both inspired by quite different conceptual frameworks.
I divide the rest of this chapter into two main parts. The first part (Section 3.1) provides a brief description of the mechanics of collaboration, and how I adapted them into eight heuristics. The second part (Section 3.2) follows a similar structure; details of the Locales Framework are presented along with a brief description of the five heuristics stemming from this framework.
3.1 Mechanics of collaboration
3.1.1 Background
The mechanics of collaboration frames the low level actions and interactions that small groups of people do if they are to complete a collaborative task effectively. These mechanics are specific to shared workspace groupware. They include communication, coordination, planning, monitoring, assistance, and protection.
Gutwin developed this framework both from his experience building shared workspace systems and how they were used, and from an extensive research of shared workspace usage and theory developed by others (e.g., Bly 1988, Tang and Leifer 1988, Tang 1991, Gutwin 1997, Gutwin and Greenberg 1999).
3.1.2 Heuristics for supporting the mechanics of collaboration
I believe that the framework can help inspectors identify usability problems of both groupware prototypes and existing systems. While the framework was developed with low-cost evaluation methods in mind, I had to adapt, restructure, and rephrase it as heuristics, and augment it with a few other important points omitted from the framework.
I should emphasize that this was done in cooperation with Gutwin and Greenberg, and there has been a mutual debate and evolution of both my heuristics and how the actual mechanics are being articulated by them over time.
The resulting eight mechanics of collaboration heuristics are listed in Table 3.1.
1. Provide the means for intentional and appropriate verbal communication
The prevalent form of communication between group members is verbal conversations. This establishes a common understanding of the task at hand. Support verbal exchanges or a viable alternative.
2. Provide the mean for intentional and appropriate gestural communication
Allow explicit gestures and other visual actions to be visible since they are done in direct support of the conversation and help convey task information. Support illustration, emblem and deixis.
3. Provide consequential communication of an individual’s embodiment
A person’s body interacting with a computational workspace must unintentionally give off information to others. This is the primary mechanism for maintaining awareness and sustaining teamwork. Couple unintentional body language with both the workspace and its artifacts, and the conversation.
4. Provide consequential communication of shared artifacts (i.e. artifact feedthrough)
Make artifacts expressive so they give off information as they are manipulated. Support artifact feedthrough.
5. Provide protection
Protect users from inadvertently interfering with work that others are doing now, or altering or destroying work that they have done. Provide mechanisms to support social protocols and/or implement technical means to ensure protection.
6. Manage the transitions between tightly and loosely-coupled collaboration
Users should be able to focus their attention on different parts of the workspace when performing individual work in order to maintain awareness of others. Provide techniques for making relevant parts of the workspace visible.
7. Support people with the coordination of their actions
Support awareness of others’ activities to ensure people can coordinate their actions in order to avoid conflicts and make tasks happen in the correct sequence.
8. Facilitate finding collaborators and establishing contact
Provide information on potential collaborators so that they can be easily found and their availability for group work can be determined. Initiation of contact should be possible with minimal effort.
======Table 3.1 Mechanics of Collaboration heuristics=======
3.1.3 Summary
In summary, these eight heuristics look at the essential mechanical acts of collaboration. The physics of the everyday world as well as people’s natural actions means these mechanics ‘just happen’. In the computer world, they must be explicitly recognized as well as designed and implemented into the groupware system.
3.2 Locales Framework
3.2.1 Background
The Locales Framework is an approach to help people understand the nature of social activity and teamwork, and how a locale (or place) can support these activities (Fitzpatrick et al. 1996, Fitzpatrick 1998). More formally, the Locales Framework comprises five highly interdependent and overlapping aspects, as summarized below.
1. Locale foundations define a collection of people and artifacts (tools, information, objects) in relation to the central purpose of the social world. A locale within a social world is best considered as a “center” of collective purpose that is part of a dynamic and continually evolving system. Locales are fluid places with social meaning that may be mapped onto physical spaces.
2. Mutuality considers those interactions within locales that maintain a sense of shared place. Mutuality includes presence information that people and artifacts make available to others, and how people maintain awareness of that information. It also includes capabilities that entities have to transmit and receive information, and how entities choose from these capabilities to create a particular presence-awareness level.
3. Individual view over multiple locales acknowledges that individuals can be participating in many locales. Each person’s view is an aggregation of their views onto their locales of interest. People also manifest a “view intensity” onto particular locales as they vary their focus and participation across locales.
4. Interaction trajectories concern how courses of action evolve and change over time. In essence, people come into locales and social worlds with past experiences, plans, and actions.
5. Civic structures concern how interactions fit within a broader communal level. Civic structures can be considered a “meta-locale” that describe how social worlds and locales relate to one another, how people find their way between them, and how new locales are formed and old ones dissipated.
3.2.2 Heuristics for supporting the Locales Framework
The Locales Framework was not originally developed as a usability evaluation method, rather as a means to understand the nature of social practices in the workaday world and to help inform groupware design. Regardless, it seems amenable to expand this framework into a set of groupware heuristics for several reasons.
First, this is a general framework for understanding the fundamental aspects of teamwork. It describes a small set of inter-dependent perspectives of the characteristics of teamwork, where each could be recast as a heuristic.
Second, it has been validated as a way to understand existing work practices, and to motivate the design of new systems.
Greenberg et al. (1999) first introduced the Locales Framework heuristics. Based on the text and descriptions found in Chapters 1 and 8 of Fitzpatrick (1998), each of the aforementioned aspects was rewritten as a separate heuristic asking whether a groupware’s interface afforded certain social phenomena. Table 3.2 presents a brief summary of these heuristics.
1. Provide centers (locales)
Provide virtual locales as the site, means and resources for a group to pursue team and task
work. The system should collect people, artifacts and resources in relation to the central
purpose of the social world.
2. Provide awareness (mutuality) within locales
People and artifacts must make presence information available to others. People must be aware of others and their activities as they evolve.
3. Allow individual views
Individuals should be able to adapt their own idiosyncratic view of the locale or aggregate multiple locales in a way that reflect their responsibilities, activities, and interests.
4. Allow people to manage and stay aware of their evolving interactions over time
People must be able to manage and stay aware of their evolving interactions. This involves control over past, present and future aspects of routine and non-routine work.
5. Provide a way to organize and relate locales to one another (civic structures)
Locales are rarely independent of one another: people need a way to structure the locales in a meaningful way, to find their way between locales, to create new locales, and to remove old ones.
======Table 3.2 Locales Framework heuristics======
3.3 Conclusion
In support of my research goal, I have developed two sets heuristics for the purposes of
identifying usability problems in groupware.
I would not claim that all groupware applications should support all heuristics. Rather, the heuristics can be used to suggest areas of system strengths, and also areas of weakness that might need to be compensated for in other ways.
While these heuristics are tentative, I believe they are good candidates. Unlike Nielsen’s heuristics, each set is derived from a well-defined theory or framework of group work.
Source: Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Groupware Heuristics
Nielsen’s existing heuristics were derived from how well they explained usability problems resident within single-user systems. However, these heuristics are insufficient for groupware evaluation since they do not cater to groupware usability, which is distinctly different from single-user usability.
Single-user usability has been defined as the degree to which a system is effective, efficient, and pleasant to use, given a certain set of users and tasks (e.g., Shackel 1990).
Within this context, usability emphasizes ‘task work’: how a person performs the domain tasks and activities that result in the end products like drawings, documents, or models.
Groupware must also support task work to proceed effectively, efficiently, and pleasantly; however, these systems must go one step further and support teamwork—the work of working together—in order to be trulyusable.
Thus we can define groupware usability as the degree to which a system supports both single-user usability and teamwork. While this definition is a starting point, we need a better understanding of what we actually mean by ‘support for teamwork’.
Teamwork involves activities ranging from low level mechanical acts necessary for almost any type of real time collaboration, to those that are social and affective in nature. If we want to apply the heuristic evaluation methodology to groupware, we need new heuristics that identify those interface aspects necessary for effective teamwork.
Using these, the inspector can then examine the interface to see if adequate support is provided to a group. The problem is that developing new heuristics to evaluate teamwork is complicated since—unlike the single-user interface literature that helped Nielsen set up his heuristics—there is no broad corpus of design guidelines specific to teamwork issues.
As a starting point, I have instead adapted two sources as the basis for two potential sets of groupware heuristics.
1. Mechanics of Collaboration (Gutwin and Greenberg 2000)
2. Locales Framework (Fitzpatrick 1998)
These sources were chosen because they contain some of the few theories or frameworks dealing with teamwork that were specifically created with groupware in mind.
Gutwin and Greenberg’s (2000) mechanics of collaboration identifies those activities that support the mechanics of how people interact over a shared visual workspace. These activities include group members communicating, providing assistance, coordinating activity, dividing labour, and monitoring each other’s work.
I believe that heuristics derived from these mechanics can be applied to task-based groupware (e.g., shared whiteboards, brainstorming tools, etc.), which comprise the majority of existing groupware systems today.
In contrast, Fitzpatrick’s (1998) Locales Framework deals more with the social issues surrounding the use of groupware. Because they define social versus mechanical aspects of teamwork, I believe that heuristics derived from the Locales Framework are better suited for evaluating design subtleties of how social interaction is supported in general groupware environments that support a broad variety of tasks (such as Teamwave Workplace) rather than the mechanical aspects of task-specific groupware. While there is some overlap between heuristics suggested by the mechanics of collaboration and by the Locales Framework, they are both inspired by quite different conceptual frameworks.
I divide the rest of this chapter into two main parts. The first part (Section 3.1) provides a brief description of the mechanics of collaboration, and how I adapted them into eight heuristics. The second part (Section 3.2) follows a similar structure; details of the Locales Framework are presented along with a brief description of the five heuristics stemming from this framework.
3.1 Mechanics of collaboration
3.1.1 Background
The mechanics of collaboration frames the low level actions and interactions that small groups of people do if they are to complete a collaborative task effectively. These mechanics are specific to shared workspace groupware. They include communication, coordination, planning, monitoring, assistance, and protection.
Gutwin developed this framework both from his experience building shared workspace systems and how they were used, and from an extensive research of shared workspace usage and theory developed by others (e.g., Bly 1988, Tang and Leifer 1988, Tang 1991, Gutwin 1997, Gutwin and Greenberg 1999).
3.1.2 Heuristics for supporting the mechanics of collaboration
I believe that the framework can help inspectors identify usability problems of both groupware prototypes and existing systems. While the framework was developed with low-cost evaluation methods in mind, I had to adapt, restructure, and rephrase it as heuristics, and augment it with a few other important points omitted from the framework.
I should emphasize that this was done in cooperation with Gutwin and Greenberg, and there has been a mutual debate and evolution of both my heuristics and how the actual mechanics are being articulated by them over time.
The resulting eight mechanics of collaboration heuristics are listed in Table 3.1.
1. Provide the means for intentional and appropriate verbal communication
The prevalent form of communication between group members is verbal conversations. This establishes a common understanding of the task at hand. Support verbal exchanges or a viable alternative.
2. Provide the mean for intentional and appropriate gestural communication
Allow explicit gestures and other visual actions to be visible since they are done in direct support of the conversation and help convey task information. Support illustration, emblem and deixis.
3. Provide consequential communication of an individual’s embodiment
A person’s body interacting with a computational workspace must unintentionally give off information to others. This is the primary mechanism for maintaining awareness and sustaining teamwork. Couple unintentional body language with both the workspace and its artifacts, and the conversation.
4. Provide consequential communication of shared artifacts (i.e. artifact feedthrough)
Make artifacts expressive so they give off information as they are manipulated. Support artifact feedthrough.
5. Provide protection
Protect users from inadvertently interfering with work that others are doing now, or altering or destroying work that they have done. Provide mechanisms to support social protocols and/or implement technical means to ensure protection.
6. Manage the transitions between tightly and loosely-coupled collaboration
Users should be able to focus their attention on different parts of the workspace when performing individual work in order to maintain awareness of others. Provide techniques for making relevant parts of the workspace visible.
7. Support people with the coordination of their actions
Support awareness of others’ activities to ensure people can coordinate their actions in order to avoid conflicts and make tasks happen in the correct sequence.
8. Facilitate finding collaborators and establishing contact
Provide information on potential collaborators so that they can be easily found and their availability for group work can be determined. Initiation of contact should be possible with minimal effort.
======Table 3.1 Mechanics of Collaboration heuristics=======
3.1.3 Summary
In summary, these eight heuristics look at the essential mechanical acts of collaboration. The physics of the everyday world as well as people’s natural actions means these mechanics ‘just happen’. In the computer world, they must be explicitly recognized as well as designed and implemented into the groupware system.
3.2 Locales Framework
3.2.1 Background
The Locales Framework is an approach to help people understand the nature of social activity and teamwork, and how a locale (or place) can support these activities (Fitzpatrick et al. 1996, Fitzpatrick 1998). More formally, the Locales Framework comprises five highly interdependent and overlapping aspects, as summarized below.
1. Locale foundations define a collection of people and artifacts (tools, information, objects) in relation to the central purpose of the social world. A locale within a social world is best considered as a “center” of collective purpose that is part of a dynamic and continually evolving system. Locales are fluid places with social meaning that may be mapped onto physical spaces.
2. Mutuality considers those interactions within locales that maintain a sense of shared place. Mutuality includes presence information that people and artifacts make available to others, and how people maintain awareness of that information. It also includes capabilities that entities have to transmit and receive information, and how entities choose from these capabilities to create a particular presence-awareness level.
3. Individual view over multiple locales acknowledges that individuals can be participating in many locales. Each person’s view is an aggregation of their views onto their locales of interest. People also manifest a “view intensity” onto particular locales as they vary their focus and participation across locales.
4. Interaction trajectories concern how courses of action evolve and change over time. In essence, people come into locales and social worlds with past experiences, plans, and actions.
5. Civic structures concern how interactions fit within a broader communal level. Civic structures can be considered a “meta-locale” that describe how social worlds and locales relate to one another, how people find their way between them, and how new locales are formed and old ones dissipated.
3.2.2 Heuristics for supporting the Locales Framework
The Locales Framework was not originally developed as a usability evaluation method, rather as a means to understand the nature of social practices in the workaday world and to help inform groupware design. Regardless, it seems amenable to expand this framework into a set of groupware heuristics for several reasons.
First, this is a general framework for understanding the fundamental aspects of teamwork. It describes a small set of inter-dependent perspectives of the characteristics of teamwork, where each could be recast as a heuristic.
Second, it has been validated as a way to understand existing work practices, and to motivate the design of new systems.
Greenberg et al. (1999) first introduced the Locales Framework heuristics. Based on the text and descriptions found in Chapters 1 and 8 of Fitzpatrick (1998), each of the aforementioned aspects was rewritten as a separate heuristic asking whether a groupware’s interface afforded certain social phenomena. Table 3.2 presents a brief summary of these heuristics.
1. Provide centers (locales)
Provide virtual locales as the site, means and resources for a group to pursue team and task
work. The system should collect people, artifacts and resources in relation to the central
purpose of the social world.
2. Provide awareness (mutuality) within locales
People and artifacts must make presence information available to others. People must be aware of others and their activities as they evolve.
3. Allow individual views
Individuals should be able to adapt their own idiosyncratic view of the locale or aggregate multiple locales in a way that reflect their responsibilities, activities, and interests.
4. Allow people to manage and stay aware of their evolving interactions over time
People must be able to manage and stay aware of their evolving interactions. This involves control over past, present and future aspects of routine and non-routine work.
5. Provide a way to organize and relate locales to one another (civic structures)
Locales are rarely independent of one another: people need a way to structure the locales in a meaningful way, to find their way between locales, to create new locales, and to remove old ones.
======Table 3.2 Locales Framework heuristics======
3.3 Conclusion
In support of my research goal, I have developed two sets heuristics for the purposes of
identifying usability problems in groupware.
I would not claim that all groupware applications should support all heuristics. Rather, the heuristics can be used to suggest areas of system strengths, and also areas of weakness that might need to be compensated for in other ways.
While these heuristics are tentative, I believe they are good candidates. Unlike Nielsen’s heuristics, each set is derived from a well-defined theory or framework of group work.
Source: Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Wednesday, November 4, 2009
Nov 5 - Baker, Heuristic Evaluation of Shared Workspace Groupware (MSc thesis)
Kevin F. Baker
Comments: My interest in Baker's works is due to the fact that I have read about his works indirectly in other people's dissertations and research.
Publications from this Research
An earlier version of the mechanics of collaboration heuristics (similar to Chapter 3) has appeared in the following peer-reviewed publication:
Baker, K., Greenberg, S. and Gutwin, C. (2001) Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration. In M. Little and L. Nigay (Eds) Engineering for Human-Computer Interaction, LNCS Vol 2254, pp. 123-139.
The methodology, analysis, and results from this research study (Chapters 4 and 5) have been summarized in the report listed below. This report has been peer-reviewed and accepted for the upcoming ACM Computer Supported Cooperative Work conference (CSCW 2002).
Baker, K., Greenberg, S. and Gutwin, C. (2002) Empirical Development of a Heuristic Evaluation Methodology for Shared Workspace Groupware. Report 2002-700-03, Department of Computer Science, University of Calgary, Alberta, Canada.
Abstract
Despite the increasing availability of groupware, most systems are not widely used. One main reason is that groupware is difficult to evaluate. In particular, there are no discount usability evaluation methodologies that can discover problems specific to teamwork.
In this research study, I adapt Nielsen’s heuristic evaluation methodology, designed originally for single user applications, to help inspectors rapidly, cheaply, and effectively identify usability problems within groupware systems.
Specifically, I take the Gutwin and Greenberg’s (2000) mechanics of collaboration and restate them as heuristics for the purposes of discovering problems in shared visual work surfaces for distance-separated groups.
As a secondary objective, I revise existing Locales Framework heuristics and assess their compatibility with the mechanics.
I evaluate the practicality of both sets of heuristics by having individuals with varying degrees of HCI and CSCW expertise use them to uncover usability problems in two groupware systems. The results imply that practitioners can effectively inspect and evaluate groupware with the mechanics of collaboration heuristics where they can identify obstacles to real-time interactions over shared workspaces.
The Locales Framework heuristics are not as promising: while inspectors do identify problems inhibiting groupware acceptance, their practicality is limited and they require further improvements.
Chapter 1
Introduction
1.2 A brief survey of single-user evaluation techniques
Research in HCI has developed a multitude of evaluation techniques for analyzing and then improving the usability of conventional single user interfaces. Each methodology highlights different usability issues and identifies different types of problems; therefore, evaluators can choose and mix an appropriate technique to fit the needs and nuances of their situation (McGrath 1996).
There are three primary categories that have been used to distinguish these different types of methods: user observations, field studies, and interface inspections.
1.2.1 User observations
Techniques in this category are conducted in a lab, ideally using a representative sample of the eventual users performing tasks that depict how the product will be used in the “real” world. Evaluators uncover problems, called ‘usability bugs’, by observing the participants completing the tasks with the interface under evaluation.
User observation methodologies include controlled experiments and usability testing.
Controlled experiments.
Controlled experiments are used to establish a cause-and-effect relationship; it must be shown with certainty that the variation of experimental factors (i.e. the independent variables), and only those factors, could have caused the effect observed in the data (i.e., the dependent variable).
Rigorous control is used in these experiments to ensure that all other uncontrolled variables (i.e., confounding variables) do not affect the results and their interpretation.
Usability testing.
The goal of usability testing is to identify and rectify usability deficiencies. This is in conjunction with the intent to create products that are easy to learn, satisfying to use, and that provide high utility and functionality for the users (Rubin 1994).
Designers can manipulate the design of the product to allow them to see how particular features encourage or discourage usability. Participants can do several perhaps unrelated tasks that allow an evaluator to see how the human computer system performs over a broad set of expected uses. The product itself can be changed as the test progresses. For example, if pilot testing reveals certain problems, then the product can be modified midway to correct them.
When the product is tested with one individual, the participant is encouraged to think aloud during the test. This involves users talking out loud while they are performing a particular task in order to reflect cognitive processes. An evaluator observes the participant performing the task in question by focusing on occurrences such as errors made and difficulties experienced.
The information collected can then be applied to remedy the observed usability problems by going through another design iteration of the product, eventually leading to another usability test.
1.2.2 Field studies
A significant problem with performing an evaluation within the laboratory is the failure to account for conditions, context, and tasks that are central to the system’s real world use. Part of this failure stems from the fact that many developers of systems often have only partial or naïve knowledge of the “real world” setting where the end system will be used.
Field studies allow us to study systems in use on real tasks in real work settings, and to observe or discover important factors that are not easily found in a laboratory setting.
Two field study techniques are ethnography and contextual inquiry.
Ethnography
Ethnography is a naturalistic methodology grounded in sociology and anthropology (Bentley et al. 1992, Hughes et al. 1994, Randall 1996). Its premise is that human activities are socially organized; therefore, it looks into patterns of collaboration and interaction.
Randall (1996) stresses four features of ethnography that make it distinct as a method:
1. Naturalistic: involves studying real people and their activities within their natural environment. Only by studying work under these circumstances can one rightfully inform the system’s design.
2. Prolonged: it takes time to form a coherent view of what is going on especially for a complex domain.
3. Seeks to elicit the social world from the point of view of those who inhabit it: the appropriate level of analysis is the significance of the behaviour and not the behaviour itself.
4. Data resists formalization: the methodology stresses the importance of context; therefore, there is no ‘right’ data to be collected.
Data is gathered by the ethnographer observing and recording participants in their environment as they go about their work activities using the technology and tools available to them. This includes focusing on social relationships and how they affect the nature of work. To understand what the culture is doing, the ethnographer must immerse oneself within the cultural framework.
The goal of an ethnographic study for system design is to identify routine practices, problems, and possibilities for development within a given activity or setting. The data gathered usually takes the form of field notes but can be supplemented by audio and video data.
Contextual inquiry
To facilitate designing products, contextual inquiry employs an interview methodology to gain knowledge of what people do within their real world context (Holzblatt and Beyer 1996, 1999).
Specifically, this is accomplished by first conducting interviews through observations and discussions with users as they work. Target users are representatives of those for whom the system is being developed.
1.2.3 Inspection methods
Inspection methods have evaluators ‘inspect’ an interface for usability bugs according to a set of criteria, usually related to how individuals see and perform a task. These methods use judgement as a source of feedback when evaluating specific elements of a user interface (Mack and Nielsen 1994). Inspection techniques include heuristic evaluations, task-centered walkthroughs, pluralistic walkthroughs, and cognitive walkthroughs.
Heuristic evaluation
Heuristic evaluation is a widely accepted discount evaluation method for diagnosing potential usability problems in user interfaces (Mack and Nielsen 1994, Nielsen 1992,1993, 1994a,b).
With this methodology, a small number of usability experts visually inspect an interface and judge its compliance with recognized usability principles (the “heuristics”) (Nielsen 1992, 1993, 1994a).
Heuristics are general rules used to describe common properties of usable interfaces (Nielsen 1994a). During a heuristic evaluation, heuristics help evaluators focus their attention on aspects of an interface that are often trouble spots, making detection of usability problems easier. Noncompliant aspects of the interface are captured as interface bug reports, where evaluators
describe the problem, its severity, and perhaps even suggestions of how to fix it.
Through a process called results synthesis, these raw usability problem reports are then transformed into a cohesive set of design recommendations that are passed on to developers (Cox 1998).
Cognitive walkthroughs
Cognitive walkthroughs (Wharton 1994) are intended to evaluate the design of an interface for ease of learning, particularly by exploration. This is an extension of a model of learning by exploration proposed by Polson and Lewis (1990). The model is related to Norman’s theory of action that forms the theoretical foundation for his work on cognitive engineering (Norman 1988). Cognitive walkthroughs also incorporate the construction-integration model developed by Kintsch (1988).
These ideas help the evaluators examine how the interface guides the user to generate the correct goals and sub goals to perform the required task, and to select the necessary actions to fulfill each goal.
Task-centered walkthroughs
The task-centered walkthrough is a discount usability variation of cognitive walkthroughs.
It was developed as one step in the task-centered design process (Lewis and Rieman 1994). This process looks to involve end users in the design process and provide context to the evaluation of the interface in question.
Pluralistic walkthroughs
Pluralistic walkthroughs (Bias 1994) are meetings where users, developers, and human factors people step through an interface for the purposes of identifying usability problems.
A pre-defined scenario dictates the participants’ interaction with the interface. The scenario ensures that the participants confront the screens just as they would during the successful conduct of the specified task online.
The walkthrough begins when the participants are presented a hardcopy snapshot of the first screen they would encounter in the scenario. Participants are asked to write on the hardcopy of the first panel the actions they would perform while attempting the specified task. After all participants have written their independent responses, the walkthrough administrator announces the “right” answer. The participants verbalize their responses and discuss potential usability problems due to “incorrect” answers.
1.3 Problems applying single-user techniques to groupware evaluation
1.3.3 Inspection methods
As in single-user applications, groupware must effectively support task work. However, groupware must also support teamwork, the ‘work of working together’. Inspection methods are thus limited when we use them ‘as-is’, for they do not address the teamwork components necessary for effective collaboration with groupware.
For example, Nielsen lists many heuristics to guide inspectors, yet none address ‘bugs’ particular to groupware usability.
Similarly, a cognitive walkthrough used to evaluate groupware gave mixed and somewhat inconclusive results (Erback and Hook 1994). Other researchers are providing a framework for typical groupware scenarios that can form a stronger basis for walkthroughs (Cugini et al. 1997).
I speculate in this research study that some of these inspection techniques can be altered to evaluate groupware.
Specifically, I chose to adapt Nielsen’s heuristic evaluation methodology since it is popular with both researchers and industry for several important reasons. It is low cost in terms of time since it can be completed in a relatively short amount of time (i.e., a few hours). End-users are also not required; therefore, resources are inexpensive. Because the heuristics are well documented and worked examples have been made available (e.g., Nielsen 1994a, b), they can be easy to learn and apply. Also, heuristic evaluation is becoming part of the standard HCI curriculum (e.g., Greenberg 1996) and thus known to many HCI practitioners. Non-usability experts can also use this technique fairly successfully (Nielsen 1994a). As well, it is cost-effective: an aggregate of 3-5 usability specialists will typically identify ~75% of all known usability problems for a given interface (Nielsen 1994b). All these factors contribute to the significant uptake of heuristic evaluation in today’s industry since this technique can be easily and cost-effectively integrated into existing development processes while producing instant results.
In expanding heuristic evaluation for the purposes of evaluating groupware, I look to capitalize on all these factors that make this methodology a success.
1.4 Problem statement and research goals
The motivation behind this research is that current real-time distributed groupware systems are awkward and cumbersome to use, a situation partly caused by the lack of practical groupware evaluation methodologies.
My general research goal is to develop and validate a groupware evaluation methodology that is practical in terms of time, cost, logistics, and evaluator experience, while still identifying significant problems in a groupware system.
To narrow the scope, I adopt an existing discount usability technique to real-time, distributed groupware supporting shared workspaces. Real-time distributed groupware encompasses collaborative systems that enable multiple people to work together at the same time but from different locations. A shared workspace is “a bounded space where people can see and manipulate artifacts related to their activities.” (Gutwin 1997). This application genre is very common (e.g., real-time systems for sharing views of conventional applications).
Specifically, I focused on heuristic evaluation as this methodology in its current state satisfies the practicality criteria of time, cost, logistics, evaluator experience while still identifying significant problems in a single user systems. I believe that this technique and its strengths can be extended to assessing collaborative systems.
From this general research goal, my specific research sub-goals follow.
1. I will propose a new set of heuristics that can be used within the heuristic evaluation methodology to detect usability problems in real-time, distributed groupware with a shared workspace.
2. I will demonstrate that the adapted heuristic evaluation for groupware remains a ‘discount’ usability technique by analyzing the ability of inspectors to identify problems in collaborative applications.
1.5 Research direction
At this point, I need to elaborate on the circumstance and my resulting decisions that led to the main thrust of my research study: to derive and validate groupware heuristics based on the mechanics of collaboration. The purpose is to provide insight into my disproportionate focus with the Locales Framework heuristics.
My original objective was to build upon Greenberg et al.’s (1999) preliminary work on the Locales Framework heuristics. While conventional heuristics are easy to learn and apply, an outstanding concern from the original study was that heuristics based on the Locales Framework are complex, which in turn might require a greater level of evaluator training and experience. To that extent, I set out to assess these heuristics by studying how well inspectors unfamiliar with the Locales Framework were able to apply these heuristics to identify usability problems in groupware systems.
Shortly afterwards Gutwin and Greenberg (2000) introduced the mechanics of collaboration framework. This framework was created with low-cost evaluation methods for groupware in mind; therefore, I decided to refocus my research in this direction.
Subsequently, this research study concentrates on creating and validating the mechanics of collaboration heuristics.
While I still explore the locales framework heuristics, they are not my primary area of interest and hence I have devoted less time and effort in this study to their validation.
1.6 Research overview
Chapter 2 chronicles Jakob Nielsen’s design, validation, and subsequent evolution of his original 10 heuristics for the purposes of evaluating single-user interfaces. ...I believe it is necessary to provide a brief history on how the existing heuristic methodology was developed, validated, and updated.
Chapter 3 describes in detail the eight heuristics derived from Gutwin and Greenberg’s (2000) mechanics of collaboration framework. These heuristics form the basis for the rest of the research. In addition, five complementary heuristics evolving from the Locales Framework (Fitzpatrick 1998) are also briefly introduced.
Chapter 4 details the two-step methodology I employed to validate the groupware heuristics as a discount usability method for groupware.
First, a pilot study was conducted to review and subsequently improve the heuristics. Next, two categories of inspectors with varying levels of expertise in HCI and CSCW used the revised heuristics to evaluate two groupware systems. The resulting problem reports form the raw data for the forthcoming analysis.
Chapter 5 describes the results synthesis process employed to transform the inspectors’ raw problem reports into a consolidated list of usability problems for each groupware system.
Next, I systematically analyze these lists to derive conclusions regarding the practicality of both sets of groupware heuristics.
Finally, I discuss some of the factors affecting my results and how I interpret these results.
Chapter 6 summarizes how the goals of my research have been satisfied and the contributions made. In addition, I look to the future and discuss what still needs to be done to help evolve the heuristics for the purposes of developing a robust and effective low-cost technique for evaluating groupware.
Source: Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Comments: My interest in Baker's works is due to the fact that I have read about his works indirectly in other people's dissertations and research.
Publications from this Research
An earlier version of the mechanics of collaboration heuristics (similar to Chapter 3) has appeared in the following peer-reviewed publication:
Baker, K., Greenberg, S. and Gutwin, C. (2001) Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration. In M. Little and L. Nigay (Eds) Engineering for Human-Computer Interaction, LNCS Vol 2254, pp. 123-139.
The methodology, analysis, and results from this research study (Chapters 4 and 5) have been summarized in the report listed below. This report has been peer-reviewed and accepted for the upcoming ACM Computer Supported Cooperative Work conference (CSCW 2002).
Baker, K., Greenberg, S. and Gutwin, C. (2002) Empirical Development of a Heuristic Evaluation Methodology for Shared Workspace Groupware. Report 2002-700-03, Department of Computer Science, University of Calgary, Alberta, Canada.
Abstract
Despite the increasing availability of groupware, most systems are not widely used. One main reason is that groupware is difficult to evaluate. In particular, there are no discount usability evaluation methodologies that can discover problems specific to teamwork.
In this research study, I adapt Nielsen’s heuristic evaluation methodology, designed originally for single user applications, to help inspectors rapidly, cheaply, and effectively identify usability problems within groupware systems.
Specifically, I take the Gutwin and Greenberg’s (2000) mechanics of collaboration and restate them as heuristics for the purposes of discovering problems in shared visual work surfaces for distance-separated groups.
As a secondary objective, I revise existing Locales Framework heuristics and assess their compatibility with the mechanics.
I evaluate the practicality of both sets of heuristics by having individuals with varying degrees of HCI and CSCW expertise use them to uncover usability problems in two groupware systems. The results imply that practitioners can effectively inspect and evaluate groupware with the mechanics of collaboration heuristics where they can identify obstacles to real-time interactions over shared workspaces.
The Locales Framework heuristics are not as promising: while inspectors do identify problems inhibiting groupware acceptance, their practicality is limited and they require further improvements.
Chapter 1
Introduction
1.2 A brief survey of single-user evaluation techniques
Research in HCI has developed a multitude of evaluation techniques for analyzing and then improving the usability of conventional single user interfaces. Each methodology highlights different usability issues and identifies different types of problems; therefore, evaluators can choose and mix an appropriate technique to fit the needs and nuances of their situation (McGrath 1996).
There are three primary categories that have been used to distinguish these different types of methods: user observations, field studies, and interface inspections.
1.2.1 User observations
Techniques in this category are conducted in a lab, ideally using a representative sample of the eventual users performing tasks that depict how the product will be used in the “real” world. Evaluators uncover problems, called ‘usability bugs’, by observing the participants completing the tasks with the interface under evaluation.
User observation methodologies include controlled experiments and usability testing.
Controlled experiments.
Controlled experiments are used to establish a cause-and-effect relationship; it must be shown with certainty that the variation of experimental factors (i.e. the independent variables), and only those factors, could have caused the effect observed in the data (i.e., the dependent variable).
Rigorous control is used in these experiments to ensure that all other uncontrolled variables (i.e., confounding variables) do not affect the results and their interpretation.
Usability testing.
The goal of usability testing is to identify and rectify usability deficiencies. This is in conjunction with the intent to create products that are easy to learn, satisfying to use, and that provide high utility and functionality for the users (Rubin 1994).
Designers can manipulate the design of the product to allow them to see how particular features encourage or discourage usability. Participants can do several perhaps unrelated tasks that allow an evaluator to see how the human computer system performs over a broad set of expected uses. The product itself can be changed as the test progresses. For example, if pilot testing reveals certain problems, then the product can be modified midway to correct them.
When the product is tested with one individual, the participant is encouraged to think aloud during the test. This involves users talking out loud while they are performing a particular task in order to reflect cognitive processes. An evaluator observes the participant performing the task in question by focusing on occurrences such as errors made and difficulties experienced.
The information collected can then be applied to remedy the observed usability problems by going through another design iteration of the product, eventually leading to another usability test.
1.2.2 Field studies
A significant problem with performing an evaluation within the laboratory is the failure to account for conditions, context, and tasks that are central to the system’s real world use. Part of this failure stems from the fact that many developers of systems often have only partial or naïve knowledge of the “real world” setting where the end system will be used.
Field studies allow us to study systems in use on real tasks in real work settings, and to observe or discover important factors that are not easily found in a laboratory setting.
Two field study techniques are ethnography and contextual inquiry.
Ethnography
Ethnography is a naturalistic methodology grounded in sociology and anthropology (Bentley et al. 1992, Hughes et al. 1994, Randall 1996). Its premise is that human activities are socially organized; therefore, it looks into patterns of collaboration and interaction.
Randall (1996) stresses four features of ethnography that make it distinct as a method:
1. Naturalistic: involves studying real people and their activities within their natural environment. Only by studying work under these circumstances can one rightfully inform the system’s design.
2. Prolonged: it takes time to form a coherent view of what is going on especially for a complex domain.
3. Seeks to elicit the social world from the point of view of those who inhabit it: the appropriate level of analysis is the significance of the behaviour and not the behaviour itself.
4. Data resists formalization: the methodology stresses the importance of context; therefore, there is no ‘right’ data to be collected.
Data is gathered by the ethnographer observing and recording participants in their environment as they go about their work activities using the technology and tools available to them. This includes focusing on social relationships and how they affect the nature of work. To understand what the culture is doing, the ethnographer must immerse oneself within the cultural framework.
The goal of an ethnographic study for system design is to identify routine practices, problems, and possibilities for development within a given activity or setting. The data gathered usually takes the form of field notes but can be supplemented by audio and video data.
Contextual inquiry
To facilitate designing products, contextual inquiry employs an interview methodology to gain knowledge of what people do within their real world context (Holzblatt and Beyer 1996, 1999).
Specifically, this is accomplished by first conducting interviews through observations and discussions with users as they work. Target users are representatives of those for whom the system is being developed.
1.2.3 Inspection methods
Inspection methods have evaluators ‘inspect’ an interface for usability bugs according to a set of criteria, usually related to how individuals see and perform a task. These methods use judgement as a source of feedback when evaluating specific elements of a user interface (Mack and Nielsen 1994). Inspection techniques include heuristic evaluations, task-centered walkthroughs, pluralistic walkthroughs, and cognitive walkthroughs.
Heuristic evaluation
Heuristic evaluation is a widely accepted discount evaluation method for diagnosing potential usability problems in user interfaces (Mack and Nielsen 1994, Nielsen 1992,1993, 1994a,b).
With this methodology, a small number of usability experts visually inspect an interface and judge its compliance with recognized usability principles (the “heuristics”) (Nielsen 1992, 1993, 1994a).
Heuristics are general rules used to describe common properties of usable interfaces (Nielsen 1994a). During a heuristic evaluation, heuristics help evaluators focus their attention on aspects of an interface that are often trouble spots, making detection of usability problems easier. Noncompliant aspects of the interface are captured as interface bug reports, where evaluators
describe the problem, its severity, and perhaps even suggestions of how to fix it.
Through a process called results synthesis, these raw usability problem reports are then transformed into a cohesive set of design recommendations that are passed on to developers (Cox 1998).
Cognitive walkthroughs
Cognitive walkthroughs (Wharton 1994) are intended to evaluate the design of an interface for ease of learning, particularly by exploration. This is an extension of a model of learning by exploration proposed by Polson and Lewis (1990). The model is related to Norman’s theory of action that forms the theoretical foundation for his work on cognitive engineering (Norman 1988). Cognitive walkthroughs also incorporate the construction-integration model developed by Kintsch (1988).
These ideas help the evaluators examine how the interface guides the user to generate the correct goals and sub goals to perform the required task, and to select the necessary actions to fulfill each goal.
Task-centered walkthroughs
The task-centered walkthrough is a discount usability variation of cognitive walkthroughs.
It was developed as one step in the task-centered design process (Lewis and Rieman 1994). This process looks to involve end users in the design process and provide context to the evaluation of the interface in question.
Pluralistic walkthroughs
Pluralistic walkthroughs (Bias 1994) are meetings where users, developers, and human factors people step through an interface for the purposes of identifying usability problems.
A pre-defined scenario dictates the participants’ interaction with the interface. The scenario ensures that the participants confront the screens just as they would during the successful conduct of the specified task online.
The walkthrough begins when the participants are presented a hardcopy snapshot of the first screen they would encounter in the scenario. Participants are asked to write on the hardcopy of the first panel the actions they would perform while attempting the specified task. After all participants have written their independent responses, the walkthrough administrator announces the “right” answer. The participants verbalize their responses and discuss potential usability problems due to “incorrect” answers.
1.3 Problems applying single-user techniques to groupware evaluation
1.3.3 Inspection methods
As in single-user applications, groupware must effectively support task work. However, groupware must also support teamwork, the ‘work of working together’. Inspection methods are thus limited when we use them ‘as-is’, for they do not address the teamwork components necessary for effective collaboration with groupware.
For example, Nielsen lists many heuristics to guide inspectors, yet none address ‘bugs’ particular to groupware usability.
Similarly, a cognitive walkthrough used to evaluate groupware gave mixed and somewhat inconclusive results (Erback and Hook 1994). Other researchers are providing a framework for typical groupware scenarios that can form a stronger basis for walkthroughs (Cugini et al. 1997).
I speculate in this research study that some of these inspection techniques can be altered to evaluate groupware.
Specifically, I chose to adapt Nielsen’s heuristic evaluation methodology since it is popular with both researchers and industry for several important reasons. It is low cost in terms of time since it can be completed in a relatively short amount of time (i.e., a few hours). End-users are also not required; therefore, resources are inexpensive. Because the heuristics are well documented and worked examples have been made available (e.g., Nielsen 1994a, b), they can be easy to learn and apply. Also, heuristic evaluation is becoming part of the standard HCI curriculum (e.g., Greenberg 1996) and thus known to many HCI practitioners. Non-usability experts can also use this technique fairly successfully (Nielsen 1994a). As well, it is cost-effective: an aggregate of 3-5 usability specialists will typically identify ~75% of all known usability problems for a given interface (Nielsen 1994b). All these factors contribute to the significant uptake of heuristic evaluation in today’s industry since this technique can be easily and cost-effectively integrated into existing development processes while producing instant results.
In expanding heuristic evaluation for the purposes of evaluating groupware, I look to capitalize on all these factors that make this methodology a success.
1.4 Problem statement and research goals
The motivation behind this research is that current real-time distributed groupware systems are awkward and cumbersome to use, a situation partly caused by the lack of practical groupware evaluation methodologies.
My general research goal is to develop and validate a groupware evaluation methodology that is practical in terms of time, cost, logistics, and evaluator experience, while still identifying significant problems in a groupware system.
To narrow the scope, I adopt an existing discount usability technique to real-time, distributed groupware supporting shared workspaces. Real-time distributed groupware encompasses collaborative systems that enable multiple people to work together at the same time but from different locations. A shared workspace is “a bounded space where people can see and manipulate artifacts related to their activities.” (Gutwin 1997). This application genre is very common (e.g., real-time systems for sharing views of conventional applications).
Specifically, I focused on heuristic evaluation as this methodology in its current state satisfies the practicality criteria of time, cost, logistics, evaluator experience while still identifying significant problems in a single user systems. I believe that this technique and its strengths can be extended to assessing collaborative systems.
From this general research goal, my specific research sub-goals follow.
1. I will propose a new set of heuristics that can be used within the heuristic evaluation methodology to detect usability problems in real-time, distributed groupware with a shared workspace.
2. I will demonstrate that the adapted heuristic evaluation for groupware remains a ‘discount’ usability technique by analyzing the ability of inspectors to identify problems in collaborative applications.
1.5 Research direction
At this point, I need to elaborate on the circumstance and my resulting decisions that led to the main thrust of my research study: to derive and validate groupware heuristics based on the mechanics of collaboration. The purpose is to provide insight into my disproportionate focus with the Locales Framework heuristics.
My original objective was to build upon Greenberg et al.’s (1999) preliminary work on the Locales Framework heuristics. While conventional heuristics are easy to learn and apply, an outstanding concern from the original study was that heuristics based on the Locales Framework are complex, which in turn might require a greater level of evaluator training and experience. To that extent, I set out to assess these heuristics by studying how well inspectors unfamiliar with the Locales Framework were able to apply these heuristics to identify usability problems in groupware systems.
Shortly afterwards Gutwin and Greenberg (2000) introduced the mechanics of collaboration framework. This framework was created with low-cost evaluation methods for groupware in mind; therefore, I decided to refocus my research in this direction.
Subsequently, this research study concentrates on creating and validating the mechanics of collaboration heuristics.
While I still explore the locales framework heuristics, they are not my primary area of interest and hence I have devoted less time and effort in this study to their validation.
1.6 Research overview
Chapter 2 chronicles Jakob Nielsen’s design, validation, and subsequent evolution of his original 10 heuristics for the purposes of evaluating single-user interfaces. ...I believe it is necessary to provide a brief history on how the existing heuristic methodology was developed, validated, and updated.
Chapter 3 describes in detail the eight heuristics derived from Gutwin and Greenberg’s (2000) mechanics of collaboration framework. These heuristics form the basis for the rest of the research. In addition, five complementary heuristics evolving from the Locales Framework (Fitzpatrick 1998) are also briefly introduced.
Chapter 4 details the two-step methodology I employed to validate the groupware heuristics as a discount usability method for groupware.
First, a pilot study was conducted to review and subsequently improve the heuristics. Next, two categories of inspectors with varying levels of expertise in HCI and CSCW used the revised heuristics to evaluate two groupware systems. The resulting problem reports form the raw data for the forthcoming analysis.
Chapter 5 describes the results synthesis process employed to transform the inspectors’ raw problem reports into a consolidated list of usability problems for each groupware system.
Next, I systematically analyze these lists to derive conclusions regarding the practicality of both sets of groupware heuristics.
Finally, I discuss some of the factors affecting my results and how I interpret these results.
Chapter 6 summarizes how the goals of my research have been satisfied and the contributions made. In addition, I look to the future and discuss what still needs to be done to help evolve the heuristics for the purposes of developing a robust and effective low-cost technique for evaluating groupware.
Source: Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Friday, October 30, 2009
Oct 20 - ProQuest dissertation search "Usability Evaluation"
Oct 20 - Keywords were "Usability Evaluation." Continued to search and download, as continuation from Oct 15.
Al-Nuaim, Hana Abdullah. Development and Validation of a Multimedia User Interface Usability Evaluation Tool in the context of Educational Web Sites. [Dissertation, Doctor of Science] George Washington University. Jan 31, 2000.
Andre, Terence S. Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems. [Dissertation, PhD in Industrial & Systems Engineering] Virginia Polytechnic Institute and State University, Blacksburg, Virginia. April 3, 2000.
Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Capra, Miranda G. Usability Problem Description and the Evaluator Effect in Usability Testing. [Dissertation, PhD in Industrial & Systems Engineering] Virginia Polytechnic Institute and State University, Blacksburg, Virginia. March 13, 2006.
Chang, Yaowen. A Theory-based Usability Study of the Mouseover Abstract Interface. [Dissertation, Doctor of Education] Columbia University. 2005.
Clapsaddle, Donna J. Measuring Usability: Categorically Modeling Successful Websites Using Established Metrics. [dissertation, Doctor of Professional Studies in Computing] Pace University. May 2004.
Dykstra, Dean Julian. A Comparison of Heuristic Evaluation and Usability Testing: The Efficacy of a Domain-specific Heuristic Checklist. [dissertation, PhD] Texas A&M University. Dec 1993.
Elgin, Peter D. VALIDATING THE USER-CENTERED HYBRID ASSESSMENT TOOL (USER-CHAT): A COMPARATIVE USABILITY EVALUATION. [Dissertation, PhD] Kansas State University, Manhattan, Kansas. 2007.
Faulkner, Laura Lynn. Structured Software Usability Evaluation: An Experiment in Evaluation Design. [Dissertation, PhD] University of Texas at Austin. May 2006.
Govindaraju, Majorkumar. Development of Generic Design Guidelines to Manufacture Usable Consumer Products. [Dissertation, PhD] University of Cincinnati. 1999.
Hebb, Christopher Louis. Website Usability Evaluation using Sequential Analysis. [dissertation, PhD] Indiana University, Bloomington. May 2005.
Ho, Janet Chingyun. Evaluation of A Virtual Campus: Bell University Labs. [Thesis, Master of Applied Science] University of Toronto. 2000.
Hu, Xiangqun. Development and Evaluation of a Web-based Architectural Design Tool. [Thesis, Master of Computer Science] Technical University of Nova Scotia, Halifax, Nova Scotia. 1997.
Ivory, Melody Yvette. An Empirical Foundation for Automated Web Interface Evaluation. [Dissertation, PhD in Computer Science] University of California at Berkeley. Fall 2001.
Jenkins, Lillie Ruth. DESIGNING SYSTEMS THAT MAKE SENSE: WHAT DESIGNERS SAY ABOUT THEIR COMMUNICATION WITH USERS DURING THE USABILITY TESTING CYCLE. [Dissertation, PhD] the Ohio State University. 2004.
Jobrack-McDaniel, Naomi Ruth. Remote versus Laboratory Usability Evaluations with Static versus Interactive Graphical User Interfaces. [Thesis, M.A.] San Jose State University. May 1999.
Al-Nuaim, Hana Abdullah. Development and Validation of a Multimedia User Interface Usability Evaluation Tool in the context of Educational Web Sites. [Dissertation, Doctor of Science] George Washington University. Jan 31, 2000.
Andre, Terence S. Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems. [Dissertation, PhD in Industrial & Systems Engineering] Virginia Polytechnic Institute and State University, Blacksburg, Virginia. April 3, 2000.
Baker, Kevin F. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. [Thesis, M.Sc.] University of Calgary, Calgary, Alberta, Canada. May 2002.
Capra, Miranda G. Usability Problem Description and the Evaluator Effect in Usability Testing. [Dissertation, PhD in Industrial & Systems Engineering] Virginia Polytechnic Institute and State University, Blacksburg, Virginia. March 13, 2006.
Chang, Yaowen. A Theory-based Usability Study of the Mouseover Abstract Interface. [Dissertation, Doctor of Education] Columbia University. 2005.
Clapsaddle, Donna J. Measuring Usability: Categorically Modeling Successful Websites Using Established Metrics. [dissertation, Doctor of Professional Studies in Computing] Pace University. May 2004.
Dykstra, Dean Julian. A Comparison of Heuristic Evaluation and Usability Testing: The Efficacy of a Domain-specific Heuristic Checklist. [dissertation, PhD] Texas A&M University. Dec 1993.
Elgin, Peter D. VALIDATING THE USER-CENTERED HYBRID ASSESSMENT TOOL (USER-CHAT): A COMPARATIVE USABILITY EVALUATION. [Dissertation, PhD] Kansas State University, Manhattan, Kansas. 2007.
Faulkner, Laura Lynn. Structured Software Usability Evaluation: An Experiment in Evaluation Design. [Dissertation, PhD] University of Texas at Austin. May 2006.
Govindaraju, Majorkumar. Development of Generic Design Guidelines to Manufacture Usable Consumer Products. [Dissertation, PhD] University of Cincinnati. 1999.
Hebb, Christopher Louis. Website Usability Evaluation using Sequential Analysis. [dissertation, PhD] Indiana University, Bloomington. May 2005.
Ho, Janet Chingyun. Evaluation of A Virtual Campus: Bell University Labs. [Thesis, Master of Applied Science] University of Toronto. 2000.
Hu, Xiangqun. Development and Evaluation of a Web-based Architectural Design Tool. [Thesis, Master of Computer Science] Technical University of Nova Scotia, Halifax, Nova Scotia. 1997.
Ivory, Melody Yvette. An Empirical Foundation for Automated Web Interface Evaluation. [Dissertation, PhD in Computer Science] University of California at Berkeley. Fall 2001.
Jenkins, Lillie Ruth. DESIGNING SYSTEMS THAT MAKE SENSE: WHAT DESIGNERS SAY ABOUT THEIR COMMUNICATION WITH USERS DURING THE USABILITY TESTING CYCLE. [Dissertation, PhD] the Ohio State University. 2004.
Jobrack-McDaniel, Naomi Ruth. Remote versus Laboratory Usability Evaluations with Static versus Interactive Graphical User Interfaces. [Thesis, M.A.] San Jose State University. May 1999.
Saturday, September 12, 2009
Sep 12 - Au et al, Automated Usability Testing Framework
Automated Usability Testing Framework.
Fiora T. W. Au, Simon Baker, Ian Warren, Gillian Dobbie.
Department of Electrical and Computer Engineering, University of Auckland, Auckland, New Zealand, Private Bag 92019, Auckland, New Zealand. {ian-w,gill}@cs.auckland.ac.nz
Proc. 9th Australasian User Interface Conference (AUIC2008), Wollongong, Australia
Abstract
Handheld device applications with poor usability can reduce the productivity of users and incur costs for businesses, thus usability testing should play a vital role in application development. Conventional usability testing methodologies, such as formal user testing, can be expensive, time consuming and labour intensive; less resource-demanding alternatives can yield unreliable results. Automating aspects of usability testing would improve its efficiency and make it more practical to perform throughout development.
An automated usability testing tool should capture as input the properties of an application’s graphical user interface, the sequence of user actions as they use the application to achieve particular tasks, their behaviour and comments, as well as a description of these tasks.
The tool should evaluate both the static and dynamic properties of the interface, examine navigational burden and suggest modifications or templates that would improve usability. Results should be quick and easy to interpret, and be understandable by personnel other than
specialised testers.
Several existing tools that are typical of the tools available today meet some but not all of these
requirements. In this paper we describe the design of the HUIA testing framework, in which we have to meet as many of these requirements as possible.
1 Introduction
Handheld devices continue to feature in most companies’ mobile business solutions, as a means of improving their workers and thus the companies’ productivity. Handheld device applications (HDAs) with poor usability can undermine the value of such solutions, but despite this usability testing is often neglected due to its relatively high demand in time and resources.
These issues apply also to functional testing, for which many automated testing tools and frameworks, such as JUnit, were developed and are now widely used making the process
far more efficient.
1.1 Usability
Usability can be defined by IEEE (1990) as ‘the ease with which a user can operate, prepare inputs for, and interpret outputs of a system or component’, and comprises of five main attributes as outlined in Le Peuple and Scane (2003) and Nielsen, J. (1993): Learnability, Efficiency, Memorability, Errors, and Satisfaction.
The very nature of handheld devices make their programs particularly susceptible to poor usability; the smaller size, lower resolution screens, limited memory, lack of keyboard and mouse, and frequency of use among other hardware constraints and usage factors make program and user interface design particularly challenging as reported in Weiss (2005), Lee and Grice (2004) and Moe, Dwolatzky, and Olst(2004).
1.2 Usability testing
Heuristic evaluations are relatively cheap, quick and easy to carry out, but it has been claimed by Le Peuple and Scane (2003), Scholtz (2006), and Spolsky (2001) that such evaluations are likely to identify only approximately fifty percent of actual problems, with a significant number of false problems raised and actual problems missed.
When carried out correctly, user testing is likely to identify most of the major usability issues, as it involves real users attempting real tasks, and is very useful for collecting feedback on subjective aspects of usability (such as satisfaction and aesthetic appeal).
Studies by Scholtz (2006), and Spolsky (2001) have shown that six to eight test users (per type of intended users for that particular software) are usually sufficient to identify most of the major usability issues.
The type of analyses performed on the gathered data depends on the goal of the user test.
Typical areas that are examined include mistake and failure rates, types of mistakes, time taken,
amount of interactions (such as number of clicks or amount of scrolling), user behavior and user feedback as reported in Le Peuple and Scane (2003), Nielsen, J. (1993) and Spolsky (2001).
Scholtz (2006) reports that because user tests are expensive and time consuming to conduct, they are usually performed infrequently and towards the end of the development process.
Listed below are examples of the two types of quantitative measures/statistics:
Interface Properties:
• Number of fonts used, font sizes
• Average size of buttons
• Deepest level of menus
• Average loading time of graphics
Interaction Statistics (for performing specified tasks):
• Number of clicks
• Average drag distance
• Amount of scrolling
• Error rate
• Failure rate
2.2 Automating usability testing
There are many challenges and issues associated with traditional usability testing methodologies, many of which are to do with inefficiency, management complexities and high resource demands; all these contribute to the industry’s general reluctance to integrate usability testing as an essential activity on par with functional testing, despite its importance. Instead, it is
often considered as a ‘nice-to-have’ reserved for larger projects with generous budgets.
Conducting usability testing towards the end of the development process runs the risk of leaving insufficient time and resources to respond to the usability issues raised.
User testing is arguably the most effective testing methodology, but it is very inefficient to carry out. The process is labour intensive and costly, having to design the test, recruit suitable test users, set up test sessions, run the test, collect the data and then analyze the results.
Other testing methods such as heuristic evaluations and cognitive walkthroughs involving just one or several usability experts are less costly, but not to the point where it can be performed on a frequent, iterative basis.
Currently there is a distinct separation of responsibilities between developers and usability testing specialists.
These specialists may or may not have a good understanding of the application domain, which could affect the validity of their opinions and findings.
Furthermore, developers who do not play an active role in the usability evaluations end up learning of their product’s usability ‘second-hand’, having to base design decisions on their own (sometimes incorrect) interpretations of usability evaluation results.
A logical solution would be to automate as many aspects of usability testing as possible, in much the same way that aspects of functional testing has been automated as described in Patton (2006). There has been some excellent work that investigates the state of the art in automated usability evaluation, such as Ivory and Hearst (2001).
The challenge is thus to create an automated usability testing tool that is aimed for use by developers.
3 Objectives
The main objectives of an automated usability testing tool are similar in many ways to any automated software tool as outlined in Patton (2006), and may be summarized as follows:
1. Effective usability testing:
The tool should detect as many if not more real usability issues as conventional usability testing methodologies.
2. Increased speed:
The tool should perform analyses (checks, calculations, comparisons etc.) and other processes quicker than if they were performed manually.
3. Increased efficiency:
By taking over parts of the testing process, developers/testers are free to perform other tasks, and need to dedicate less time to testing.
4. Improved accuracy and precision:
Results produced will always be functionally correct; errors are limited to those produced by inappropriate input or direction by the developer/tester.
5. Reduced resource demand:
Automation should reduce time, human resources, equipment and cost requirements.
6. Increased flexibility:
The tool should perform testing on a range of usability aspects, and allow customisation of test settings. It should also facilitate usability testing for all stages of development (as well as iterative/regression testing).
7. Consistency:
The tool should provide a means of maintaining set testing standards throughout development, so that testing can be performed by different people but the same standard of usability is enforced.
8. Promote usability:
By simplifying the usability testing process, the tool should encourage all personnel related to product development (including developers, managers, testers, sales personnel and clients) to be involved in the process.
Also, the tool itself should promote good usability practices by encouraging the use of proven design paradigms.
7 Conclusions
Usability is very important for handheld device applications, because those with poor usability can lower the productivity of its users and incur costs for businesses, thus undermining the value of a mobile business solution. Automating aspects of usability testing can improve testing efficiency and better facilitate its integration with the development process.
Ideally, an automated usability testing tool should capture a range of inputs, perform analyses on different aspects of usability, present results clearly, be simple and flexible to use, and able to be used throughout development.
None of the existing tools discussed meet all the requirements described. Notably however, none of the tools are able to suggest good usability solutions, they can only perform evaluations. Of the tools discussed, the HUIA testing framework addresses most requirements, though still requires future development, which may include GUI consistency checking such as that outlined in Mahajan et al. (1997), and the inclusion of standard interface description languages
such as those described in Souchon et al. (2003).
References that I may want to read further in future:
Baker, S; Au, F; Warren, I; Dobbie, G. (2007): HUIA: A Tool for Automated Usability Testing, UoASE-2007-1, University of Auckland.
Ivory, M. and Hearst, M. (2001): The State of the Art in Automated Usability Evaluation of User Interfaces, ACM Computing Surveys, 33 (4), pp. 173-197.
Lee, K; Grice, R. (2004): Developing a New Usability Testing Method for Mobile Devices. Professional Communication Conference, pp. 115 – 127.
Morae: Usability Testing for Software and Websites. http://www.techsmith.com/morae.asp, visited 8th March, 2006.
Scholtz, J. (2006); Usability Evaluation. http://www.itl.nist.gov/iad/IApapers/2004/Usability%20Evaluation_rev1.pdf, visited 9th March.
Weiss, S. (2005): Handheld Usability: Design, Prototyping, & Usability Testing for Mobile Phones. Proceedings of the 7th Conference on Human-Computer Interaction with Mobile Devices and Services.
Fiora T. W. Au, Simon Baker, Ian Warren, Gillian Dobbie.
Department of Electrical and Computer Engineering, University of Auckland, Auckland, New Zealand, Private Bag 92019, Auckland, New Zealand. {ian-w,gill}@cs.auckland.ac.nz
Proc. 9th Australasian User Interface Conference (AUIC2008), Wollongong, Australia
Abstract
Handheld device applications with poor usability can reduce the productivity of users and incur costs for businesses, thus usability testing should play a vital role in application development. Conventional usability testing methodologies, such as formal user testing, can be expensive, time consuming and labour intensive; less resource-demanding alternatives can yield unreliable results. Automating aspects of usability testing would improve its efficiency and make it more practical to perform throughout development.
An automated usability testing tool should capture as input the properties of an application’s graphical user interface, the sequence of user actions as they use the application to achieve particular tasks, their behaviour and comments, as well as a description of these tasks.
The tool should evaluate both the static and dynamic properties of the interface, examine navigational burden and suggest modifications or templates that would improve usability. Results should be quick and easy to interpret, and be understandable by personnel other than
specialised testers.
Several existing tools that are typical of the tools available today meet some but not all of these
requirements. In this paper we describe the design of the HUIA testing framework, in which we have to meet as many of these requirements as possible.
1 Introduction
Handheld devices continue to feature in most companies’ mobile business solutions, as a means of improving their workers and thus the companies’ productivity. Handheld device applications (HDAs) with poor usability can undermine the value of such solutions, but despite this usability testing is often neglected due to its relatively high demand in time and resources.
These issues apply also to functional testing, for which many automated testing tools and frameworks, such as JUnit, were developed and are now widely used making the process
far more efficient.
1.1 Usability
Usability can be defined by IEEE (1990) as ‘the ease with which a user can operate, prepare inputs for, and interpret outputs of a system or component’, and comprises of five main attributes as outlined in Le Peuple and Scane (2003) and Nielsen, J. (1993): Learnability, Efficiency, Memorability, Errors, and Satisfaction.
The very nature of handheld devices make their programs particularly susceptible to poor usability; the smaller size, lower resolution screens, limited memory, lack of keyboard and mouse, and frequency of use among other hardware constraints and usage factors make program and user interface design particularly challenging as reported in Weiss (2005), Lee and Grice (2004) and Moe, Dwolatzky, and Olst(2004).
1.2 Usability testing
Heuristic evaluations are relatively cheap, quick and easy to carry out, but it has been claimed by Le Peuple and Scane (2003), Scholtz (2006), and Spolsky (2001) that such evaluations are likely to identify only approximately fifty percent of actual problems, with a significant number of false problems raised and actual problems missed.
When carried out correctly, user testing is likely to identify most of the major usability issues, as it involves real users attempting real tasks, and is very useful for collecting feedback on subjective aspects of usability (such as satisfaction and aesthetic appeal).
Studies by Scholtz (2006), and Spolsky (2001) have shown that six to eight test users (per type of intended users for that particular software) are usually sufficient to identify most of the major usability issues.
The type of analyses performed on the gathered data depends on the goal of the user test.
Typical areas that are examined include mistake and failure rates, types of mistakes, time taken,
amount of interactions (such as number of clicks or amount of scrolling), user behavior and user feedback as reported in Le Peuple and Scane (2003), Nielsen, J. (1993) and Spolsky (2001).
Scholtz (2006) reports that because user tests are expensive and time consuming to conduct, they are usually performed infrequently and towards the end of the development process.
Listed below are examples of the two types of quantitative measures/statistics:
Interface Properties:
• Number of fonts used, font sizes
• Average size of buttons
• Deepest level of menus
• Average loading time of graphics
Interaction Statistics (for performing specified tasks):
• Number of clicks
• Average drag distance
• Amount of scrolling
• Error rate
• Failure rate
2.2 Automating usability testing
There are many challenges and issues associated with traditional usability testing methodologies, many of which are to do with inefficiency, management complexities and high resource demands; all these contribute to the industry’s general reluctance to integrate usability testing as an essential activity on par with functional testing, despite its importance. Instead, it is
often considered as a ‘nice-to-have’ reserved for larger projects with generous budgets.
Conducting usability testing towards the end of the development process runs the risk of leaving insufficient time and resources to respond to the usability issues raised.
User testing is arguably the most effective testing methodology, but it is very inefficient to carry out. The process is labour intensive and costly, having to design the test, recruit suitable test users, set up test sessions, run the test, collect the data and then analyze the results.
Other testing methods such as heuristic evaluations and cognitive walkthroughs involving just one or several usability experts are less costly, but not to the point where it can be performed on a frequent, iterative basis.
Currently there is a distinct separation of responsibilities between developers and usability testing specialists.
These specialists may or may not have a good understanding of the application domain, which could affect the validity of their opinions and findings.
Furthermore, developers who do not play an active role in the usability evaluations end up learning of their product’s usability ‘second-hand’, having to base design decisions on their own (sometimes incorrect) interpretations of usability evaluation results.
A logical solution would be to automate as many aspects of usability testing as possible, in much the same way that aspects of functional testing has been automated as described in Patton (2006). There has been some excellent work that investigates the state of the art in automated usability evaluation, such as Ivory and Hearst (2001).
The challenge is thus to create an automated usability testing tool that is aimed for use by developers.
3 Objectives
The main objectives of an automated usability testing tool are similar in many ways to any automated software tool as outlined in Patton (2006), and may be summarized as follows:
1. Effective usability testing:
The tool should detect as many if not more real usability issues as conventional usability testing methodologies.
2. Increased speed:
The tool should perform analyses (checks, calculations, comparisons etc.) and other processes quicker than if they were performed manually.
3. Increased efficiency:
By taking over parts of the testing process, developers/testers are free to perform other tasks, and need to dedicate less time to testing.
4. Improved accuracy and precision:
Results produced will always be functionally correct; errors are limited to those produced by inappropriate input or direction by the developer/tester.
5. Reduced resource demand:
Automation should reduce time, human resources, equipment and cost requirements.
6. Increased flexibility:
The tool should perform testing on a range of usability aspects, and allow customisation of test settings. It should also facilitate usability testing for all stages of development (as well as iterative/regression testing).
7. Consistency:
The tool should provide a means of maintaining set testing standards throughout development, so that testing can be performed by different people but the same standard of usability is enforced.
8. Promote usability:
By simplifying the usability testing process, the tool should encourage all personnel related to product development (including developers, managers, testers, sales personnel and clients) to be involved in the process.
Also, the tool itself should promote good usability practices by encouraging the use of proven design paradigms.
7 Conclusions
Usability is very important for handheld device applications, because those with poor usability can lower the productivity of its users and incur costs for businesses, thus undermining the value of a mobile business solution. Automating aspects of usability testing can improve testing efficiency and better facilitate its integration with the development process.
Ideally, an automated usability testing tool should capture a range of inputs, perform analyses on different aspects of usability, present results clearly, be simple and flexible to use, and able to be used throughout development.
None of the existing tools discussed meet all the requirements described. Notably however, none of the tools are able to suggest good usability solutions, they can only perform evaluations. Of the tools discussed, the HUIA testing framework addresses most requirements, though still requires future development, which may include GUI consistency checking such as that outlined in Mahajan et al. (1997), and the inclusion of standard interface description languages
such as those described in Souchon et al. (2003).
References that I may want to read further in future:
Baker, S; Au, F; Warren, I; Dobbie, G. (2007): HUIA: A Tool for Automated Usability Testing, UoASE-2007-1, University of Auckland.
Ivory, M. and Hearst, M. (2001): The State of the Art in Automated Usability Evaluation of User Interfaces, ACM Computing Surveys, 33 (4), pp. 173-197.
Lee, K; Grice, R. (2004): Developing a New Usability Testing Method for Mobile Devices. Professional Communication Conference, pp. 115 – 127.
Morae: Usability Testing for Software and Websites. http://www.techsmith.com/morae.asp, visited 8th March, 2006.
Scholtz, J. (2006); Usability Evaluation. http://www.itl.nist.gov/iad/IApapers/2004/Usability%20Evaluation_rev1.pdf, visited 9th March.
Weiss, S. (2005): Handheld Usability: Design, Prototyping, & Usability Testing for Mobile Phones. Proceedings of the 7th Conference on Human-Computer Interaction with Mobile Devices and Services.
Subscribe to:
Posts (Atom)
