Aug 25 - Vavoula et al, A Lifecycle approach to evaluating MyArtSpace

Giasemi Vavoula, Julia Meek, Mike Sharples, Peter Lonsdale, Paul Rudman. A Lifecycle approach to evaluating MyArtSpace.
Proceedings: Fourth IEEE International Workshop on Wireless, Mobile and Ubiquitous Technology in Education. WMUTE 2006

The Lifecycle Evaluation Approach [2][3] places evaluation at the centre of the development process, from the very early stages of design to the final assessment of the technology. The approach draws on evaluation methods and ideas from software engineering, educational evaluation, and models for evaluating learning technology. Evaluation activities are undertaken at key points in the lifecycle of the system design process.

The overall goal of the evaluationwas agreed by the key stakeholders to be the assessment of the service’s usability, accessibility, effectiveness, and capability to engage the user.

The requirements (112 in total) that emerged from the user consultation workshop were divided into six main categories:
1. Teacher requirements: requirements for designing the teacher experience
2. Student requirements: requirements for designing the student experience
3. Museum requirements: requirements for designing the museum staff experience and
also institutional requirements
4. Culture Online requirements: requirements relevant to the funding body, pertaining to
overall project objectives, qualitative and quantitative measurement criteria, etc.
5. General system requirements: all other requirements that did not directly concern a
specific group
6. Usability requirements: requirements for designing an easy to use, easy to learn and enjoyable service/system for all user groups.

Due to the number of requirements that had been generated, the representatives were divided into groups and each group was given a subset of requirements to review and was asked to verify that each requirement was unambiguous and testable.
Subsequently, the stakeholders were asked to use the MoSCoW [4] technique from Dynamic Systems Development Method to indicate that a requirement is a:
􀂃 MUST: must have this
􀂃 SHOULD: should have this if at all possible
􀂃 COULD: could have this if it does not affect anything else
􀂃 WOULD: won’t have this time but would like to have in the future
The outcomes of the discussion in the subgroups were then presented to the larger group, who finalised the categorisation and prioritisation.
The workshop was successful in enabling the requirements to be discussed, clarified and prioritised. The outcome was an agreed set of user requirements and benchmarks for both the Web portal of MyArtSpace and the Mobile phone service.

MyArtSpace is a service that makes use of different technologies (mobile phones and the Web) to deliver a learning experience that traverses the museum and the
classroom. It was therefore essential that the evaluation of the service did not stop at the evaluations of its individual components; rather, it had to go beyond the technology to look at the learning experience as a whole. The Lifecycle Evaluation Approach has made this possible through its focus on both the usability and the pedagogy of MyArtSpace, which enabled a full picture of the use of a new educational technology to be understood, assessed and improved.

References that I would like to read further in the future:
[2] Meek, J. (2006) Adopting a Lifecycle Approach to the Evaluation of Computers and Information Technology. Unpublished PhD Thesis, The University of Birmingham, UK.
[3] Meek, J. and Sharples, M. (2001) A lifecycle approach to the evaluation of learning technology. Proceedings of CAL2001 Conference, Warwick, 2-1 April, pp. 195-196.
[4] http://www.dsdm.org/