Sep 25 - Nielsen, Card Sorting: How Many Users to Test (Alertbox)

Summary: Testing ever-more users in card sorting has diminishing returns, but you should still use three times more participants than you would in traditional usability tests.

Card Sorting: How Many Users to Test

One of the biggest challenges in website and intranet design is creating the information architecture: what goes where?
A classic mistake is to structure the information space based on how you view the content -- which often results in different subsites for each of your company's departments or information providers.

You can better enhance usability by creating an information architecture that reflects how users view the content.
In each of our intranet studies, we've found that some of the biggest productivity gains occur when companies restructure their intranet to reflect employees' workflow.
And in e-commerce, sales increase when products appear in the categories where users expect to find them.

All very good, but how do you find out the users' view of an information space and where they think each item should go?
For researching this type of mental model, the primary method is card sorting:
1. Write the name (and perhaps a short description) of each of the main items on an index card. Yes, good old paper cards.
2. Shuffle the cards and give the deck to a user. (The standard recommendations for recruiting test participants apply: they must be representative users, etc.)
3. Ask each user to sort the cards into piles, placing items that belong together in the same pile. Users can make as many or as few piles as they want; some piles can be big, others small.
4. Optional extra steps include asking users to arrange the resulting piles into bigger groups, and to name the different groups and piles. The latter step can give you ideas for words and synonyms to use for navigation labels, links, headlines, and search engine optimization.

Research Study

First, they tested 168 users, generating very solid results. They then simulated the outcome of running card sorting studies with smaller user groups by analyzing random subsets of the total dataset. For example, to see what a test of twenty users would generate, they selected twenty users randomly from the total set of 168 and analyzed only that subgroup's card sorting data. By selecting many such samples, it was possible to estimate the average findings from testing different numbers of users.
The main quantitative data from a card sorting study is a set of similarity scores that measures the similarity of user ratings for various item pairs. If all users sorted two cards into the same pile, then the two items represented by the cards would have 100% similarity. If half the users placed two cards together and half placed them in separate piles, those two items would have a 50% similarity score.
We can assess the outcome of a smaller card sorting study by asking how well its similarity scores correlate with the scores derived from testing a large user group. (A reminder: correlations run from -1 to +1. A correlation of 1 shows that the two datasets are perfectly aligned; 0 indicates no relationship; and negative correlations indicate datasets that are opposites of each other.)

How Many Users?

For most usability studies, I recommend testing five users, since that's enough data to teach you most of what you'll ever learn in a test. For card sorting, however, there's only a 0.75 correlation between the results from five users and the ultimate results. That's not good enough.
You must test fifteen users to reach a correlation of 0.90, which is a more comfortable place to stop. After fifteen users, diminishing returns set in and correlations increase very little: testing thirty people gives a correlation of 0.95 -- certainly better, but usually not worth twice the money. There are hardly any improvements from going beyond thirty users: you have to test sixty people to reach 0.98, and doing so is definitely wasteful.
Tullis and Wood recommend testing twenty to thirty users for card sorting. Based on their data, my recommendation is to test fifteen users.

Why More Users for Card Sorting?

We know that five users are enough for most usability studies, so why do we need three times as many participants to reach the same level of insight with card sorting?
Because the methods differ in two key ways:

User testing is an evaluation method: we already have a design, and we're trying to find out whether or not it's a good match with human nature and user needs. Although people differ substantially in their capabilities (domain knowledge, intelligence, and computer skills), if a certain design element causes difficulties, we'll see so after testing a few users.
A low-end user might experience more severe difficulties than a high-end user, but the magnitude of the difficulties is not at issue unless you are running a measurement study (which requires more users).
All you need to know is that the design element doesn't work for humans and should be changed.

Card sorting is a generative method: we don't yet have a design, and our goal is to find out how people think about certain issues.
There is great variability in different people's mental models and in the vocabulary they use to describe the same concepts.
We must collect data from a fair number of users before we can achieve a stable picture of the users' preferred structure and determine how to accommodate differences among users.

Source:
Jakob Nielsen's Alertbox, July 19, 2004:
Card Sorting: How Many Users to Test
http://www.useit.com/alertbox/20040719.html
Card Sorting: How Many Users to Test (Jakob Nielsen's Alertbox)

Aug 29 - Jokela et al, ..Standard Definition of Usability: Analyzing ISO 13407 against ISO

The Standard of User-Centered Design and the Standard Definition of Usability: Analyzing ISO13407 against ISO9241-11.
Timo Jokela, Netta Iivari. Oulu University, P.O. Box 3000, 90014 Oulu, Finland. +358 8 5531011 {timo.jokela, mailto:netta.iivari%7D@oulu.fi
Juha Matero, Minna Karukka. Nokia, P.O. Box 50, 90571 Oulu, Finland. {juha.p.matero, mailto:minna.karukka%7D@nokia.com

ABSTRACT
ISO 9241-11 and ISO 13407 are two important standards related to usability: the former one provides the definition of usability and the latter one guidance for designing usability. We carried out an interpretative analysis of ISO 13407 from the viewpoint of the standard definition of
usability from ISO 9241-11. The results show that ISO 13407 provides only partly guidance for designing usability as presumed by the definition. Guidance for describing users and environments are provided but very limited guidance is provided for the descriptions of user goals and usability measures, and generally for the process of producing the various outcomes.

Probably the best known definition of usability is by Nielsen: usability is about learnability, efficiency, memorability, errors, and satisfaction [16].

However, the definition of usability from ISO 9241-11 (Guidance on usability) [11] – “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” - is becoming the main reference of usability.

In addition that it is largely recognized in literature, this ‘standard’ definition of usability is used in the recent Common Industry Format, CIF, for usability testing [1].

To improve the usability of software and information systems, the paradigm of user-centered design1, UCD, has been proposed by a number of method and methodology books, starting from Nielsen [16] to ones published in late 90’s, [8], [4], [5], [15] and ending up with a set of very recent ones, [17] and [18].

1 Called ‘human-centered design’ in ISO 13407. Also called ‘usability engineering’.

My Comments: I think this article is a good resource for "Definition of Usability."

ISO 13407 [9], Human-centred design processes for interactive systems, is a standard that provides guidance for user-centered design. ...it describes usability at a level of principles, planning and activities. A third important aspect is that ISO 13407 explicitly uses the standard definition of usability from ISO 9241-11 as a reference for usability.

Usability is defined in ISO 9241-11 [11] as follows:
Usability: The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
The terms are further defined as follows:
Effectiveness: the accuracy and completeness with which users achieve specified goals
Efficiency: the resources expended in relation to the accuracy and completeness with which users achieve goals
Satisfaction: freedom from discomfort, and positive attitude to the use of the product
Context of use: characteristics of the users, tasks and the organizational and physical environments
Goal: intended outcome
Task: activities required to achieve a goal

Generally, this definition of usability is a ‘broad’ approach to usability [2]: usability is about supporting users in achieving their goals in their work, it is not only a characteristic of a user interface.

..usability is a function of users of a product or a system (specified users). Further, for each user, usability is a function of achieving goals in terms of a set of attributes (i.e. effectiveness, efficiency and satisfaction) and environment of use.

As an example, one usability measure of a bank machine
could be: • 90 % users achieve the goal (Es) in less than 1 minute (Ey) with an average satisfaction rating ‘6’ (S) when users are novice ones (U), and they want to have a desired sum of cash withdrawn (G) with any bank machine (Et).

The analysis of the definition of usability shows that one needs to determine the following outocomes when the definition is used in a development project:
(1) The users of the system,
(2) Goals of users,
(3) Environments of use
(4) Measures of effectiveness, efficiency and satisfaction.

ISO 13407 is aimed to provide ‘overview’ guidance for the planning and management of user-centered design, not to provide detailed coverage of the methods and technique.

ISO 13407 is an international standard established in 1999. The standard “provides guidance on humancentred design activities throughout the life cycle of computer-based interactive systems”. The standard aims at “those managing design processes” and does not provide detailed coverage of methods and techniques.

ISO 13407 describes user-centered design from four
different aspects:
• Rationale for UCD
• Planning UCD
• Principles of UCD
• Activities of UCD.

Rationale. The rationale part briefly describes the benefits that usable systems provide, such as reduction of training and support costs, improved user satisfaction and productivity of users.

Principles. The standard identifies four general principles that characterize user-centered design, and that are not bound to any specific phase of development cycle:
• The active involvement of users and a clear understanding of user and task requirements
• An appropriate allocation of functions between users and technology
• Iteration of design solutions
• Multi-disciplinary design.

Planning. The planning part provides guidance in fitting user-centered design activities into the overall system development process. Among other things, the standard emphasizes that project plans should reserve time and resources for iteration and user feedback. The importance of teamwork and communication is also mentioned.

Activities. The core of the standard – stated explicitly– is the description of user-centered design activities. The standard identifies four main activities of UCD, illustrated
in Figure 2:

Understand and Specify Context of Use. Know the user, the environment of use, and the tasks that he or she uses the product for.
Specify the User and Organizational Requirements. Determine the success criteria of usability for the product in terms of user tasks, e.g. how quickly a typical user should be able to complete a task with the product.
Determine the design guidelines and constraints. Produce Design Solutions. Incorporate HCI knowledge (of visual design, interaction design, usability) into design solutions.
Evaluate Designs against Requirements. The usability of designs is evaluated against user tasks.

Usability is one type of a quality characteristic in a product [10] among others, such as functionality, efficiency, reliability, maintainability and portability. In the requirement phase, when the quality requirements for a product are determined, also the usability requirements should be determined.
While all activities of life-cycle are relevant in the design of usability, the definition of usability has a critical impact especially in the requirements phase of a development project. The outcomes of these requirements activities (identification of users, goals, environments, usability measures) provide direction for the design phase and basis for planning evaluations.

In practice, Nielsen’s attributes as such are too ambiguous to be used in determining the usability requirements.

My Comments: This part of the article, particularly on User-Centred Design, is good on the process of applying usability throughout the entire design and development process.

Understand and specify the context of use
The standard describes the activity ‘Understand and
specify the context of use’ as follows:
The characteristics of the users, tasks and the organizational and physical environment define the context in which the system is used. It is important to understand and identify the details of this context in order to guide early design decisions, and to provide a basis for evaluation.
Information should be gathered about the context of use of new products and systems. If an existing system is upgraded or enhanced, this information may already be available but should be checked. If there are extensive results form user feedback, help desk reports and other data, these provide a basis fro prioritizing user requirements for system modifications and changes.
The context in which the system is to be used should be identified in terms of the following:
a) The characteristics of the intended users: relevant characteristics of the users can include knowledge, skill, experience, education, training, physical
attributes, habits, preferences and capabilities. If necessary, define the characteristics of different types of users, for example, with different levels of
experience of performing different roles (maintainers, installers, etc).
b) The tasks the users are to perform: the description should include the overall goals of the use of the system. The characteristics of tasks that can influence usability should be described, e.g. the frequency and the duration of performance”…. Tasks should not be described solely in terms of the functions or features.
c) The environment in which the users are to use the system: the environment includes the hardware, software and materials to be used. Their description
can be in terms of a set of products, one or more of which can be the focus of human-centred specification or evaluation, or it can be in terms of a set of
attributes or performance characteristics of the hardware, software and other materials. Relevant characteristics of the physical and social environment
should also be described. These can include relevant standards, attributes of the wider technical environment, the physical, ambient, legislative and the social and cultural environment.

The output from this activity should be a description of the relevant characteristics of users, tasks and environment, which identifies what aspects have an important impact on the system design. (See ISO 9241-11 for more information about the context of use and a sample report.)
The context of use description should
a) Specify the range of intended users, tasks and environments in sufficient detail to support design activity;
b) Be derived from suitable sources;
c) Be confirmed by users or if they are not available, by those representing their interests in the process;
d) Be adequately documented;
e) Be made available to the design team at appropriate times and in appropriate forms to support design activities.

Specify the user and organizational requirements
In most design processes, there is a major activity specifying the functional and other requirements for the product or system. For human-centred design, this activity should be extended to create an explicit statement of user and organizational requirements in relation to the context of use description. The following aspects should
be considered in order to identify relevant requirements:
a) Required performance of the new system against operational and financial objectives;
b) Relevant statutory or legislative requirements, including safety and health;
c) Co-operation and communication between users and other relevant parties;
d) The users’ jobs (including allocation of tasks, users’ well-being, and motivation);
e) Task performance;
f) Work design and organization;
g) Management of change, including training and personnel to be involved;
h) Feasibility of operation and maintenance;
i) The human-computer interface and workstation design.
User and organizational requirements should be derive and objectives set with appropriate trade-offs identified between the different requirements.
The specification of user and organizational requirements should:
a) Identify the range of relevant users and other personnel in the design;
b) Provide a clear statement of the human-centred design goals;
c) Set appropriate priorities for the different requirements;
d) Provide measurable criteria against which the emerging design can be tested;
e) Be confirmed by the users or those representing their interests in the process;
f) Include any statutory or legislative requirements;
g) Be adequately documented.

ISO 13407 does not address the general complexity and specific challenges related to systematic identification of different users, identification of the different goals that users may have; nor determination of measures (effectiveness, efficiency, satisfaction) of usability. The determination of environments of use is addressed in most detailed manner.

DISCUSSION
ISO 9241-11 and ISO 13407 are two important standards related to usability: the former one provides the definition of usability and the latter one guidance for designing usability. We carried out an interpretive analysis of ISO 13407 from the viewpoint of the standard definition of usability from ISO 9241-11. The results show that ISO 13407 provides only partly guidance for designing usability as presumed by the definition. Guidance for describing users and environments are provided but very limited guidance is provided for the descriptions of user goals and usabilty measures, and generally for the process of producing the various outcomes.

References that I may want to read further in future:

1. ANSI. Common Industry Format for Usability Test Reports., NCITS 354-2001, 2001.

3. Bevan, N., Claridge, N., Maguire, M. and Athousaki, M., Specifying and evaluating usability requirements using the Common Industry Format: Four case studies. in IFIP 17th World Computer Conference 2002 - TC 13 Stream on Usability: Gaining a Competitive Edge, (Montreal, Canada, 2002), Kluwer Academic Publishers, 133-148.

8. Hix and Hartson. Developing User Interfaces: Ensuring Usability Through Product & Process.
John Wiley & Sons, 1993.
9. ISO/IEC. 13407 Human-Centred Design Processes for Interactive Systems, ISO/IEC 13407: 1999 (E), 1999.

12. Jokela, T., Making User-Centred Design Common Sense: Striving for an Unambiguous
and Communicative UCD Process Model. in NordiCHI 2002, (Aarhus, Denmark, 2002), ACM, 19-26.
13. Jokela, T. and Iivari, N., Systematic Determination of Quantitative Usability Requirements. in to be published in the proceedings of HCI International 2003, (Crete, 2003).

15. Mayhew, D.J. The Usability Engineering Lifecycle. Morgan Kaufman, San Fancisco,
1999.
16. Nielsen, J. Usability Engineering. Academic Press, Inc., San Diego, 1993.
17. Rosson, M.B. and Carroll, J.M. Usability Engineering. Scenario-Based Development of Human-Computer Interaction. Morgan Kaufmann Publishers, 2002.

Aug 27 - Syvanen & Nokelainen, Evaluation of the technical and pedagogical mobile usability

Evaluation of the technical and pedagogical mobile usability.
Antti Syvänen. antti.syvänen@uta.fi Hypermedia Laboratory, University of Tampere, FIN-33014 Tampereen yliopisto, Finland.
Petri Nokelainen. petri.nokelainen@uta.fi Research Centre for Vocational Education, University of Tampere, FIN–13101 Hämeenlinna, Finland.

Attewell & Cavill-Smith (editor). Mobile Learning Anytime Everywhere. A book of papers from MLearn 2004. pg 191-185

Abstract
The major goal of the multidisciplinary Digital Learning 2 project is to develop an online tool to evaluate the usability of digital learning materials. This paper concentrates on describing the technical and pedagogical usability criteria and how they are specified in order to take account of the special features of mobile learning materials and environments. This was done by incorporating a ‘components of mobile learning’ (CML) evaluation model, which had been previously developed and empirically tested, into the criteria, after reviewing the other earlier categorisations for evaluation of mobile learning.


The major goal of the project is to develop a tool to evaluate the usability of digital learning materials and environments. In order to reach that goal, numerous concepts and models have been defined; for example, the criteria for technical and pedagogical usability and the Components of Mobile Learning (CML) model.


The following components of the technical usability criteria were specified (Nokelainen 2004):
1 accessibility
2 ‘learnability’ and memorability
3 user control
4 help
5 graphical layout
6 reliability
7 consistency
8 efficiency
9 memory load
10 errors.

In addition, the following pedagogical usability components were specified (Nokelainen 2004):
1 learner control
2 learner activity
3 cooperative learning
4 goal orientation
5 applicability
6 effectiveness
7 motivation
8 valuation of previous knowledge
9 flexibility
10 feedback.

The Components of Mobile Learning (CML) model was developed for evaluation of mobile learning. The model (Ahonen et al. 2002) consists of:
1 continuity and adaptability
2 learning as a personal process
3 contextuality in learning
4 accessibility
5 support for time and learning management
6 flexible interaction.

My Comments: This paper is very relevant as a benchmark (reference point) for my PhD research. Syvanen and Nokelainen had developed a usability evaluation tool, and used a set of usability criteria (comprising technical & pedagogical usability).

The criteria for evaluation of the technical and pedagogical mobile usability have been
developed in the following stages:
# investigation of the previous research reports dealing with the characteristics of mobility and
mobile learning (2001–2002)
# development of the first version of the CML model (2002–2003; Ahonen et al. 2002)
# development and empirical testing of the criteria (2003; Syvänen et al. 2003, 2004)
# review of earlier categorisations for the evaluation of mobile learning and incorporation
into the technical and pedagogical usability criteria (2004).


Table 1 Summary of the earlier categorisations for evaluation of mobile learning
>>>summarise Technical Usability from various sources
>>>summarise Pedagogical Usability from various sources.

Table 2 Evaluation of mobile learning materials and environments
>>>summarise Questions for each Technical/Pedagogical Usability Criteria.


Conclusions
In this paper we have specified the criteria for evaluation of mobile learning material in detail
and shown how mobile learning materials and environments are evaluated with an online evaluation tool. The real-life empirical testing of the evaluation tool was difficult to conduct as at
the time, there were only a few learning materials or environments defined solely for mobile use. However, the progress of developing such materials all over the world is rapid, and thus we see the mobile evaluation criteria as a meaningful part of the evaluation tool. In the future, we intend to develop the evaluation tool through empirical testing of real-life mobile learning materials and environments.

References that I may want to read further in the future:

Nokelainen P (2004). Conceptual definition of the technical and pedagogical usability criteria
for digital learning material. In Proceedings of ED–MEDIA 2004: World Conference on Educational Multimedia, Hypermedia and Telecommunications, Lugano, Switzerland, 2004(1), 4249–4254.

Aug 27 - Mostakhdemin-Hosseini & Tuimala, Mobile Learning Framework

MOBILE LEARNING FRAMEWORK.
Ali Mostakhdemin-Hosseini. Helsinki University of Technology, konemiehentie 2, Espoo Finland.
Jarno Tuimala. Innoforss Research &/ Development Center, Wanherinkatu 11 3 krs., 30100 Forssa, Finland.


Mobile learning is not learning through mobiles phones or learning over a wireless connection even though the capabilities of running multimedia features has increased in recent years. But mobile learning is the evolution of elearning, which completes the missing component of an e-learning solution. Mobile learning most suits for those mobile parties in education institutes. So, utilizing mobile devices in education is mainly considered as enhanced tools.

Unlike PC, mobile devices have some restriction for displaying content e.g. screen size and resolution. When designing user interface for mobile devices, especially for heterogeneous environments, we have to consider the special user requirements, as well as, the capabilities of the devices (Calvary et al. 2001) and (Roth et al. 2000). As (McClard et. Al. 2000) have stated, the portable devices are not a replacement for the PC.

There has been extensive progress in m learning since this study was initiated. Basically, the term has been defined more precisely (Brown, 2003).
1. Ubiquitous Computing- all members of the academic community have appropriate and timely access to the Internet, usually via a computer they own. Access may be either by desktop or laptop or handheld.
2. Portable Computing- same as #1 except computer must be laptop
3. Mobile Computing- same as #2 except laptop must be wireless
4. Very mobile Computing- same as #3 (i.e. wireless) except that the computer is a Palm Pilot, Blackberry or Equivalent.

Based on the users study conducted at Helsinki University of Technology and analyzed the gathered data (Mostakhdemin-Hosseini et.al, 2004) this study defined the framework of the mobile learning system. The Mobile learning system development is based on three main domains, Mobile usability, wireless technology and e-learning system.

Mobile usability is the main domain in the mobile learning system.
Issues, which involve mobile devices’ usability, are as such as mobile device type, mobile device features and mobile content design methods.
The contributions of mobile usability research to this study assists to identify the requirements of each mobile device capability to offer services in a usable manner. Providing services for mobile devices without considering the usability issues is a useless effort. Users utilize the mobile devices and use the services if they feel that services are usable and do not consume extra time. The main issue, which should be study carefully when developing education services for mobile devices, is the content of the mobile services, the services types, requirement of each service and the service limitations.

The second essential element of mobile learning system is the network connection, especially wireless network technology. Issues, which are related to the wireless networks which directly influence the mobile learning systems, are network infrastructure and operators rolls.

The unique feature of the m- learning system is the mobility. The users utilize the mobile devices if the network provides fast, secure and reliable network connections. Also the users are willing to pay a reasonable amount of money for the services they receive.


The final domain, which affects the mobile learning system, is the existing e-learning concept. The mobile devices were enhanced to the existing e-learning systems. Issues related to e-learning, which affects the mobile learning system, is the requirements of the e- learning system and the type of utilized e-learning platform.

...mobile learning system’s developer has to consider the following issues for developing the mobile learning system.
• Define the education components and services, required by the mobile learning system. These
components and services depend to the type of course.
• Identify the existing wireless network capabilities and boundaries. If there are different networks, select the one, which is most appropriate for the education components and mobile devices capabilities.
• Determine the types of mobile devices, which are intended to be utilized in the system.
• Distribute the education components and services based on network capabilities and usability requirements of each mobile device.
• Write scenarios, which express the mobile learning system. In the scenarios all the components are distributed to appropriate devices with consideration of their capabilities and the usability
requirements.
• Design prototype based on the scenarios. It is important to follow the usability guideline for each device very carefully.
• Test and validate the concept and service distributions to different devices.

References that I may read further in the future:

Mostakhdemin-Hosseini Ali, Mustajärvi, Jari “Steps Required Developing Mobile Learning Services” , International association for development of Information Society, IADIS July 17-19th, 2004, Avila, Spain
Mostakhdemin-Hosseini Ali, Mustajärvi, Jari “Evaluation of Mobile Learning System Prototype”, International association for development of Information Society, IADIS February 23-25th, 2005, Algarve, Portugal

Aug 25 - Karevaara, From educational usability to context-specific teachability...

Katja Karevaara, Media Education Centre, University of Helsinki.
From educational usability to context-specific teachability: Development and use of the network-based teaching material contents in higher engineering education.

Central concept and target for the research is "teachability". Also "content teachability"
will be used as a synonym. The reason why the concept "teachability" is chosen for this research is that often the term "pedagogical usability" (e.g. Melis et al. 2003) is still rather connected to the term "usability".

Pedagogical usability is often focusing on the aspects whether the interface, tools, content, and the tasks of the e-learning environments support learning in various contexts according to certain pedagogical objectives (Silius et al. 2003, Tervakari et al. 2002).

"Studiability" of the materials is close to the pedagogical usability.

The term "usability" has a stable status in engineering sciences in analysing the use of the technical interface of a software application (e.g. Nielsen 2000), which viewpoint is not central in this research.
In this research teachability is a sub-concept for "adaptability" introduced by Ruokamo et al. (2005). According to them, adaptability can be seen as a general approach for improving the usability of network-based teaching materials. Adaptability could be called as pedagogical usability. However, their approach for the usability is deeper connected to the pedagogical issues as they argue that it is not confined to the enhancement of user interfaces but rather is a means of facilitating the teaching-studying-learning –process. (Ruokamo et al. 2005) Figure 1. will present the relationships of pedagogical usability in this study. In the figure the viewpoints for adaptability are based on the arguments by Tella et al. (2004) and Ruokamo et al. (2005).
Teachability in this reserach means the efficiency of the subject matters. It can be also described as teachers' creative and relevant use of the network-based teaching material contents. Concept is related to the flexi-mode approach where the use of the contents is analysed in the didactical framework of the whole course. However, eventually teachers will internally validitate the concept teachability.

Research in pedagogical usability has been active recently. For example Kukulska-Hulme et al. (2004) conducted a project during 2001-2003 in Open University (UK). During the project it was recognised that "to get to the heart of pedagogical usability, we have to understand more about the impact of reguirements in relation to communities, contexts and disciplines." The research group found therefore several layers of usability: contextspecific, academic, general and technical. In detail:
• Context specific usability relates to the requirements of particular disciplines and courses.
• Academic usability deals with educational issues, such as pedagogical strategy.
• General usability issues are common to most websites and include aspects such as clear navigation and accessibility for users with special needs.
• Technical usability addresses issues such as broken links and server reliability.

My comments: He was a PhD student at the time of writing as he talked about his Doctoral dissertation.

References that I may want to read further in future:

Kukulska-Hulme, A. & Shields, L. (2004). The keys to usability in e-learning websites. Networked learning conference. Lancaster.

Melis, E., Weber, M. & Andrès, E. (2003). Lessons for (Pedagogic) Usability of eLearning Systems. World Conference on E-Learning in Corporate, Government, Healthcare, & Higher education. 2003(1), 281-284.

Nielsen, J. (2000). Designing web usability: the practice of simplicity. Indianapolis: New riders publishing.