Poster for HCI International '97, San Francisco, CA

Following common practice of usability engineering

A common practice in User Interface specification is to write down the user tasks. This practice is well known as Task Analysis. Typically, the user tasks are specified hierarchically, using general concepts at the top level, using specific user goals and methods at the interim levels and using detailed description of the operational procedures at the bottom levels. Typically, the user task breakdown thus obtained is used also for defining the Help/Topics feature, used for on-line user assistance.

When using ErgoLight, The User Interface designer is required to store the task breakdown, which is typically written down as required when using the method of task analysis, in a special database delivered with ErgoLight.

User task database

ErgoLight� stores a breakdown of all user tasks supported by the software application in a database, thereby providing a common source of user interface specification.

The process of user interface specification using the task breakdown database is very effective in detecting usability problems as early as at the specification stage.

The data structure is hierarchical, following methodologies of task analysis commonly accepted by the community of human factors engineers. The task hierarchy includes: main tasks, sub tasks, goals, methods, operation and context.

The hierarchical data structure is perceived as intuitive and easy to follow by end users.

Reusing the user information

ErgoLight provides a means for converting data from the Windows clipboard to lists in the task break down. This feature is useful to reuse task related information stored in any source of user documentation, such as the user documentation

Reusing the user information saves the overhead of data entry of the task break down, shortening the task break down procedure to few hours.

Prototyping

Prototyping involves the following activities:

�      Definition of the user operation:

�    Procedure implementation: association of the steps of the operation procedures to the Windows controls of the product implementation

�    Definition of the system modes that restrict the applicability of the user tasks.

�      Definition of the indicators for user difficulty.

Connection by point and click

ErgoLight� connects each step of an operational procedure to a set of alternative user interface components, such as menu items, buttons and shortcut keys, used to actuate the procedure step. ErgoLight �provides easy connection, by running the Windows application simultaneously, so that the user interface designer need just to point and click on the component chosen to implement a procedure step.

Connection by point and click is very fast and very reliable, shortening the connection procedure to few hours.

Task verification

ErgoLight� identifies defects in the user task break down database, including:

�       Tasks that are not properly defined, namely, those entities in the task breakdown that have only few children, or not at all

�       User interface components that are �orphans�, namely, those components that do not have any corresponding procedure step or parameter in the task breakdown. Typically, the effect of each user interface component should be reflected in the task breakdown. Therefore, the existence of an orphan may suggest that either the task breakdown is not fully specified or that the user interface component is redundant.

Task verification allows detection of defects in the User Interface as early as at the prototype stage.

Windowing the system modes

ErgoLight identifies mode errors, if the system mode is �visible� to Windows. Visibility to Windows means that a Windows control, such as a check box or a radio button, is associated with the system mode and that Windows has the handle for this control. For example, ErgoLight identifies the� �print to file� mode by checking the �print to file� check box of the �Print� dialog box.

At the mode definition phase, the User Interface designer is required to store the dependence of the user tasks on the Windows controls that hold the system mode. The assignment is made by �point and click�.

Automated identification of mode errors is useful for on-line recovery from confusing situations, when the application behaves not according to the user�s expectations.

Types of Problem Indicators

ErgoLight �uses two types of problem indicators:

�      The user time response delay. It is the Test Manager who assigns threshold values for the user time delay, for deciding when a delay in the user response will be considered as a candidate for an instance of user difficulty. The user interface designer is required to assign a default value for this threshold

�      The user activation of particular Windows controls, used by the user interface designer to implement Cancel, Undo and Help features. For example, the user activation of the Edit/Undo menu item is typically an indicator of a candidate for an instance of user difficulty.

Problem indicators allow computer initiated reporting, essential for capturing many instances of user difficulty that users tend to ignore.

Following common practice of usability testing

A common practice in Beta testing is to group the users according to their role in using the Windows application. ErgoLight supports this practice by providing means for assigning setup parameters to user groups. The setup parameters supported are the user time delay threshold and the setup of the level of intervention in case that a putative instance of user difficulty is identified.

Classification of the test results by user groups is essential for analyzing the benefits of design features.

User initiated problem reporting

While the end user operates the Windows application, ErgoLight runs in the background. The end user can invoke a reporting session by activating a special key combination, assigned at design time by the dialog designers.

The user can initiate a reporting session any time during the operation of the Windows application.

Reporting the user intention

Problem reporting is computer instructed. ErgoLight prompts the end user to specify the user intention by a sequence of list selections. Each reporting session starts with a list tasks associated with the most recent user action. The end user can select a task from the list, or else, to obtain another list of main tasks, followed by sub task, goal, method, procedure step and context. The sequence of list selections follows the hierarchy of the task break down.

The top-down sequence of list selections make the reporting on a usability problem short and easy to follow.

Understanding the user expectations

ErgoLight prompts the user to report on intentions not found in the task breakdown database. The information reported thus is useful:

�       To learn about useful features not implemented in the software application

�       To identify defects in the task breakdown stored in the user task database

�       To exhibit problems in the user information, such as terminology problems, that prevent the users from understanding the actual capability of the software application.

Exceptional user expectations are typical to users new to the Windows application. The information acquired is similar to that obtained using the traditional �think aloud� technique, but, due to automation, the reporting is much more efficient.

Reporting problems� in the user information

ErgoLight prompts the user to report on operational procedure they could not find in the user information. The user is asked to specify the source of information s/he tried, such as the User�s Guide, the on-line Help and the Tutorial.

The classification of user information problems facilitates the decision making regarding fixes and the distribution of the problem reports to the persons that should fix the problems.

Tracing the user actions

ErgoLight� runs in the background of the Windows application, captures the user actions and stores them in the User Action database.

The user actions are used for automated and manual analysis of the reasons for the user confusion, at run time and at the evaluation phase

Identifying instances of user confusion

Using the problem indicators specified at the design phase, ErgoLight analyzed the user actions on line, identifying situations of the user experience of difficulty in operating the Windows application. ErgoLight initiates a dialog with the end user whenever an instance of possible user confusion is indicated.

The automatic detection of user confusion allows to reduce to minimum the interference with normal operation flow.

Dialog control

Each time a possible user confusion is indicated, ErgoLight prompts the user to either start a reporting session or to resume normal operation. The user always has full control over the operation flow, reducing the interference with normal operation flow down to the barest minimum.

The user control over the operation flow is essential for goal driven operation.

Reducing false alarms

False alarms are situations when a user difficulty is wrongly identified, namely, when a problem is indicated although the user does not feel that s/he has any problem at all. Example of false alarms are:

�      The activation of a Help feature for exploring the behavior of the Windows application at the initial learning phase

�      The activation of a Cancel control in a process of learning by �trial and error�

�      Response delay that occur when the end user pauses operation, because s/he was doing something else.

False alarms are typical of the phase of initial learning the Windows application and they interfere with the user learning process. ErgoLight allows the user to prevent false alarms by disabling the problem indicators. ErgoLight provides three levels of false alarm prevention.

�      A testing administrator can disable the problem indicators for a whole session

�      The end user can disable the problem indicators for a sub session

�      In normal operation, when the problem indicators are enabled, the user can bypass them using special bypass key combinations, assigned at design time by the dialog designers.

The hierarchy of three level false alarm prevention allows fine tuning the computer intervention, to balance between the need for maximal reporting opportunity and minimal interference with the user learning process.

On-line user assistance

ErgoLight� provides to the end user various services of on-line assistance, including:

�      Automated back tracking of recent user actions, that the user can compare to his intended action

�      Automated analysis of the reasons for unexpected behavior of the Windows application, including a list of attributes that could affect the behavior of the software application.

The on-line support service reinforces the users to report on problems they encounter.

Backtrack

ErgoLight �identifies situations of user confusion that rarely occur. The identification of such situations is provided by on-line detection and backtrack of the user actions. At the analysis stage, the evaluator can identify those instances which resulted in user confusion that significantly hampered the user performance and to backtrack the user actions. ErgoLight �identifies possible reasons for the problem, if the reason is sensitivity of the software application to psychomotoric user errors, or to mode errors.

Many of the severe usability problems are hard to detect, because they are rare and because they are hard to retrieve.

Problem classification

ErgoLight classifies the reported problems by their type, as follows:

�      Exceptional user intention: The user intention is not found in the task break down. Exceptional user intention problems include user tasks not supported by the application, user tasks omitted from the task break down or user tasks specified using terms different from those which the end user uses

�      Unexpected application response: The response of the Windows application to the user action was not as the user has expected. Unexpected application response problems include problems due to the sensitivity of the user interface to the user errors and problems due to mode errors. Unexpected application response is typical to users that have some experience in using the application

Problem classification is used to derive different reports, each aimed to those persons that should take care of fixing the problems.

Statistics

ErgoLight �collects statistics regarding all identified usability problems, and quantifies the usability problems in terms of the percentage of total time waste.

The total time waste is a measure of costs. Using statistics, a designer can:

�      Prioritize the usability problems according to the costs saved by fixing them

�      Compare alternative implementations of a design feature

Controversial design decisions can be judged using objective measures, based on their real costs.

Adjusting the application terminology

ErgoLight�  provides integrated �Search and Replace�, specialized for controlled changing of terms used in the user task breakdown by those that the end users used in order to describe their intention when they encounter a usability problem.

Much of the confusion that end users typically encounter is due to the use of technical terms instead of user terms.

Knowledge transfer

ErgoLight�  provides means for the test evaluator to transfer the knowledge which s/he gains while going through the usability problems to the designers and to the Help Desk personnel. The knowledge is recorded as comments to the usability problems that can be organized later on by the type of usability problem and the user task.