COGNITIVE TASKS:COGNITIVE TASK ANALYSIS (CTA)

COGNITIVE TASK ANALYSIS (CTA)

Ergonomics interventions in terms of man–machine interfaces, cognitive aids, or training programs require a thorough understanding of the work constraints and user strategies. Work constraints can range from system constraints (e.g., time pressure and conflicting goals) to cognitive constraints or limitations and use constraints imposed by the tool itself (e.g., an interface or a computer program). On the other hand, user strategies can range from informal heuristics, retrieved from similar situations experienced in the past, to optimization strategies for unfamiliar events. The analysis of work con- straints and use strategies is the main objective of cognitive task analysis (CTA).

Cognitive task analysis involves a consideration of user goals, means, and work constraints in order to identify the what, how, and why of operator’s work (Rasmussen et al. 1994; Marmaras and Pavard 1999). Specifically, CTA can be used to identify:

• The problem-solving and self-regulation strategies* adopted by operators

• The problem-solving processes followed (heuristics)

• The specific goals and subgoals of operators at each stage in these processes

• The signs available in the work environment (both formal and informal signs), the information carried by them, and the significance attached to them by humans

• The regulation loops used

• The resources of the work environment that could help manage workload

• The causes of erroneous actions or suboptimal performance CTA differs from traditional task analysis, which describes the performance demands imposed upon the human operator in a neutral fashion (Drury et al. 1987; Kirwan and Ainsworth 1992) regardless of how operators perceive the problem and how they choose their strategies. Furthermore, CTA differs from methods of job analysis that look at occupational roles and positions of specific personnel categories (Davis and Wacker 1987; Drury et al. 1987).

A Framework for Cognitive Task Analysis

Cognitive task analysis deals with how operators respond to tasks delegated to them either by the system or by their supervisors. Tasks is used here to designate the operations undertaken to achieve certain goals under a set of conditions created by the work system** (Leplat 1990). CTA looks at several mental activities or processes that operators rely upon in order to assess the current situation, make decisions, and formulate plans of actions. CTA can include several stages:

1. Systematic observation and recording of operator’s actions in relation to the components of the work system. The observed activities may include body movements and postures, eye movements, verbal and gestural communications, etc.

2. Interviewing the operators with the aim of identifying the why and when of the observed actions.

3. Inference of operator’s cognitive activities and processes.

4. Formulation of hypotheses about operator’s competencies,*** by interpreting their cognitive activities with reference to work demands, cognitive constraints and possible resources to manage workload.

5. Validation of hypotheses by repeating stages 1 and 2 as required.

Techniques such as video and tape recording, equipment mock-ups, and eye tracking can be used to collect data regarding observable aspects of human performance. To explore what internal or cognitive processes underlie observable actions, however, we need to consider other techniques, such as thinking aloud while doing the job (i.e., verbal protocols) and retrospective verbalizations. For high-risk industries, event scenarios and simulation methods may be used when on-site observations are difficult or impossible.

CTA should cover a range of work situations representing both normal and degraded conditions. As the analysts develop a better image of the investigated scenario, they may become increasingly aware of the need to explore other unfamiliar situations. Cognitive analysis should also be expanded into how different operating crews respond to the same work situation. Presumably, crews may differ in their performance because of varying levels of expertise, different decision-making styles, and different coordination patterns (see, e.g., Marmaras et al. 1997).

* Self-regulation strategies are strategies for deciding how to adapt to different circumstances, monitor complete or interrupted tasks, and detect errors.

** The work system consists of the technological system, the workplace, the physical environment, the organizational and management system, and the socioeconomic policies.

*** Competencies is used here to designate the specific cognitive strategies and heuristics the operators develop and use to respond to the task’s demands within the constraints imposed by a specific work environment.

The inference of cognitive activities and formulation of hypotheses concerning user competencies requires familiarity with models of human cognition offered by cognitive psychology, ethnology, psycholinguistics, and organizational psychology. Several theoretical models, such as those cited earlier, have already found practical applications in eliciting cognitive processes of expert users. Newell and Simon’s (1972) human problem-solving paradigm, for instance, can be used as a back- ground framework for inferring operator’s cognitive processes when they solve problems. Hutchins’s (1990; 1992) theory of distributed cognition can support analysts in identifying operator resources in managing workload. Rasmussen’s (1986) ladder model of decision making can be used to examine how people diagnose problems and evaluate goals. Norman’s (1988) action-cycle model can be used to infer the cognitive activities in control tasks. Finally, Reason’s (1990) model of human errors can be used to classify, explain, and predict potential errors as well as underlying error-shaping factors. Human performance models, however, have a hypothetical rather than a normative value for the analyst. They constitute his or her background knowledge and may support interpretation of observ- able activities and inference of cognitive activities. Cognitive analysis may confirm these models (totally or partially), enrich them, indicate their limits, or reject them. Consequently, although the main scope of cognitive analysis is the design of artifacts and cognitive advisory systems for complex tasks, it can also provide valuable insights at a theoretical level.

Models of human cognition and behavior can provide practical input to ergonomics interventions when cast in the form of cognitive probes or questions regarding how operators search their envi- ronment, assess the situation, make decisions, plan their actions, and monitor their own performance. Table 1 shows a list of cognitive probes to help analysts infer these cognitive processes that underlie observable actions and errors.

This need to observe, interview, test, and probe operators in the course of cognitive analysis implies that the role of operators in the conduct of analysis is crucial. Inference of cognitive activities and elicitation of their competencies cannot be realized without their active participation. Conse- quently, explicit presentation of the scope of analysis and commitment of their willingness to provide information are prerequisites in the conduct of CTA. Furthermore, retrospective verbalizations and online verbal protocols are central to the proposed methodology (Ericsson and Simon 1984; Sander- son et al. 1989).

CTA permits the development of a functional model of the work situation. The functional model should:

1. Describe the cognitive constraints and demands imposed on operators, including multiple goals and competing criteria for the good completion of the task; unreliable, uncertain, or excessive information to make a decision; and time restrictions.

2. Identify situations where human performance may become ineffective as well as their potential causes—e.g., cognitive demands exceeding operator capacities and strategies that are effective under normal situations but may seem inappropriate for the new one.

3. Identify error-prone situations and causes of errors or cognitive biases—e.g., irrelevant or superfluous information, inadequate work organization, poor workplace design, and insufficient knowledge.

4. Describe the main elements of operator’s competencies and determine their strengths and weaknesses.

5. Describe how resources of the environment can be used to support the cognitive processes.

The functional model of the work situation can provide valuable input into the specification of user requirements, prototyping, and evaluation of cognitive aids. Specifically, based on elements 1, 2, and 3 of the functional model, situations and tasks for which cognitive aid would be desirable and the ways such aid must be provided can be determined and specified. This investigation can be made responding to questions such as:

• What other information would be useful to the operators?

• Is there a more appropriate form in which to present the information already used as well as the additional new information?

• Is it possible to increase the reliability of information?

• Could the search for information be facilitated, and how?

• Could the treatment of information be facilitated, and how?

• Could we provide memory supports, and how?

• Could we facilitate the complex cognitive activities carried out, and how?

• Could we promote and facilitate the use of the most effective diagnosis and decision-making strategies, and how?

• Could we provide supports that would decrease mental workload and mitigate degraded per- formance, and how?

• Could we provide supports that would decrease human errors occurrence, and how?

Cognitive aids can take several forms, including memory aids, computational tools, decision aids to avoid cognitive biases, visualization of equipment that is difficult to inspect, and situation- assessment aids. The functional model is also useful in designing man–machine interfaces to support retrieval of solutions and generation of new methods. By representing system constraints on the interface, operators may be supported in predicting side effects stemming from specific actions. On the other hand, the functional model can also be useful in specifying the competencies and strategies required in complex tasks and hence providing the content of skill training.

Furthermore, based on the information provided by elements 4 and 5 of the functional model, the main features of the human–machine interface can also be specified, ensuring compatibility with operators’ competencies. The way task’s objects should be represented by the system, the type of man–machine dialogues to be used, the procedures to be proposed, and generic or customizable elements of the system are examples of human–computer interface features that can be specified using the acquired data.

Close cooperation among ergonomics specialists, information technology specialists, and stake- holders in the design project is required in order to examine what system functions should be sup- ported by available information technology, what features of the human-computer interface should be realized, and what functions should be given priority.

Techniques for Cognitive Task Analysis

CTA can be carried out using a variety of techniques, which, according to Redding and Seamster (1994), can include cognitive interviewing, analysis of verbal protocols, multi-dimensional scaling, computer simulations of human performance, and human error analysis. For instance, Rasmussen (1986) has conducted cognitive interviews to examine the troubleshooting strategies used by elec- tronics technicians. Roth et al. (1992) have used cognitive environment simulation to investigate cognitive activities in fault management in nuclear power plants. Seamster et al. (1993) have carried out extensive cognitive task analyses to specify instructional programs for air traffic controllers. These CTA techniques have been used both to predict how users perform cognitive tasks on prototype systems and to analyze the difficulties and errors in already functioning systems. The former use is associated with the design and development of user interfaces in new systems, while the latter use is associated with the development of decision support systems or cognitive aids and training pro- grams.

The results of CTA are usually cast in the form of graphical representations that incorporate the work demands and user strategies. For cognitive tasks that have been encountered in the past, op- erators may have developed well-established responses that may need some modifications but nev- ertheless provide a starting framework. For unfamiliar tasks that have not been encountered in the past or are beyond the design-basis of the system, operators are required to develop new methods or combine old methods in new ways. To illustrate how the results of CTA can be merged in a graphical form, two techniques are presented: hierarchical task analysis and the critical decision method.

Hierarchical Task Analysis

The human factors literature is rich in task analysis techniques for situations and jobs requiring rule- based behavior (e.g., Kirwan and Ainsworth 1992). Some of these techniques can also be used for the analysis of cognitive tasks where well-practiced work methods must be adapted to task variations and new circumstances. This can be achieved provided that task analysis goes beyond the recom- mended work methods and explores task variations that can cause failures of human performance. Hierarchical task analysis (Shepherd 1989), for instance, can be used to describe how operators set goals and plan their activities in terms of work methods, antecedent conditions, and expected feed- back. When the analysis is expanded to cover not only normal situations but also task variations or changes in circumstances, it would be possible to record possible ways in which humans may fail and how they could recover from errors. Table 2 shows an analysis of a process control task where operators start up an oil refinery furnace. This is a safety-critical task because many safety systems are on manual mode, radio communications between control room and on-site personnel are intensive, side effects are not visible (e.g., accumulation of fuel in the fire box), and errors can lead to furnace explosions.

A variant of hierarchical task analysis has been used to examine several cognitive activities, such as goal setting and planning, and failures due to slips and mistakes. Variations in human performance were examined in terms of how teams in different shifts would perform the same task and how the same team would respond to changes in circumstances. A study by Kontogiannis and Embrey (1997) has used this technique to summarize findings from online observations of performance, interviews with process operators about their work methods, near-miss reviews, and critical incident analysis. The task analysis in Table 2 has provided valuable input in revising the operating procedures for start-up: the sequence of operations was reorganized, contingency steps were included for variations in circumstances, check boxes were inserted for tracking executed steps, and warnings and cautions were added to prevent human errors or help in their detection and recovery. In addition, the task analysis in Table 2 has been used to redesign the computer-based process displays so that all infor- mation required for the same task could be grouped and presented in the same screen. For instance, the oxygen content in flue gases is an indicator of the efficiency of combustion (see last row in Table

2) and should be related to the flow rates of air and fuel; this implies that these parameters are functionally related in achieving the required furnace temperature and thus should be presented on the same computer screen. The analysis of cognitive tasks, therefore, may provide input into several forms of human factors interventions, including control panel design, revision of operating proce- dures, and development of job aids and training.

Critical Decision Method

The Critical Decision Method (CDM) (Klein et al. 1989) is a retrospective cognitive task analysis based on cognitive interviews for eliciting expert knowledge, decision strategies and cues attended to, and potential errors. Applications of the CDM technique can be found in fireground command, tactical decision making in naval systems, ambulance emergency planning, and incident control in offshore oil industries. The technique relies on subject matter experts (SMEs) recalling a particularly memorable incident they have experienced in the course of their work. The sequence of events and actions are organized on a timeline that can be rearranged as SMEs remember other details of the

incident. The next stage is to probe SMEs to elicit more information concerning each major decision point. Cognitive probes address the cues attended to, the knowledge needed to make a decision, the way in which the information was presented, the assessment made of the situation, the options considered, and finally the basis for the final choice. The third stage of the CDM technique involves comparisons between experts and novices. The participants or SMEs are asked to comment on the expected performance of a less-experienced person when faced with the same situation. This is usually done in order to identify possible errors made by less experienced personnel and potential recovery routes through better training, operating procedures, and interface design. The results of the cognitive task analysis can be represented in a single format by means of a decision analysis table.

One of the most important aspects of applying the CDM technique is selecting appropriate inci- dents for further analysis. The incidents should refer to complex events that challenged ordinary operating practices, regardless of the severity of the incident caused by these practices. It is also important that a ‘‘no-blame’’ culture exist in the organization so that participants are not constrained in their description of events and unsuccessful actions. To illustrate the use of the CDM technique, the response of the operating crew during the first half hour of the Ginna nuclear power incident is examined below, as reported in Woods (1982) and INPO (1982).

The operating crew at the Ginna nuclear plant encountered a major emergency in January 1982 due to a tube rupture in a steam generator; as a result, radioactive coolant leaked into the steam generator and subsequently into the atmosphere. In a pressurized water reactor such as Ginna, water coolant is used to carry heat from the reactor to the steam generator (i.e., the primary loop); a secondary water loop passes through the steam generator and the produced steam drives the turbine that generates electricity. A water leak from the primary to the secondary loop can be a potential risk when not isolated in time. In fact, the delayed isolation of the faulted or leaking steam generator was one of the contributory factors in the evolution of the incident. Woods (1982) uses a variant of the CDM technique to analyze the major decisions of the operating crew at the Ginna incident.

Figure 7 shows a timeline of events and human actions plotted against a simplified version of the SRK model of Rasmussen. Detection refers to the collection of data from control room instruments and the recognition of a familiar pattern; for instance, the initial plant symptoms leads to an initial recognition of the problem as a steam generator tube rupture (i.e., a SGTR event). However, alter- native events may be equally plausible, and further processing of information at the next stage is required. Interpretation then involves identifying other plausible explanations of the problem, pre- dicting the criticality of the situation, and exploring options for intervention. The formulation of

specific actions for implementing a plan and executing actions are carried out at the following stage of control. The large number of actions at the control stage provides a context for the work conditions that may affect overall performance; for instance, high workload at this stage could prevent the crew from detecting emerging events. The feedback stage is similar to the detection stage because both are based on the collection of data; however, in the detection stage, observation follows an alarm, while in the feedback stage, observation is a follow-up to an action. The decision flowchart in Figure 7 has been developed on the basis of cognitive interviews with the operators involved and investi- gations of the operating procedures in use.

To understand the cognitive processes involved at critical decision points, analysts should employ several cognitive probes similar to those shown in Table 1. It is worth investigating, for instance, the decision of the crew to seek further evidence to identify the tube rupture in one of the two steam generators (Figure 7, column 3). At this point in time, the crew was aware that the level in the B steam generator was increasing more rapidly than the level of the A steam generator (i.e., due to tube rupture) with auxiliary feedwater flows established to both steam generators. However, the crew needed additional evidence to conclude their fault diagnosis. Was this delay in interpretation caused by insufficient plant knowledge? Did the crew interpret the consequences of the problem correctly? Were the instructions in the procedures clear? And in general, what factors influenced this behavior of the crew?

These are some cognitive probes necessary to help analysts explore the way that the crew inter- preted the problem. As it appeared, the crew stopped the auxiliary feedwater flow to the B steam generator and came to a conclusion when they observed that its level was still increasing with no input; in addition, an auxiliary operator was sent to measure radioactivity in the suspect steam gen- erator (Figure 7, column 3). Nevertheless, the crew isolated the B steam generator before feedback was given by the operator with regard to the levels of radioactivity. The cognitive interviews estab- lished that a plausible explanation for the delayed diagnosis was the high cost of misinterpretation. Isolation of the wrong steam generator (i.e., by closing the main steam isolation valve [MSIV]) would require a delay to reopen it in order to repressurize and establish this steam generator as functional. The crew also thought of a worse scenario in which the MSIV would stick in the closed position, depriving the crew of an efficient mode of cooldown (i.e., by dumping steam directly to the con- densers); in the worst-case scenario, the crew would have to operate the atmospheric steam dump valves (ASDVs), which had a smaller cooling function and an increased risk of radiological release.

Other contextual factors that should be investigated include the presentation of instructions in the procedures and the goal priorities established in training regimes. Notes for prompt isolation of the faulty steam generator were buried in a series of notes placed in a separate location from the instruc- tions on how to identify the faulty steam generator. In addition, the high cost of making an incorrect action (an error of commission) as compared to an error of omission may have contributed to this delay. In other words, experience with bad treatment of mistaken actions on the part of the organi- zation may have prompted the crew to wait for redundant evidence. Nevertheless, the eight-minute delay in diagnosis was somewhat excessive and reduced the available time window to control the rising water level in the B steam generator. As a result, the B steam generator overflowed and contaminated water passed through the safety valves; subsequent actions (e.g., block ASDV, column

5) due to inappropriate wording of procedural instructions also contributed to this release.

Cognitive analysis of the diagnostic activity can provide valuable insight into the strategies employed by experienced personnel and the contextual factors that led to delays and errors. It should be emphasized that the crew performed well under the stressful conditions of the emergency (i.e., safety-critical event, inadequate procedures, distractions by the arrival of extra staff, etc.) given that they had been on duty for only an hour and a half. The analysis also revealed some important aspects of how people make decisions under stress. When events have high consequences, experienced personnel take into account the cost of misinterpretation and think ahead to possible contingencies (e.g., equipment stuck in closed position depriving a cooling function). Intermingling fault interpretation and contingency planning could be an efficient strategy under stress, provided that people know how to switch between them. In fact, cognitive analysis of incidents has been used as a basis for developing training programs for the nuclear power industry (Kontogiannis 1996).

The critical decision method has also been used in eliciting cognitive activities and problemsolving strategies required in ambulance emergency planning (O’Hara et al. 1998) and fire-fighting command (Klein et al. 1986). Taxonomies of operator strategies in performing cognitive tasks in emergencies have emerged, including maintaining situation assessment, matching resources to situation demands, planning ahead, balancing workload among crew members, keeping track of unsuccessful or interrupted tasks, and revising plans in the light of new evidence. More important, these strategies can be related to specific cognitive tasks and criteria can be derived for how to switch between alternative strategies when the context of the situation changes. It is conceivable that these problem-solving strategies can become the basis for developing cognitive aids as well as training courses for maintaining them under psychological stress.