COGNITIVE TASKS:DESIGNING COGNITIVE AIDS FOR COMPLEX COGNITIVE TASKS

DESIGNING COGNITIVE AIDS FOR COMPLEX COGNITIVE TASKS

Recent developments in information technology and system automation have provided challenging occasions for developing computerized artifacts aiming to support cognitive processes such as di- agnosis, decision making, and planning. This enthusiasm with the capabilities of new technology has resulted in an overreliance on the merits of technology and the development of stand-alone systems capable of undertaking fault diagnosis, decision making, and planning. Recent reviews of such in- dependent computer consultants (e.g., Roth et al. 1997), however, have found them to be difficult to use and brittle in the face of novel circumstances. Woods et al. (1990) argue that many of these systems cannot anticipate operator responses, provide unsatisfactory accounts of their own goals, and cannot redirect their line of reasoning in cases of misperception of the nature of the problem. To make cognitive aids more intelligent and cooperative, it is necessary to examine the nature of human cognition in complex worlds and its coupling with the wider context of work (Marmaras and Pavard 2000). Because of the increasing interest in developing cognitive aids for complex tasks, this section presents two case studies that address this issue in an applied context.

The Case of a Cognitive Aid for CNC Lathe Programming

The scope of this study was to design a cognitive aid for CNC lathe programming (Marmaras et al. 1997) by adopting a problem-driven approach. This approach combines an analysis of the task in terms of constraints and cognitive demands with an analysis of user strategies to cope with the problem (Marmaras et al. 1992; Woods and Hollnagel 1987). A cognitive task analysis was under- taken in the first place in order to identify the cognitive demands of the task and investigate likely problem-solving strategies leading to optimal and suboptimal solutions. On the basis of the cognitive task analysis and the results of a follow-up experiment, an information technology cognitive aid was developed for CNC lathe programming. A prototype of the cognitive aid was evaluated in a second experiment.

Cognitive Task Analysis

Programming a CNC lathe requires the development of a program that will guide the lathe to trans- form a simple cylindrical part into a complex shape. This cognitive task requires planning and codification, and it is very demanding due to several interdependent constraints, competing criteria for task efficiency, delayed feedback, and lack of memory aids for maintaining a mental image of the manufacturing process.

A cognitive analysis of the activities and strategies was conducted on the basis of real-time observations of experienced operators programming their CNC lathes and off-the-job interviews. The analysis identified three main cognitive tasks:

1. Planning the whole process of manufacturing, which entails deciding upon:

• The order of cutting several elements of the complex shape

• The cutting tool to be used at each stage of the cutting process

• The movement of the tool at each stage of the cutting process (e.g., starting and ending points, type of the movement, and speed)

• The number of iterative movements of the tool at each stage

• The speed of rotation of the part

2. Codification of the manufacturing process to the programming language of the CNC machine.

The operator has to consider the vocabulary of this language to designate the different objects

and functions of the manufacturing process, as well as the grammar and syntax of the language.

3. Introduction of the program to the machine by using various editing facilities and by starting the execution of the program. It is worth noting that at this stage certain omission and syntax errors can be recognized by the computer logic and the operator may be called upon to correct them.

Table 3 summarizes the results of the cognitive analysis for the planning task. Experienced op- erators must take a number of constraints into account when deciding upon the most suitable strategy to manufacture the part. These include:

• The shape of the part and the constraints imposed by its material.

• The constraints of the machine tool, e.g., the rotating movement of the part, the area and movement possibilities of the cutting tool, its dimensions, and its cutting capabilities.

• The product quality constraints derived from the specification; these are affected by the speeds of the part and tools, the type of tools, and the designation of their movements.

• The manufacturing time, which is affected by the number of tool changes, the movements the tools make without cutting (idle time of the tool) and the speeds of the part and cutting tools.

• The safety considerations, such as avoiding breaking or destroying the part or the tools; safety rules concerning the area of tool movement and the speeds of the part and cutting tools.

An analysis of how experienced operators decide to use the CNC lathe in cutting parts into complex shapes revealed a variety of problem strategies ranging from serial to optimized strategies. The simplest strategy, at one end of the spectrum, involved cutting each component of the part having a different shape separately and in a serial order (e.g., from right to left). A number of heuristics have been introduced in the problem-solving process resulting in such a serial strategy, including:

• Decomposition of the part to its components (problem decomposition)

• Specification and prioritization of several criteria affecting task efficiency (criteria prioritization)

• Simplification of mental imagery of the manufacturing process (mental representation)

• Use of a unique frame of reference regarding the part to be manufactured (frames of reference minimization)

At the other end of the spectrum, a more complex strategy was adopted that relied on optimizing a number of criteria for task efficiency, such as product quality, manufacturing time, and safety considerations. A case in point is a cutting strategy that integrates more than one different shape into one tool movement, which alters the serial order of cutting the different shapes of the part. This strategy would decrease the time-consuming changes of tools and the idle time, thus minimizing the manufacturing time. Optimized cutting strategies require complex problem-solving processes, which may include:

• Adopting a holistic view of the part to be manufactured

• Considering continuously all the criteria of task efficiency

• Using dynamic mental imagery of the cutting process that includes all intermediate phases of the whole part

• Adopting two frames of reference regarding the part to be manufactured and the cutting tools in order to optimize their use Two hypotheses have been formulated with respect to the observed differences in the performance of CNC lathe programmers: (1) very good programmers will spend more time on problem formulation before proceeding to the specification of the program than good programmers, and (2) very good programmers will adopt an optimized cutting strategy while good programmers would settle for a serial strategy. Consequently, very good programmers will adopt a more complex problem-solving process than good programmers. To test these hypotheses, an experiment was designed that would also provide valuable input into the cognitive task analysis utilized in this study. Details on the design of the experiment and the results obtained can be found in Marmaras et al. (1997).

User Requirements for a Cognitive Aid Supporting CNC Lathe Programming

The cognitive task analysis of programming the CNC lathe provided useful insights into the design of an information technology system that supports CNC programmers. The scope of this cognitive aid was twofold. On the one hand, the aid should guide programmers in using efficient cutting strategies, and on the other hand, it should alleviate the cognitive demands that are often associated with optimized problem-solving strategies. Specifically, the cognitive aid could have the following features:

• Support users in deciding upon a cutting strategy at the front-end stages of the programming process

• Generate several suggestions about efficient strategies, e.g., ‘‘try to integrate as many different shapes as possible into one movement of the cutting tool’’ and ‘‘avoid many changes of the cutting tool’’

• Support human memory throughout the whole programming process

• Provide real-time feedback by showing programmers the effects of their intermediate decisions through real-time simulation

• Facilitate the standard actions of the whole programming process

A description of the prototype cognitive aid that was developed in accordance with the user requirements can be found in Marmaras et al. (1997). Another experiment showed that the quality of the CNC lathe programs was improved and the time required to program the lathe was decreased by approximately 28% when operators used the proposed cognitive aid. Not only did the performance of good CNC lathe programmers improve, but so did the performance of the very good programmers.

The Case of a Cognitive Aid for Managerial Planning

This section presents a study of user requirements in the design of an information technology system that supports high-level managerial planning tasks in small to medium-sized enterprises (Laios et al. 1992; Marmaras et al. 1992). High-level managerial planning tasks, usually referred to as strategic or corporate planning, involve a series of cognitive processes preceding or leading to crucial decisions about the future course of a company. Managerial planning entails:

1. Strategies or high-level decisions that have a long-lasting influence on the enterprise

2. Tactics or lower-level strategies regarding policies and actions

3. Action plans that specify detailed courses of action for the future

In large companies, strategic planning is an iterative, lengthy, and highly formalized process involving many managers at several levels in the organization. In contrast, in small companies, the manager- owner is the sole, most important actor and source of information on planning issues; in this case, planning is an individual cognitive process without the formalism required in larger organizations.

Elements that affect planning decisions are past, present, and expected future states of the external environment of the firm, which have a complex relationship with the internal environment. Managers may perceive these elements as threats or opportunities in the marketplace (external environment) and strengths or weaknesses of the enterprise (internal environment). Effective planning decisions should neutralize threats from the external environment and exploit its opportunities, taking into account the strengths and weaknesses of the company. At the same time, effective planning decisions should increase the strengths of the company and decrease its weaknesses.

A literature review and three case studies were undertaken in order to identify the main elements of the external and internal environments that may constrain managerial planning. The main con- straints identified were:

• Complexity / multiplicity of factors: The external and internal environments of an enterprise include a large number of interacting factors. Market demand, intensity of competition, socio- economic situation, labor market, and technology are examples of the external environment factors. Product quality, process technology, distribution channels, and financial position are examples of the internal environment factors.

• Change / unpredictability: The world in which firms operate is constantly changing. Very often these changes are difficult to predict with regard to their timing, impact, and size effects. Man- agers therefore face a great degree of uncertainty.

• Limited knowledge with respect to the final impact of planning decisions and actions. For example, what will be the sales increase resulting from an advertising campaign costing X, and what from a product design improvement costing Y?

• Interrelation between goals and decisions: Planning decisions made in order to achieve a con- crete goal may refute—if only temporarily—some other goals. For example, renewal of pro- duction equipment aiming at increasing product quality may refute the goal of increased profits for some years.

• Risk related to planning decisions: The potential outcomes of planning decisions are crucial to the future of the enterprise. Inaccurate decisions may put the enterprise in jeopardy, often leading to important financial losses.

Providing valuable input to cognitive analysis was a series of structured interviews with managers from 60 small- to medium-sized enterprises in order to collect taxonomic data about typical planning tasks in various enterprises, environmental and internal factors, and personality characteristics. In other words, the aim of these interviews was to identify the different situations in which managerial planning takes place.

The scenario method was used to conduct a cognitive analysis of managerial planning because of the difficulties involved in direct observations of real-life situations. The scenario presented a hypothetical firm, its position in the market, and a set of predictions regarding two external factors: an increase in competition for two of the three types of products and a small general increase in market demand. The description of the firm and its external environment was quite general, but realistic so that managers could create associations with their own work environment and therefore rely on their own experience and competencies. For example, the scenario did not specify the products manufactured by the hypothetical firm, the production methods and information systems used, or the tactics followed by the firm. The knowledge elicited from the three case studies and the structured interviews was used to construct realistic problem scenarios.

Scenario sessions with 21 practicing managers were organized in the following way. After a brief explanation about the scope of the experiment, managers were asked to read a two-page description of the scenario. Managers were then asked to suggest what actions they would take had they owned the hypothetical firm and to specify their sequence of performance. Additional information about the scenario (e.g., financial data, analysis of profits and losses, production and operating costs) were provided in tables in case managers would like to use them. Managers were asked to think aloud and were allowed to take notes and make as many calculations as needed. The tape recordings of the scenario sessions were transcribed and verbal protocols were analyzed using a 10-category cod- ification scheme (see Table 4). The coded protocols indicated the succession of decision-making stages and the semantic content of each stage—for example, the sort of information acquired, the specific goals to be attained, or the specific tactic chosen. Details of the verbal protocol analysis can be found in Marmaras et al. (1992). A brief presentation of the main conclusions drawn from the cognitive analysis follow, with particular emphasis on the managers’ cognitive strategies and heuristics used in managerial planning.

• Limited generation of alternative tactics: Most managers did not generate at the outset an extensive set of alternative tactics to select the most appropriate for the situation. Instead, as

soon as they acquired some information and formed a representation of the situation, they defined a rather limited set of tactics that they would apply immediately (e.g., C1→C2, C1→C7→C3→C2, C1→C3→C2, C1→C3→C1→C2). This finding could be attributed to the fact that experienced managers possess an extensive repertoire of experiences accessed through rec- ognition rather than conscious search. Optimization strategies are time consuming and difficult

to sustain under the constraints of the work environment and the limited knowledge regarding the impact of decisions. In addition, the high risks associated with different decisions would probably push managers to adopt already tested tactics. Limited generation of alternative tactics, however, may lead to ineffective practices. For instance, a past solution may be inappropriate if the current situation has some subtle differences from others experienced in the past. In the case of managerial planning, new innovative solutions and radical departures may be of great importance.

• Acting / thinking cycles: After deciding upon a first set of actions, managers would wait for immediate feedback before proceeding to corrective actions or applying other tactics. This be- havior compensates to some extent for the potential negative consequences of the previous strategy. That is, early decisions may be based on past experiences, but their suitability can be evaluated later on when feedback becomes available; thus, managers may undertake other cor- rective actions. However, these cycles of acting / thinking behavior may lead to delays in acting, increased costs, or money loss.

• Limited use of information and analysis: Most managers avoided speculating on the available predictive quantitative account data and did not make any projections in their evaluation of selected tactics. Instead, they quickly came up with ideas about what to do and stated that they would base their evaluation and future actions on specific outcomes of their initial actions. Decisions were justified by making references to environmental factors (C2→C3, C3→C2), other

superordinate goals (C2→C6, C6→C2, C8→C2, C2→C8), and feedback from previous actions.

This behavior may be attributed to the constraints of the environment, the uncertainty in making predictions, and the interleaving goals that may render a quantitative evaluation of tactics rather difficult or even impossible. The possible negative consequences of this behavior are that data important for planning decisions may be neglected during the planning process.

• Lack of quantitative goals and evaluation: The results of the analysis suggested that, in general, managers do not set themselves quantitative goals that will influence the selection and evaluation of their actions. Instead, their planning decisions seem to be based mainly on assessments of external and internal environment factors and certain generic goals related to past experiences. This observation provides evidence supporting the criticism of several authors (e.g., Lindblom 1980) with respect to the relevance of goal-led models; it also suggests that cognitive aids should not be based on optimization models of managerial planning.

Drawing on the results of the cognitive analysis, an information technology system that supports managerial planning in small- to medium-sized enterprises was specified and designed. The system took the form of an active framework that supports cognitive processes in the assessment of envi- ronmental factors and generation, selection, and evaluation of strategies and tactics. At the same time, the support system would limit some negative consequences of the managers’ cognitive strategies and heuristics. With the use of appropriate screens and functions, the proposed cognitive aid should provide support in the following ways:

• Assisting managers in considering additional data in the planning process and identifying sim- ilarities to and differences from other experiences in the past. This could be done by presenting menus of external and internal factors in the planning environment and inviting managers to determine future states or evolutions of system states.

• Enriching the repertoire of planning decisions by presenting menus of candidate options and tactics relevant to the current situation.

• Supporting the acting / thinking process by providing crucial information concerning the state of external and internal environment factors at the successive action / thinking cycles and the different planning decisions made during these cycles.

The planning process proposed by the cognitive aid could be achieved through a series of steps. First, an assessment should be made of the past, present, and future states of environmental factors; this is in contrast to the setting of quantitative goals, as proposed by formal models of strategic planning. The next step involves the specification of alternative tactics and evaluation using qualitative and, optionally, quantitative criteria. Additional steps with regard to the setting of generic strategies and goals have been included in between. However, some degrees of freedom should be allowed in the choice and sequence of these steps. Although this procedure imposes a certain formalism in managerial planning (this is inevitable when using an information technology system), sufficient compatibility has been achieved with the cognitive processes entailed in managerial planning. A prototype of the proposed cognitive aid was evaluated by a number of managers of small- to medium- sized enterprises, and its functionality and usability were perceived to be of high quality.

6. CONCLUDING REMARKS

Advances in information technology and automation have changed the nature of many jobs by placing particular emphasis on cognitive tasks and, on the other hand, by increasing their demands, such as coping with more complex systems, higher volume of work, and operating close to safety limits. As a result, successful ergonomic interventions should consider the interactions in the triad of ‘‘users, tools or artifacts, work environment.’’ These interactions are affected by the user strategies, the constraints of the work environment, and the affordances and limitations of the work tools. In this respect, techniques of cognitive task analysis are valuable because they explore how users’ cognitive strategies are shaped by their experience and interaction with the work environment and the tools. This explains why a large part of this chapter has been devoted to illustrating how cognitive task analysis can be used in the context of applied work situations.

Cognitive analysis provides a framework for considering many types of ergonomic interventions in modern jobs, including the design of man–machine interfaces, cognitive tools, and training pro- grams. However, cognitive analysis requires a good grasp of human cognition models in order to understand the mechanisms of cognition and develop techniques for eliciting them in work scenarios or real-life situations. The cognitive probes and the critical decision method, in particular, demonstrate how models of human cognition can provide direct input into cognitive task analysis. Further advances in cognitive psychology, cognitive science, organizational psychology, and ethnography should pro- vide more insights into how experienced personnel adapt their strategies when work demands change and how they detect and correct errors before critical consequences are ensued.

The chapter has also emphasized the application of cognitive analysis in the design of information technology cognitive aids for complex tasks. The last two case studies have shown that cognitive aids should not be seen as independent entities capable of their own reasoning but as agents inter- acting with human users and work environments. Because unforeseen situations beyond the capabil- ities of cognitive aids are bound to arise, human operators should be able to take over. This requires that humans be kept in the loop and develop an overall mental picture of the situation before taking over. In other words, user-aid interactions should be at the heart of system design. Recent advantages of technology may increase the conceptual power of computers so that both users and cognitive aids can learn from each other’s strategies.