RUNNING HEAD: COGNITIVE TASK ANALYSIS
Thinking About Work/Thinking
at Work:
Cognitive Task Analysis
Rosemarie Reynolds and
Michael Brannick
University of South Florida
In R. P. Tett & J. C.
Hogan (Co-Chairs), Recent developments
in cognitive and personality approaches to job analysis. Symposium presented at the 17th
annual conference of the Society for Industrial and Organizational Psychology,
Toronto (April 2002).
During the writing of this
paper, Rosemarie Reynolds was supported as a Graduate Research Fellow by the
Naval Air Warfare Center Training Systems Division, Orlando, Florida. Correspondence concerning this article
should be addressed to Rosemarie Reynolds, Business Department, Embry-Riddle
Aeronautical University, Daytona Beach, FL 32114. Electronic mail may be sent via Internet to
mbrannic@luna.cas.usf.edu.
Abstract
The purpose of this paper is to discuss the
potential usefulness of CTA as an extension of traditional job analysis. In
order to do so, we discuss the
application of CTA to training, job design, and selection.
Thinking About Work/Thinking
at Work: Cognitive Task Analysis
The purpose of this paper is
not to provide an exhaustive
review or evaluation of cognitive task
analysis (CTA) techniques, but rather to discuss the potential usefulness of
CTA as an extension of traditional job analysis that focuses on the knowledge
and thought processes supporting observable task performance (Schraagen,
Chipman, & Shalin, 2000). As an extension of traditional job analysis, CTA
has its roots in the classical time and motion study research of the scientific
management school. In the 1950's, the
concept of mental workload, and the notion that information processing, rather
than physical effort, determined the time to complete a task, was added to the
time and motion study approach. Other
contributions from educational psychology, knowledge engineering, software
engineering, and human factors lead to the eventual emergence of CTA in the
late 1970s (Annett, 2000).
As with job analysis, the choice
of cognitive task analysis methods rests primarily on practicality and purpose
(Levine, 1983). For many jobs, the
highly labor intensive process of analyzing cognitive requirements would not
provide any value beyond that derived from a standard job analysis. However, a CTA may be indicated if a task is
complex, ill-structured, dynamic, uncertain, and involves decision making or
teamwork, as the CTA may provide critical information not provided by the job
analysis (Gordon & Gill, 1997; Reynolds & Brannick, 2000, Reynolds
& Neville, 2002)).
Thus, the CTA process begins
with an initial phase that provides an overview of the organizational needs,
the resources available for performing the CTA, and the types of knowledge and
skills involved in the job (Ryder, Gott, & Neville, 2000). The second stage, if indicated, involves the
selection of a methodology and data collection. In the third stage, the focus is on data analysis and the
presentation/representation of the information in a useful manner. Although these tasks have been presented as
discrete stages, the process is in fact a reiterative one; the types of
information needed in stage three will drive the types of data collected in
stage two.
The data collection is one of the most challenging aspects of the CTA process for several reasons: 1) much knowledge is tactic or automatic, 2) knowledge may be difficult to verbalize, and 3) subject matter experts may tend to simplify when conveying their knowledge to a non expert, or simply provide whatever knowledge is accessible (Cooke, 1994). An exhaustive review of data collection not possible here, but Table 1 shows some of the commonly used methods of knowledge elicitation, based on reviews by Hoffman, Shadbolt, Burton, and Klein (1995), who suggested a typology based on 1) the analysis of familiar tasks, 2) interviews, and 3) contrived techniques, and Cooke (1994), who suggested observations and interviews, process tracing, and conceptual techniques as typological dimensions. Broadly speaking, however, the type of CTA technique chosen depends on a number of factors, including available resources, expertise, and the type of knowledge being acquired. As McCloskey, Geiwitz, and Kornell (1991) pointed out, some techniques are better suited to elicit declarative knowledge, while others are more effective for procedural knowledge.
CTA
has been widely used in human factors, instructional design, and human-computer
interaction. Its uses in industrial
psychology are less widely known. In
the following sections, we will discuss the application of CTA to training, job
design, and selection.
|
|
Analysis of Familiar Tasks |
Contrived Techniques
|
|
Observations
And Interviews |
Task Analysis
Documentation analysis
Observation
Active Participation
Focused Observation
Structured Observation
Unstructured Interviews
Structured Interviews
Group interviews
Automated Interviewing Tools |
|
|
Process
Tracing |
Protocol Analysis
Verbal Reports
Verbal On -Line
Verbal Off-Line/Simulated Recall
Non-Verbal Reports |
Decision Analysis
Group decision making |
|
Conceptual
Techniques |
|
Ranking And Rating
Repertory Grid
Sorting
Constrained processing and limited
information problems
Event Co-Occurrence/Transition
Probabilities
Correlations/Covariance |
Table 1.
Knowledge Elicitation Techniques
As a result of their
analysis, recommendations were made regarding training a generic strategy for
troubleshooting that prevents information overload, in addition to
system-specific functional models and domain knowledge. The generic strategy
consisted of training students to systematically apply the four subcomponents
of troubleshooting: problem description, generation of possible causes,
testing, and diagnosis.
Problem description involves
identifying and comparing normal and abnormal system behavior. Generating
possible causes required the ability to decide, for example, if the problem
lies in the power supply, CPU, memory, or peripherals, as well as the ability
to determine what level of possible causes are worth investigating, e.g. a
computer may have thousands of relays; to examine each relay would be highly
ineffective.
Testing requires the ability
to make hypotheses, to choose the right test, to execute the test, to set up conditions
for measuring test outcomes, and to compare the results of testing to the
hypotheses. Finally, the troubleshooter
must evaluate what needs to be done in order to fix the problem.
In order to evaluate the new
training, one group of subjects took a one-week course incorporating the new
training in addition to the regular training course, while another group took
only the regular training course. Subsequently, both groups were scored on a
theoretical knowledge test, as well as on their ability to solve four
unfamiliar systems problems. There was
no difference between the two groups on the test of theoretical knowledge, but
the experimental group, which had received training how to systematically THINK
about problems, performed significantly better, solving twice as many problems
as the control group.
1. The total number of
actions in a period should have an upper and lower limit so that there is
sufficient time to carry out these actions
2. The task must call for
several levels of information processing. Skill-based actions are barely
cognitively demanding; the ideal ratio between rule-and knowledge-based actions
in cognitive tasks has an upper and a lower limit
3. There should be no
long-term period in which only one sort of skill is performed continuously
(e.g. performance may decrease after 10 minutes of continuous vigilance)
4. There should be no
momentary overloading; several knowledge-based actions should not have to be
performed in rapid succession within a short period of time, and rule- or
knowledge-based actions should not have to be performed almost simultaneously
(p. 545)
A two-part process was used
for the analysis in which a fairly traditional hierarchical task model was
followed by a time-line analysis. For the hierarchical task analysis, the goal
was to obtain an ordered description of the tasks allocated to one person In the case of railway controllers, the
three major tasks were (a) to deal with irregularities, (b) to carry out a work
plan, and (c) to inform passengers. For
the time line analysis, a sampling of observations was done at peak periods and
off peak periods, at different stations with different trainloads and different
task allocations. In each observation,
a single traffic controller was observed and videotaped while performing his
tasks and thinking out loud. Actions
were charted and evaluated in terms of the guidelines listed above.
The authors found that at
two sites, the controllers were under loaded with regards to Guidelines 1 and
2, while at one site they were overloaded.
With regards to continuous action (Guideline 3), at two sites,
continuous action was required almost constantly, while at one site it was
required occasionally. One of the
periods of overloading (Guideline 4) also coincided with a number of
irregularities, which increased the number of knowledge based actions required,
adding to the overload situation.
The final step in their
analysis involved redesigning the task so as to meet the guidelines. For one of
the sites in which the task load was low, the task could be extended by
extending the domain to control part of another railway yard. In the second
site, workload was also low, and this task could be extended by adjusting task
allocation to include carrying out a work plan and informing passengers. The third site was overloaded. Possible solutions included reducing the
control area during irregularities and designing a computer support system.
Hunt and Joslyn (2000)
pointed out that selection presupposes that the skills required by a job are
known. They began developing their test
for public safety dispatchers by having four observers observe operations at a
dispatch center, consisting of three eight hour shifts, each staffed by two or
three call receivers, two or three police dispatchers, a fire-medical
dispatcher, and a shift supervisor.
When a call is received, a
call receiver obtains details and enters them, as well as an incident
classification and priority, into the computer system, at which point the
information is sent to a dispatcher.
The dispatcher reviews the description of the incident, a list of
available and assigned police units, and a list of all incidents that have been
assigned and are still waiting assignment and their priority. The dispatcher may change the priority
classification. In addition, the
dispatcher receives reports directly from officers in the field regarding new
incidents, and the handling of existing incidents, which the dispatcher logs
into the computer. The dispatcher is
also responsible for checking on officers periodically after they have been
assigned to an incident.
As a result of their
observations, the researchers decided that the major tasks were classifying
(which involves gathering information so classification can be made, either by
questioning the caller, or searching databases) and assigning resources (the
ability to prioritize actions). They
also decided that their test should identify people who do well in situations
requiring multitasking and decision making under time pressure.
The next stage was an
experimental manipulation. A task was
constructed (The Abstract Decision Making
(ADM) Task) that required allocation
of resources based on the classification of incidents in which the attributes
used to make the classifications would only be partially revealed to the test
taker. ADM is a computer game, in which
points are earned by sorting objects into bins based on size, shape, and color. The objects are not shown to the game
player, however; instead the game player must ask questions about the object
until they have enough information to sort it into a bin. ADM was evaluated by
comparing performance on ADM with performance on dispatching and air traffic
control. Performance on ADM
reliably predicted performance in both simulations.
The next step was to develop
a test in which participants participated in a call-receiver simulation. Scripted actors called in, and the
participants gathered information that they entered into a computer screen
similar to the one used in actual dispatching.
The length of time it took participants to collect the information and
classify the call was recorded, as well as the classification assigned. The correlation between performance in the
call simulation and the ADM task was .53 (p<.05). Thus, this simple computer game (ADM) appears to be able to
identify the skills required for successful performance, without requiring the
extensive domain specific knowledge that using the simulations themselves for
selection would require; e.g. to select candidates based on a dispatch
simulation, candidates would have to know how to do dispatching.
CTA offers an extension to
job analysis that can be useful in applications other than the
the traditional ones of human factors design,
instructional design, and human-computer interaction. We have attempted to demonstrate its potential usefulness in training,
job design, and selection.
However, we would like to
offer several cautions regarding unbridled enthusiasm for CTA. The added
expense and time required by a CTA analysis is not always helpful; the added
expense has to be considered against the increased understanding of mental
aspects of job performance (the mental strategies, factual knowledge, and so
forth). The analysis will be fruitful
to the degree that understanding the mental process is necessary to solving the
problem that initially called for the job analysis.
In addition, there are a
large number of techniques of CTA. Much
better guidance about
the circumstances in which each is most appropriate,
probably linked to the type of task (e.g., negotiation vs. surgery), the
purpose of the analysis (e.g. training vs. selection) and the available
resources (e.g., task SME, personnel specialist) is needed.
References
Annett, J. (2000). Theoretical and pragmatic influences on task analysis methods. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin, (Eds.), Cognitive Task Analysis, (pp. 25-40). Mahwah, NJ: Lawrence Erlbaum Associates.
Axton,
T. R., Doverspike, D., Park, S.R. and Barrett, G.V. (1997) A model of the
information processing and cognitive ability requirements for mechanical
troubleshooting. International
Journals of Cognitive Ergonomics, 1(3), 245-266.
Cooke, N.J., (1994).
Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41,
801-849.
Dhaliwal, J.S. &
Benbasat, I. (1990). A framework for the comparative evaluation of knowledge
acquisition tools and techniques. Knowledge
Acquisition, 2, 146-166.
Dougherty, W.T., Ebert, R.J., & Callender, J.C., (1986)
Policy capturing tin the employment interview. Journal of Applied Psychology, 71(1), 9-15.
DuBois, D. & Shalin, V.L. (2000). Describing job expertise using cognitively-oriented task analysis. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin, (Eds.), Cognitive Task Analysis, (pp. 41-56). Mahwah, NJ: Lawrence Erlbaum Associates.
Enkawa, T., & Salvendy,
G. (1989). Underlying dimensions of human problem solving and learning:
Implications for personnel selection, training, task design, and expert system.
International Journal of Man-Machine
Studies, 30(3), 235-254.
Glaser,
R. (1986). A cognitive science perspective on selection and classification and
on technical training. Advances in
Reading/Language Research, 253-268.
Glaser,
R., Schwarz, P.A., & Flanagan, J.C. (1958). The contribution of interview
and situational performance procedures to the selection of supervisory
personnel. Journal of Applied
Psychology, 42, 69-73.
Gordon, S.E., and Gill, R.T. (1997). Cognitive Task Analysis. In C. Zsambok and G. Klein (Eds.). Naturalistic Decision Making. (pp. 131-140). Mahwah, NJ: Lawrence Erlbaum Associates.
Hoffman, R.R., Shadbolt,
N.R., Burton, A.M., & Klein, G. (1995) Eliciting knowledge from experts: A
methodological analysis. Organizational
Behavior and Human Decision Processes, 62(2), 129-158.
Hunt, E., & Joslyn, S. (2000). A functional task analysis of time-pressured decision making. In J.M. Schraagen, S.F. Chipman, & V.L. Shalin, (Eds.), Cognitive Task Analysis, (pp. 119-134). Mahwah, NJ: Lawrence Erlbaum Associates.
Koubek,
R.J., Salvendy, G., and Noland. S. (1994) The use of protocol analysis for
determining ability requirements for personnel selection on a computer-based
task. Ergonomics, 37,
1787-1800.
Levine, E.L. (1983) Everything you always wanted
to know about job analysis, and more: A primer. Tampa, FL: Edward L.
Levine..
McCloskey, B.P., Geiwitz,
J., & Kornell , J. (1991) Empirical comparisons of knowledge acquisition
techniques. Proceedings of the Human Factors Society, USA, 35, 268-272.
Neerincx, M.A., &
Griffioen, E. (1996). Cognitive task analysis: Harmonizing tasks to human
capabilities. Ergonomics, 39(4),
543-561.
Reynolds, R.T., &
Brannick, M. (2000) Is job analysis doing the job: Extending job analysis with
cognitive task analysis, The
Industrial Psychologist, 39(1), 63 67.
Reynolds, R.T. &
Neville, K. (2002). Job Analysis
versus Cognitive Task Analysis of Air Traffic Controllers, Paper presented
at the Military Psychology Conference, VA.
Ryder, J.M., Gott, S., &
Neville, K.J., (2000). Using Cognitive Task Analysis for Instructional Design.
Orlando, FL: Naval Air Warfare Center Training Systems Division. (Technical
Report 000322.9914)
Schaafstal, A., Schraagen,
J.M., & Van Berlo, M. (2000). Cognitive task analysis and innovation of
training: The case of structured troubleshooting. Human Factors, 42(1), 75-86.
Schraagen, J.M., Chipman,
S.F., & Shalin, V.L. (2000) Introduction to Cognitive Task Analysis. In
J.M. Schraagen, S.F. Chipman, & V.L. Shalin, (Eds.), Cognitive Task
Analysis, (pp. 3-23). Mahwah, NJ: Lawrence Erlbaum Associates.
Appendix
1: Bibliography of Knowledge Elicitation Resources
Agarwal, R. & Tanniru,
M. R. (1991). Knowledge extraction using content analysis. Knowledge Acquisition, 3, 421- 441.
Bainbridge. L. (1990).
Verbal protocol analysis. In J. R. Wilson & E. N. Corlett, (Eds.) Evaluation of Human Work: A Practical
Ergonomics Methodology, (pp. 161-179). London: Taylor & Francis.
Boose, J. H. (1990). Uses of
repertory grid-centered knowledge acquisition tools for knowledge-based
systems. In J. Boose & B. Gaines (Eds.), The Foundations Of Knowledge
Acquisition: Knowledge- Based Systems, Vol. 4, (pp.61- 84). San Diego, CA:
Academic Press.
Boose, J. H. (1989). A
survey of knowledge acquisition techniques and tools. Knowledge Acquisition, 1,
3-37.
Brenner, M., Brown, J. &
Canter, D. (1985). The Research Interview: Uses and Approaches. London:
Academic Press.
Burton, A. M., Shadbolt, N. R., Hedgecock, A. P, &
Rugg, G. (1987). A formal evaluation of knowledge elicitation techniques for
expert systems: Domain 1. Proceedings of the First European Workshop on
Knowledge Acquisition far Knowledge-Based Systems, Reading University, D
3.1- D 3.20.
Burton, A. M., Shadbolt, N. R., Rugg, G. , &
Hedgecock, A. P. (1990). The efficacy of
knowledge elicitation techniques: A comparison across domains and levels
of expertise. Journal of Knowledge
Acquisition, 2, 167-178.
Cooke, N. J. (1994) Varieties of knowledge
elicitation techniques. International
Journal of Human-Computer Studies, 41, 801- 849.
Corter, J.E. & Tversky,
A. (1986) Extended similarity trees. Psychometrika, 51, 429-451.
Dhaliwal, J.S. &
Benbasat, I (1990). A framework for the comparative evaluation of knowledge
acquisition tools and techniques. Knowledge Acquisition, 2, 146-166.
Flanagan, J.C. (1954) The
critical incident technique. Psychological Bulletin, 51, 327-358.
Geiwitz, J., Klatzky, R.L.,
& McCloskey, B.P. (1988). Knowledge acquisition techniques for expert
systems: Conceptual and empirical comparisons. Santa Barbara, CA: Anacapa
Sciences.
Geiwitz, J., Kornell, J.,
& McCloskey, B.P. (1990). An expert system for the selection of
knowledge acquisition techniques. Santa Barbara, CA: Anacapa Sciences.
(Technical Report 785-2).
Gordon, S. E. & Gill, R. T. (1989). Question
probes: A structured method for eliciting declarative knowledge. AI
Applications In Resource Management, 3, 13-20.
Grabowski, M., Massey, A. P. &. Wallace. W. A. (1992). Focus groups
as a group knowledge acquisition technique. Knowledge Acquisition, 4,
407-425.
Grover, M.D.(1983). A
pragmatic knowledge acquisition methodology. Proceedings from the International
Joint Conference on Al, West Germany, 436-438.
Henrion, M. & Cooley, D. R. (1987). An experimental
comparison of knowledge engineering for expert systems and for decision
analysis. Proceedings of the Sixth National Conference on Artificial
Intelligence, Seattle, 471-476.
Hoffman, R.R., Shadbolt,
N.R., Burton, A.M., & Klein, G. (1995) Eliciting knowledge from experts: A
methodological analysis. Organizational Behavior and Human Decision
Processes, 62(2), 129-140.
Kitto, C. M. & Boose,
J. (1989). Selecting knowledge
acquisition tools and strategies based on application characteristics. International
Journal Of Man-Machine Studies, 31, 149-160.
Klein, G. A., Calderwood, R. &- Macgrecor, D. (1989). Critical decision
method for eliciting knowledge. IEEE Transactions On Systems, Man, &
Cybernetics, 19, 462-472.
Krippendorff, K. (1980). Content
analysis: An introduction to its methodology. Beverly Hills, Ca.:Sage.
Krumhansl, C. L. (1978). Concerning the
applicability of geometric models to similarity data: The interrelationship
between similarity and spatial density. Psychological Review, 85,115-129.
Kruskal, J. B. (1977). Multidimensional
scaling and other methods for discovering structure. In K. Enslein, A. Ralston, & H. S. Wilf, Eds. Statistical
Methods For Digital Computers. New York: Wiley.
Littmann, D.C. (1986). Modeling human
expertise in knowledge engineering: Some preliminary observations. Proceedings
of the First Knowledge Acquisition for Knowledge-Bases Systems Workshop,
Banff, 28.
McCloskey, B.P. Geiwitz,
J. and Kornell , J. (1991) Empirical
comparisons of Knowledge acquisition techniques. Proceedings of the Human
Factors Society, 268 272.
McGraw, K.L. &
Harbison-Briggs, K. (1989) Knowledge Acquisition: Principles and guidelines.
Englewood Cliffs, NJ: Prentice Hall.
Michalski, R.S. &
Chilausky, R.L. (1980). Learning by being told and learning from examples: An experimental
comparison of the two methods of knowledge acquisition in the context of developing an expert system
for soybean disease diagnosis. Policy Analysis and Information systems, 4.
Miles, D.E., Hoffman, K.A.,
Foster, L.L., King, T., Gordon, T., Riddle, D., Coovert, M., Elliott,
L.R., & Schiflett, S. (2000).
A qualitative methodology for integrating cognitive task analyses
data. Poster presented at the
15th Annual Conference of the Society for Industrial and
Organizational Psychology, New Orleans, LA, 2000.
Morik, K. (1991). Underlying
assumptions of knowledge acquisition and machine learning. Knowledge
Acquisition, 3,137-156.
Mostyn, B. (1985). The
content analysis of qualitative research data: A dynamic approach. In M.
Brenner, J. Brown & D. Canter, Eds. The Research Interview: Uses And
Approaches. (pp. 115-145). London: Academic Press.
Olson, J. R. & Rueter,
H. (1987). Extracting expertise from experts: Methods for knowledge acquisition.
Expert Systems, 4, 152-168.
Reitman, J. S. & Rueter,
H. (1980). Organization revealed by recall orders and confirmed by pauses. Cognitive
Psychology, 554-581.
Reynolds, R.T. &
Neville, K. (2002). Job Analysis
versus Cognitive Task Analysis of Air Traffic Controllers, Paper presented
at the Military Psychology Conference, VA.
Russo, J. T. &, Rosen, L D. (1975). An eye fixation
analysis of multialternative choice. Memory And Cognition, 3, 267-276.
Scherer, K. R. & Ekman, P.
(1982). Handbook of methods in nonverbal behavior research. Cambridge:
Cambridge University Press.
Schraagen, J.M., Chipman,
S.F., & Shalin, V.L. (2000) Introduction to Cognitive Task Analysis. In
J.M. Schraagen, S.F. Chipman, & V.L. Shalin, (Eds.), Cognitive Task
Analysis, (pp. 3-23). Mahwah, NJ: Lawrence Erlbaum Associates.
Schvaneveldt, R. W., Durso, F. T.,
Goldsmith, T. E., Breen,
T. J., Cooke, N., Tucker, R.G. & DeMaio, J.C. (1985). Measuring the structure of expertise. International
Journal Of Man-Machine Studies, 23, 699-728.
Scott, A. A., Clayton, J. E.
& Gibson, E. L (1991). A practical guide to knowledge acquisition. Reading, Ma: Addison-Wesley.
Shaw, M. L. & Woodward, J.B.(1988). Validation in a
knowledge support system: Replication and consistency with multiple experts. International
Journal of Man-Machine Studies, 29, 329-350.
Smith, J. B., Smith, D. K.
& Kupstas, E. (1993). Automated protocol analysis. Human Computer
Interaction, 89, 101-145.
Sowa, J. F. (1992).
Conceptual analysis as a basis for knowledge acquisition. In R. R. Hoffman, Ed.
The Psychology Of Expertise: Cognitive Research And Empirical Al, (pp.
80-96). New York: Springer-Verlag.
Thordsen, M.L. (1991). A
comparison of two tools for cognitive task analysis: Concept mapping and the
critical decision method. Proceedings Of The Human Factors Society,.
283-285.
Welbank, M. (1990). An
overview of knowledge acquisition methods. Interacting With Computers, 2,
83-91.
Woodward, B. (1990).
Knowledge acquisition at the front end: Defining the domain. Knowledge
Acquisition, 2, 73-94.
Wright, G. & Bolger, F.,
Eds. (1992). Expertise And Decision Support. New York: Plenum Press.