Motivation

Rapid changes in jobs and employment conditions are producing widening skill gaps that impact all societal sectors, and have the largest negative impact on the most vulnerable workers (1).  Job gains are expected in areas such as computers, mathematics, energy, media entertainment, and engineering–all requiring demanding training and retraining.  A majority of the nation’s chief human resources officers believe that business, government, and education are not prepared to handle the training and education needs to offset the dramatic, upcoming transformation in our workforce (2).  Yet, current expert-based training systems used in business, government, and education have not changed significantly in the past century; they largely depend on outmoded, inadequate training methods (3).

An essential component of the next generation of training systems is the need to rapidly and completely capture the skills of top experts and transfer them to efficient, accessible, just-in-time training systems.  A huge barrier to this process is that experts are largely unaware of how they think, analyze, and make complex decisions as they work.  This leads to significant gaps in training that must be filled in by trainees through “trial and error” on the job.  Considerable evidence indicates that promising support for the rapid design of effective expert-based job training and retraining is available from Cognitive Task Analysis (CTA).  Meta analyses of carefully designed CTA training studies on a variety of jobs and professions indicate that CTA captures experts’ critical analysis and decision strategies, allows for the elimination of ineffective strategies from training, and provides more complete information for trainees.  As a result, training time decreases an average of 25%, learning increases about 35%, and on-the-job mistakes made by trainees during the first months after training decrease significantly (3, 4).  However, the widespread application of CTA for job training has been challenging, primarily due to the small number of trained CTA analysts.  CTA entails a complex expert interview strategy requiring one year for an analyst to master the many tasks, knowledge, and skill categories as well as decision criteria used by difficult to access experts.

Why AI?

A major barrier to the scaling of CTA is the availability of CTA analysts and the time required to conduct CTA interviews. We expect that shifting many of the CTA operations from human to AI control will both decrease analyst training time and the time required for conducting CTA interviews and analysis.  This would allow a reduction in costs and increase the number of organizations to adopt CTA.

AI is expected to facilitate significant efficiency gains in CTA analyst training and CTA interviews. The evidence-based CTA interview protocols involve six steps and it is likely that AI can create more efficient actions at every step. The first step of CTA is to interview individual experts and ask them to describe the sequence of tasks required to perform a job or a set of job tasks, with specific examples. Each expert typically uses different terms and scopes for job tasks and so variations must be resolved and a standard list developed with the approval of all experts.  AI with voice recognition, natural language understanding, machine learning and topic clustering can identify differences and create dialogues with experts that permit them to be resolved and a master list created.

In the second to fourth CTA steps, a vital list of the conditions, actions, and decisions that must be implemented to achieve each of the tasks described in step one is collected, differences are resolved by experts, and a “gold standard” developed.  In addition, a list of the cues to start a task and any equipment or supplies needed to complete the task must be obtained.  This is the most labor intensive part of the CTA interview because different experts are aware of different aspects of the decisions and actions they perform.  Again, AI with voice recognition, natural language understanding, dialogue management, machine learning and topic clustering can mirror the strategies used by top analysts to identify the similarities and differences between experts; plan and create dialogues with experts to reconcile differences; identify, inquire about, and resolve substantive differences; recognize the most effective and efficient steps that achieve the performance goal of each task; and summarize the most effective steps in a “gold standard” that is reviewed and approved by the experts and then made available in operational terms that can be used in training. Work on cognitive AI architectures will also be of relevance here.

In the fifth and sixth steps, analysts collect information about the unique conscious knowledge the expert uses to achieve goals.  This involves the use of an evidence-based classification system for necessary facts (empirically-based statements), concepts (criterial attributes used to classify examples), processes (how things work in stages), and principles (cause and effect relationships that influence outcomes).  Here again an  AI system can be designed to assist in the identification, classification, and capturing of this knowledge to include in a gold standard for training.  Additionally, since the challenges experienced in different professions change over time, AI can also help with the constant updating of expert systems by periodically checking the validity of a gold standard.  Stored AI versions of gold standards can then be tested and revised, and any changes communicated to clients.

Our hope is that we might be able to reduce the time required of CTA analysts by training them to shift much of their work to the AI system and interact with the AI system more than with experts.  In addition, we hope to reduce the time required of both analysts and experts by having the AI system create dialogues with experts who access the AI at their own convenience online and only involve an analyst when the AI system cannot complete a task.  Finally, an adequate AI system could be further developed to automate as much as possible the use of the CTA information in a computer-based training system.  We reasonably expect a 30-50% reduction in CTA time and expense with an adequate AI CTA system, and to eventually expect that AI may take over most of the CTA process.

The Opportunity

While CTA has many benefits, widespread adoption has faced challenges. There is strong interest in CTA by large for-profit companies, but the barrier to its use is the availability of trained and experienced analysts.  Most of the initial investment in CTA research and development was made by the U.S. Army.  CTA is also used by large energy, manufacturing and pharma organizations, who do not publicize the results of their work due to concerns about competition.  Use of CTA within healthcare is gaining traction, particularly in surgery and diagnosis. These organizations are more likely to publicize results.

What inhibits the scaling of CTA is the availability of analysts, the lack of training for analysts and the logistics, time, and expense involved in conducting CTA with top experts. Another major barrier is that many of the training staff of government and business organizations have no formal training for their work and many tend to be overconfident about their skills.  This is particularly the case in government organizations.  There are currently no meaningful certifications for training designers or instructors in business or government, so decisions about CTA are being made in both large and small organizations by people who are not prepared to examine the evidence about its potential benefits.

An AI supported system will provide a less expensive and locally controllable way for organizations to sample the benefits of CTA within organizations of all sizes.  We propose to decrease the training time of CTA analysts and the time demands on top experts whose skills are being captured by developing an AI system to support the CTA capture and training design process.  Specifically, AI may assist in the knowledge elicitation and organization process using voice recognition to transcribe expert interviews, natural language processing to recognize topics, and then automated topic clustering to organize and sequence knowledge for training, assessment, and job performance.  AI cognitive architectures could encode such organized knowledge as decision or action “operators” that would help to essentially simulate experts’ individual decision steps and tasks to check for inconsistencies and knowledge gaps.  Using AI promises to significantly boost the scalability of CTA by decreasing both expert and analyst time demands while significantly increasing the impact of job training and retraining.  The USC Educational Psychology and Engineering faculty who would commit to this project are internationally recognized for their research and development work in both CTA and AI and are ideally positioned to accomplish the goals of this proposed project.

References

1. Pew Research Center. (2015). The American Middle Class is Losing Ground. http://www.pewsocialtrends.org/2015/12/09/the-american-middle-class-is-losing-ground/

2. World Economic Forum (2016). The Future of Jobs: Employment, Skills and Workforce Strategy for the Fourth Industrial Revolution. Geneva, Switzerland www.weforum.org

3. Clark, R. E., Yates, K., Early, S. & Moulton, K. (2010). An Analysis of the Failure of Electronic Media and Discovery-based learning:  Evidence for the performance benefits of Guided Training Methods.  In Silber, K. H. & Foshay, R. (Eds.). Handbook of Training and Improving Workplace Performance, Volume I: Instructional Design and Training Delivery. New York: John Wiley and Sons. 263-297. Access at: http://www.cogtech.usc.edu/recent_publications.php

4. Clark, R. E. (2014). Cognitive Task Analysis for Expert-Based Instruction in Healthcare. In Spector, J. M.  Merrill, M. D. Elen, J. and Bishop, M. J. (Eds.).  Handbook of Research on Educational Communications and Technology, 4th Edition. 541-551. ISBN 978-1-4614-3184-8 Access at:  http://www.cogtech.usc.edu/recent_publications.php

Share this
Become a CAIS partner through community projects, funding, volunteering, or research collaboration.