Validation of an Expert System: A Case Study

The steps taken to verify and validate a prototype expert system ( TRANZ) for traffic control through highway work zones are described. The prototype is viewed as an applied statement of the system requirements and a focus for the development of a complete knowledge base. The tasks used in the validation included (a) revisiting the experts who assisted in developing the program, (b) selectively distributing validation copies of TRANZ, (c) identifying problems and decisions that interface with and affect the work zone traffic control problem, and (d) conducting a validation workshop. The workshop was found to be the msot effective way to review the system because the results represent a group consensus, whereas the prior tasks encompassed only individual inputs obtained in isolated instances. The informal request for reviews from users was completely ineffective. The workshop results were interpreted as general comments on the overall concept of TRANZ and specific programming modifications that resulted from erroneous recommendations by TRANZ. The general comments indicated that the attendees would not completely rely on TRANZ for finding traffic control solutions for work zones. However, they did indicate a willingness to work with TRANZ in the field and during instructional programs and thus to bring it along slowly while carefully validating it. The conclusions show the relatively long time needed to continue to validate and update an expert system until it correctly addresses a good majority of cases. The specific changes recommended by the panel demonstrated how the basic prototype can be expanded through experience. In the case of TRANZ, the knowledge base was expanded by adding new rules. The strategy used of asking experts to use the system can quickly expand the set of problems that a system addresses via direct expert input. Overall, the validation process must address both the accuracy and the completeness of the system.