Evaluate Alternatives

Evaluating alternative solutions involves analysis, discussion, and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, or simulations may be needed to substantiate scoring and conclusions.

Often, the relative importance of criteria is imprecise and the total effect on a solution is not apparent until after the analysis is performed. In cases where the resulting scores differ by relatively small amounts, the best selection among alternative solutions may not be clearcut. Challenges to criteria and assumptions should be encouraged.

• Evaluate the proposed alternative solutions using the established evaluation criteria and selected methods.
• Evaluate the assumptions related to the evaluation criteria and the evidence that supports the assumptions.
• Evaluate whether uncertainty in the values for alternative solutions affects the evaluation and address as appropriate. For instance, if the score can vary between two values, is the difference significant enough to make a difference in the final solution set? Does the variation in score represent a high risk? To address these concerns, simulations may be run, further studies may be performed, or evaluation criteria may be modified, among other things.
• Perform simulations, modeling, prototypes, and pilots as necessary to exercise the evaluation criteria, methods, and alternative solutions. Untested criteria, their relative importance, and supporting data or functions may cause the validity of solutions to be questioned. Criteria and their relative priorities and scales can be tested with trial runs against a set of alternatives. These trial runs of a select set of criteria allow for the evaluation of the cumulative impact of the criteria on a solution. If the trials reveal problems, different criteria or alternatives might be considered to avoid biases.
• Consider new alternative solutions, criteria, or methods if the proposed alternatives do not test well; repeat the evaluations until alternatives do test well.
• Document the results of the evaluation. [PA156.IG101.SP105.SubP106] Document the rationale for the addition of new alternatives or methods and changes to criteria, as well as the results of interim evaluations.