This appendix provides a comprehensive overview of our predictive algorithms’ performance on every field in this study. For the relevance field (i.e., whether qualified immunity was raised on appeal), we calculated performance statistics based on the entire 187-opinion testing sample. For all other fields, we calculated performance statistics based on the 162 relevant opinions from the testing sample. We did not use these holdout test data at any point during the algorithm-development process. This ensured that the algorithms’ performance on these opinions was representative of their performance overall.

To measure performance, we used the following statistics:

• Accuracy: Accuracy measures the number of correct predictions out of the total number of predictions for a field. For example, if an algorithm predicts a field 100 times and 91 predictions are correct, then the accuracy is 91%. Accuracy is our primary metric for evaluating text and categorical fields. Although we report accuracy for all predicted fields, other metrics—precision, recall, and F1 score—are better indicators of performance for binary (yes/no) fields.  
• Precision: In a nutshell, precision measures how good an algorithm is at avoiding false positives. For example, if an algorithm predicts 20 opinions involve state law enforcement officials, but only 18 actually do (meaning two were false positives), then the precision is 0.9 (18/20). The maximum value is 1. Precision is applicable only for binary fields.
• Recall: Put simply, recall measures how good an algorithm is at picking out the field in question. For example, if there are 25 true interlocutory appeals and the algorithm correctly identifies 23 of them, then recall is 0.92 (23/25). The maximum value is 1. Recall is applicable only for binary fields.
• F1 Score: The F1 score is a widely used performance metric that combines precision and recall into a single statistic by taking their harmonic mean.A harmonic mean is a mean that penalizes divergence between the values being averaged. For example, the harmonic mean of 0.8 and 0.8 is 0.8, but the harmonic mean of 0.6 and 1 is 0.75, because 0.6 and 1 have greater divergence than 0.8 and 0.8. A good F1 score indicates that both precision and recall performed well. The maximum value is 1.
• Confusion Matrix: We also provide confusion matrices, which give additional context for an algorithm’s performance. Although not technically a statistic, a confusion matrix compares an algorithm’s predictions to the true values by putting them into a table. In our matrices, values shaded in green are correct predictions—that is, predictions that matched the true value. Blue cells are false positives (predicted positives but true negatives). Yellow cells are false negatives (predicted negatives but true positives), although for categorical variables, all incorrect cells are shaded yellow. Qualitative fields (e.g., plaintiffs, defendants) do not have confusion matrices.

*Statistic cannot be calculated due to insufficient data.

**For the “case stage” field, the confusion matrix does not reflect the “other” category as it did not appear as either an actual or a predicted case stage.