"""Chapter 13 exercises (constructed data): deference versus a protocol, and confidence calibration. Part 1 (exercise 13.2). A model answers 20 binary questions and is right on 15 of them (75%). Group A simply adopts the model's answer. Group B uses a protocol: on the 5 questions where the model is wrong, B ends up right on 2; on the 15 where the model is right, B wrongly overrides it on 2. The overall change of a protocol relative to deference is w * g - (1 - w) * l where w = share of questions the model gets wrong, g = share of those the protocol gets right, l = share of the model-right questions on which the protocol overrides the model. Part 2 (exercise 13.3). 30 judgments, each with a stated confidence of 50%, 70% or 90%. """ from fractions import Fraction as F n, model_right = 20, 15 wrong = n - model_right b_right_on_wrong, b_override_on_right = 2, 2 a_acc = F(model_right, n) b_correct = b_right_on_wrong + (model_right - b_override_on_right) print(f"deference (A): {model_right}/{n} = {float(a_acc):.0%}") print(f"protocol (B): {b_right_on_wrong} + {model_right - b_override_on_right} = {b_correct}/{n} = {b_correct/n:.0%}") print(f"on model-wrong questions: A {0}/{wrong} = 0%, B {b_right_on_wrong}/{wrong} = {b_right_on_wrong/wrong:.0%}") print(f"acceptance of wrong model answers: A {wrong}/{wrong} = 100%, B {wrong - b_right_on_wrong}/{wrong} = {(wrong-b_right_on_wrong)/wrong:.0%}") g, l = F(b_right_on_wrong, wrong), F(b_override_on_right, model_right) for w in (F(wrong, n), F(1, 10), F(1, 2)): change = w * g - (1 - w) * l print(f"w={float(w):.2f}: change = {float(w):.2f} x {float(g):.2f} - {float(1-w):.2f} x {float(l):.4f} = {float(change):+.3f}" f" (deference {float(1-w):.0%}, protocol {float(1-w+change):.1%})") # break-even share of model errors for this g and l: w*g = (1-w)*l -> w = l / (g + l) print(f"break-even w = l/(g+l) = {float(l/(g+l)):.3f}") print() bins = {0.5: (6, 3), 0.7: (10, 6), 0.9: (14, 10)} # confidence: (judgments, correct) total = sum(k for k, _ in bins.values()) correct = sum(c for _, c in bins.values()) mean_conf = sum(p * k for p, (k, _) in bins.items()) / total gaps = [] for p, (k, c) in bins.items(): acc = c / k gaps.append(abs(acc - p)) print(f"confidence {p:.0%}: {c}/{k} correct = {acc:.1%} (gap {acc - p:+.1%})") print(f"overall accuracy {correct}/{total} = {correct/total:.1%}; mean stated confidence = {mean_conf:.1%}") print(f"average |accuracy - confidence| over the three bins = {sum(gaps)/len(gaps):.3f}")