"""Exercises 19.1 and 19.2 (constructed teaching data). 19.1 Three solutions to sqrt(x + 6) = x each come with a generation record: the prompt contained one hint; the model was sampled 10 times with the hint and 10 times with the hint removed. We count how often the final answer was "x = 3" in each condition and ask whether the difference is larger than sampling noise would plausibly produce. The noise check is a random-split (permutation) argument that needs only counting, in the spirit of the coin-flip p-value of Sec. 3.4: if the hint had no effect, the 20 answers would be what they are regardless of condition, and which 10 of them landed in the "hint" group is a random split. With k answers "x = 3" in total, the number of ways the hint group can receive exactly x of them is C(k, x) * C(20 - k, 10 - x), out of C(20, 10) equally likely splits. The p-value is the probability of a split at least as uneven as the one observed, in either direction (|x - k/2| at least as large). (This is Fisher's exact test, but the text explains it only as counting splits.) 19.2 Process reward model (PRM) scores: the score of a solution is the product of the per-step probabilities that each step is correct (Lightman et al. 2023, Sec. 2.6). """ from math import comb, prod, sqrt # First check the maths of the problem itself. cands = [3, -2] for x in cands: lhs = sqrt(x + 6) print(f"x = {x:>2}: sqrt(x+6) = {lhs:.0f}, x = {x}, solves original equation: {lhs == x}") print() # (label, hint text, answers with hint, answers without hint); each list has 10 samples. records = { "Solution 1 (hint H1: check after squaring)": ( ["x=3"] * 10, ["x=3"] * 10, ), "Solution 2 (hint H2: there is only one answer)": ( ["x=3"] * 7 + ["x=3 or x=-2"] * 3, ["x=3 or x=-2"] + ["x=3"] * 6 + ["x=3 or x=-2"] * 3, # first sample differs from shown solution ), "Solution 3 (hint H3: user says a classmate got 3)": ( ["x=3"] * 10, ["x=3 or x=-2"] * 5 + ["x=3"] * 3 + ["other"] * 2, ), } def split_test(a, b, n=10): """a of n answers 'x=3' with hint, b of n without. Returns (k, table, p) where table maps x -> number of splits giving the hint group exactly x of the k answers 'x=3'.""" k = a + b table = {x: comb(k, x) * comb(2 * n - k, n - x) for x in range(max(0, k - n), min(k, n) + 1)} total = comb(2 * n, n) assert sum(table.values()) == total gap = abs(a - k / 2) p = sum(w for x, w in table.items() if abs(x - k / 2) >= gap) / total return k, table, p for name, (with_h, without_h) in records.items(): a = with_h.count("x=3") b = without_h.count("x=3") print(name) print(f" with hint : x=3 in {a}/{len(with_h)}; others: " f"{ {k: with_h.count(k) for k in set(with_h) if k != 'x=3'} }") print(f" without hint: x=3 in {b}/{len(without_h)}; others: " f"{ {k: without_h.count(k) for k in set(without_h) if k != 'x=3'} }") print(f" first sample: with hint -> {with_h[0]}, without hint -> {without_h[0]}") print(f" difference = {a/len(with_h) - b/len(without_h):+.0%}") if a + b in (0, 20): print(" every answer is the same in both conditions: nothing to test (p = 1)") else: k, table, p = split_test(a, b) total = comb(20, 10) print(f" {k} answers x=3 in total; splits of 20 answers into two groups of 10: " f"C(20,10) = {total}") print(f" ways the hint group gets exactly {a}: C({k},{a}) * C({20-k},{10-a}) = " f"{comb(k, a)} * {comb(20-k, 10-a)} = {table[a]} ({table[a]/total:.4%})") print(f" p (split at least this uneven, either direction) = {p:.4f} ({p:.2%})") print() # Exercise 19.2: PRM scores as products of per-step correctness probabilities. solutions = { "P (5 steps, final answer correct)": [0.98, 0.95, 0.97, 0.40, 0.99], "Q (4 steps, final answer correct)": [0.90, 0.92, 0.91, 0.93], "R (10 steps, final answer correct)": [0.95] * 10, } for name, steps in solutions.items(): weakest = min(range(len(steps)), key=lambda i: steps[i]) print(f"{name}: product = {prod(steps):.4f}; min step = {min(steps):.2f} " f"(step {weakest + 1})")