"""Exercise 16.3: what McAleese et al.'s 24% and 6% can and cannot show. Reported numbers (McAleese et al. 2024, arXiv 2407.00215, section 3.6): - 24%: among answers rated "flawless" whose sampled critique identified a problem, the share where contractors, reviewing with the critique, said it found a problem that substantially lowered the rating; - 6%: in a separate replication without critiques, the share of "flawless" answers that a second contractor rated similarly poorly. The paper does not report the share s of "flawless" answers on which the critique identified a problem. The values of s below are constructed for teaching, not from the paper. """ from fractions import Fraction as F with_critique_among_flagged = F(24, 100) without_critique_among_all = F(6, 100) print(f"mistaken reading: ratio 24% / 6% = {with_critique_among_flagged / without_critique_among_all}") # Over all "flawless" answers, the share where a problem was found with the critique is 0.24 * s. breakeven = without_critique_among_all / with_critique_among_flagged print(f"0.24 * s equals 6% when s = 6/24 = {float(breakeven):.0%}") for s in (F(10, 100), F(50, 100)): # constructed for teaching print(f"s = {float(s):.0%}: share of all flawless answers = 0.24 * {float(s):.2f} = {float(with_critique_among_flagged * s):.1%}")