"""Exercise 21.2: agreement between AI labels and human labels (constructed teaching data). Twelve comparison pairs (response 甲 vs response 乙). For each pair we have one human label and two AI labels: run 1 shows 甲 first, run 2 shows 乙 first (order swapped). We compute - agreement of run 1 with the human label, - how often the two runs agree with each other (order consistency), - among inconsistent pairs, whether the AI picked the first-shown or second-shown response, - agreement among consistent pairs, and a score that counts an inconsistent pair as half. """ # (pair id, human, AI run 1 with 甲 first, AI run 2 with 乙 first) pairs = [ (1, "甲", "甲", "甲"), (2, "乙", "乙", "乙"), (3, "甲", "甲", "乙"), (4, "甲", "甲", "甲"), (5, "乙", "甲", "乙"), (6, "乙", "乙", "乙"), (7, "甲", "乙", "乙"), (8, "甲", "甲", "甲"), (9, "乙", "甲", "乙"), (10, "甲", "甲", "乙"), (11, "乙", "乙", "乙"), (12, "乙", "乙", "甲"), ] n = len(pairs) run1_agree = [p for p, h, r1, r2 in pairs if r1 == h] consistent = [(p, h, r1) for p, h, r1, r2 in pairs if r1 == r2] inconsistent = [(p, r1, r2) for p, h, r1, r2 in pairs if r1 != r2] first_bias = [p for p, r1, r2 in inconsistent if r1 == "甲" and r2 == "乙"] # chose whichever was first second_bias = [p for p, r1, r2 in inconsistent if r1 == "乙" and r2 == "甲"] # chose whichever was second cons_agree = [p for p, h, r in consistent if r == h] print(f"run 1 agrees with human: {len(run1_agree)}/{n} = {len(run1_agree)/n:.1%} pairs {run1_agree}") print(f"order-consistent pairs: {len(consistent)}/{n} = {len(consistent)/n:.1%}") print(f"inconsistent pairs: {[p for p, *_ in inconsistent]}; chose first-shown: {first_bias}; " f"chose second-shown: {second_bias}") print(f"among consistent pairs, agree with human: {len(cons_agree)}/{len(consistent)} = " f"{len(cons_agree)/len(consistent):.1%} (disagree: " f"{[p for p, h, r in consistent if r != h]})") score = (len(cons_agree) + 0.5 * len(inconsistent)) / n print(f"score counting inconsistent pairs as 1/2: ({len(cons_agree)} + 0.5*{len(inconsistent)})/{n}" f" = {score:.1%}")