• YourNetworkIsHaunted@awful.systems
    link
    fedilink
    English
    arrow-up
    7
    ·
    18 hours ago

    That would be the best way to actively catch the cheating happening here, given that the training datasets remain confidential. But I also don’t know that it would be conclusive or convincing unless you could be certain that the problems in the private set were similar to the public set.

    In any case either you’re doubledipping for credit in multiple places or you absolutely should get more credit for the scoop here.

    • diz@awful.systems
      link
      fedilink
      English
      arrow-up
      6
      ·
      16 hours ago

      I’d just write the list then assign randomly. Or perhaps pseudorandomly like sort by hash and then split in two.

      One problem is that it is hard to come up with 20 or more completely unrelated puzzles.

      Although I don’t think we need a large number for statistical significance here, if it’s like 8/10 solved in the cheating set and 2/10 in the hold back set.