Apple Machine Learning Research has a new preprint: “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity.” [Apple, PDF] “Lar…
That would be the best way to actively catch the cheating happening here, given that the training datasets remain confidential. But I also don’t know that it would be conclusive or convincing unless you could be certain that the problems in the private set were similar to the public set.
In any case either you’re doubledipping for credit in multiple places or you absolutely should get more credit for the scoop here.
I’d just write the list then assign randomly. Or perhaps pseudorandomly like sort by hash and then split in two.
One problem is that it is hard to come up with 20 or more completely unrelated puzzles.
Although I don’t think we need a large number for statistical significance here, if it’s like 8/10 solved in the cheating set and 2/10 in the hold back set.
That would be the best way to actively catch the cheating happening here, given that the training datasets remain confidential. But I also don’t know that it would be conclusive or convincing unless you could be certain that the problems in the private set were similar to the public set.
In any case either you’re doubledipping for credit in multiple places or you absolutely should get more credit for the scoop here.
I’d just write the list then assign randomly. Or perhaps pseudorandomly like sort by hash and then split in two.
One problem is that it is hard to come up with 20 or more completely unrelated puzzles.
Although I don’t think we need a large number for statistical significance here, if it’s like 8/10 solved in the cheating set and 2/10 in the hold back set.