• scruiser@awful.systems
    link
    fedilink
    English
    arrow-up
    12
    ·
    2 days ago

    Another thing that’s been annoying me about responses to this paper… lots of promptfondlers are suddenly upset that we are judging LLMs by abitrary puzzle solving capabilities… as opposed to the arbitrary and artificial benchmarks they love to tout.

    • diz@awful.systems
      link
      fedilink
      English
      arrow-up
      10
      ·
      edit-2
      2 days ago

      Yeah any time its regurgitating an IMO problem it’s a proof it’salmost superhuman, but any time it actually faces a puzzle with unknown answer, this is not what it is for.