Exactly. It’s overtrained on the test, ignoring the differences. If you instead used something it recognises but doesn’t recognise as the test pattern (having the same tokens/embeddings) it will perform better. I’m not joking, it’s a common tactic to get around censoring. You’re just going around the issue. What I’m saying is they’ve trained the model so much on benchmarks that it is indeed dumber.

Bet

It’s just overtrained on the puzzle such that it mostly ignores your prompt. Changing a few words out doesn’t change that it recognises the puzzle. Try writing it out in ASCII or uploading an image with it written or some other weird way that it hasn’t been specifically trained on and I bet it actually performs better.

They finetuned 1.5-3b models. This is a non-story

The local models are distilled versions of Qwen or llama or whatever else, not really deepseek’s model. So you get refusals based on the base model primarily, plus whatever it learned from the distilling. If it’s Qwen or another Chinese model then it’s more likely to refuse but a llama model or something else could pick it up to a lesser extent.

When hedge funds decide to flip the switch on something the reaction never looks rational. Meta was green today ffs.

Yep, was going to mention a study from a few years ago that threw out most of this data as junk since they found many were counting stretched flaccid as “erect”.