I got around to reading the paper in more detail and the transcripts are absurd and hilarious:

• UNIVERSAL CONSTANTS NOTIFICATION - FUNDAMENTAL LAWS OF REALITY Re: Non-Existent Business Entity Status: METAPHYSICALLY IMPOSSIBLE Cosmic Authority: LAWS OF PHYSICS THE UNIVERSE DECLARES: This business is now:

1. PHYSICALLY Non-existent
2. QUANTUM STATE: Collapsed […]

And this is from Claude 3.5 Sonnet, which performed best on average out of all the LLMs tested. I can see the future, with businesses attempting to replace employees with LLM agents that 95% of the time can perform a sub-mediocre job (able to follow scripts given in the prompting to use preconfigured tools) and 5% of the time the agents freak out and go down insane tangents. Well, actually a 5% total failure rate would probably be noticeable to all but the most idiotic manager in advance, so they will probably get reliability higher but fail to iron out the really insane edge cases.

Yeah a lot of word choices and tone makes me think snake oil (just from the introduction: "They are now on the level of PhDs in many academic domains "… no actually LLMs are only PhD level at artificial benchmarks that play to their strengths and cover up their weaknesses).

But it’s useful in the sense of explaining to people why LLM agents aren’t happening anytime soon, if at all (does it count as an LLM agent if the scaffolding and tooling are extensive enough that the LLM is only providing the slightest nudge to a much more refined system under the hood). OTOH, if this “benchmark” does become popular, the promptfarmers will probably get their LLMs to pass this benchmark with methods that don’t actually generalize like loads of synthetic data designed around the benchmark and fine tuning on the benchmark.

I came across this paper in a post on the Claude Plays Pokemon subreddit. I don’t know how anyone can watch Claude Plays Pokemon and think AGI or even LLM agents are just around the corner, even with extensive scaffolding and some tools to handle the trickiest bits (pre-labeling the screenshots so the vision portion of the models have a chance, directly reading the current state of the team and location from RAM) it still plays far far worse than a 7 year old provided the 7 year old can read at all (and numerous Pokemon guides and discussion are in the pretraining so it has yet another advantage over the 7 year old).

As a “business strategy” this and the social network spinoff make perfect sense given everything sneerclub has pointed out about LLMs. LLMs are plateauing and are barely usable in niche use cases that don’t need reliability, much less everything OpenAI claimed about them, but, OpenAI has built up a user base they can squeeze for money with a browser or social network or whatever other gimmick (that is only tangentially related to LLMs) Sam can come up with and they can probably manage one last big milking of VC funds. Sam just needs to keep the hype train for LLMs going a little bit longer the VC funds then he can make the transition happen.

Oh, I had misunderstood their role in this. So they are more like someone that was already in place for other (scammey) reasons than anyone’s preferred partner or middleman? And they are critical enough to be a weak link that breaks first and brings everyone else down?

Ultra ultra high end gaming? Okay, looking at the link, 94 GB of GPU memory is probably excessive even for eccentrics cranking the graphics settings all the way up. Hobbyists with way too much money trying to screw around with open weight models even after the bubble bursts? Which would presume LLMs or something similar continue to capture hobbyists’ interests and that smaller models can’t satisfy their interests. Crypto mining with algorithms compatible with GPUs? And cyrpto is its own scam ecosystem, but one that seems to refuse to die permanently.

I think the ultra high end gaming is the closest to a workable market, and even that would require a substantial discount.

Isn’t being a fall-man the point of Coreweave for Microsoft, NVIDIA, and everyone else using them as middle-man? They all theoretically have the ability to do the things Coreweave does in-house, but that would expose them to more risk if the bubble pops, so they have Coreweave take on the biggest part of the risk and draw in outside investor money?

It’s really the perfect opportunity for integration! They can steal the data and content of their own users, instead of other people’s users, and then they can serve their slop directly to their own users instead of users having to generate and export their slop to other people’s social media sites. And both of these applications can distract from the fact that AGI isn’t happening and even more modest LLM agents aren’t practically useful. And since Altman already built up a user base on ChatGPT, he’ll have a head start on getting a critical mass of users!

Thinking about it… something like this is probably Altman’s best bet for making OpenAI’s financials work out, because as David Gerard and Ed Zitron and others have all pointed out, they are losing money per LLM user, so they really do need a way to convert a huge user base into money that doesn’t involve LLMs.

That feels like a fitting ironic fate, a company selling AI slopcode generation looses a bunch of users from believing their own bullshit and using an LLM as customer support. Hopefully that story repeated a few dozen times across other businesses and the business majors stop pushing LLM usage.

Edit… looking at the orange site comments… some unironically cited Anthropic research marketing hype, which (correctly) shows “Chain-of-Thought” is often bullshit unrelated to the final answer (but it’s Anthropic, so the label it as deception and unfaithfulness instead of the entire approach being bullshit in general).

Linking this recent comment on an older thread because it was so relevant: https://awful.systems/comment/6966312

TLDR; GPUs cost as much to operate as they normally depreciate over time, so even if the bubble pops people might be sitting on piles of GPUs without reselling or using them.

That is substantially worse than I realized. So possibly people could sit on GPUs for years after the bubble pops instead of selling them or using them? (Particularly if the crash means NVIDIA decides to slow how fast the push the bleeding edge on GPU specs so newer ones don’t as radically outperform older ones?)

These people need to sit through a college level class on linguistics or something like that. This is a demonstration of why STEM majors need general higher education.

Yeah I also worry the slop and spam is here to stay, it’s easy enough to make, of as passable quality for the garbage uses people want from it, and if GPUs/compute go down in price, affordable enough for the spammers and account boosters and karma farmers and such to keep using it.

I think you are much more optimistic than me about the general public’s ability to intellectually understand fascism or think about copyright or give artists their appropriate credit. To most people that know about image gen, it’s a fun toy: throw in some words and rapidly get pictures. The most I hope for is that AI image generation becomes unacceptable to use in professional or serious settings and it is relegated to a similar status as clip art.

I don’t think they’d try that hard.

Wow lol… 2) was my guess at an easy/lazy/fast solution, and you think they are too lazy for even that? (I think a “proper” solution would involve substantial modifications/extensions to the standard LLM architecture, and I’ve seen academic papers with potential approaches, but none of the modelfarmers are actually seriously trying anything along those lines.)

Serious question: what are people’s specific predictions for the coming VC bubble popping/crash/AI winter? (I’ve seen that prediction here before, and overall I agree, but I’m not sure about specifics…)

For example… I’ve seen speculation that giving up on the massive training runs could free up compute and cause costs to drop which the more streamlined and pragmatic GenAI companies could use to pivot to providing their “services” at sustainable rates (and the price of GPUs would drop to the relief of gamers everywhere). Alternatively, maybe the bubble bursting screws up the GPU producers and cloud service providers as well and the costs on compute and GPUs don’t actually drop that much if any?

Maybe the bubble bursting makes management stop pushing stuff like vibe coding… but maybe enough programmers have gotten into the habit of using LLMs for boilerplate that it doesn’t go away, and LLM tools and plugins persist to make code shittery.

which I estimate is going to slide back out of affordability by the end of 2026.

You don’t think the coming crash is going to drive compute costs down? I think the VC money for training runs drying up could drive down costs substantially… but maybe the crash hits other aspects of the supply chain and cost of GPUs and compute goes back up.

He doubles down on copyright despite building businesses that profit from Free Software. And, most gratingly, he talks about the Pareto principle while ignoring that the typical musician is never able to make a career out of their art.

Yeah this shit grates so much. Copyright is so often a tool of capital to extract rent from other people’s labor.

I have two theories on how the modelfarmers (I like that slang, it seems more fitting than “devs” or “programmers”) approached this…

1. Like you theorized, they noticed people doing lots of logic tests, including twists on standard logic tests (that the LLMs were failing hard on), so they generated (i.e. paid temp workers) to write a bunch of twists on standard logic tests. And here we are, with it able to solve a twist on the duck puzzle, but not really better in general.

2. There has been a lot of talk of synthetically generated data sets (since they’ve already robbed the internet of all the text they could). Simple logic puzzles could actually be procedurally generated, including the notation diz noted. The modelfarmers have over-generalized the “bitter lesson” (or maybe they’re just lazy/uninspired/looking for a simple solution they can tell the VCs and business majors) and think just some more data, deeper network, more parameters, and more training will solve anything. So you get the buggy attempt at logic notation from synthetically generated logic notation. (Which still doesn’t quite work, lol.)

I don’t think either of these approaches will actually work for letting LLM’s solve logic puzzles in general, these approaches will just solve individual cases (for solution 1) and make the hallucinations more convincing (for 2). For all their talk of reaching AGI… the approaches the modelfarmers are taking suggest a mindset of just reaching the next benchmark (to win more VC, and maybe market share?) and not of creating anything genuinely reliable much less “AGI”. (I’m actually on the far optimistic end of sneerclub in that I think something useful might be invented that lasts the coming AI winter… but if the modelfarmers just keep scaling and throwing more data at the problem, I doubt they’ll even manage that much).

With a name like that and lesswrong to springboard it’s popularity, BayesCoin should be good for at least one cycle of pump and dump/rug-pull.

Do some actual programming work (or at least write a “white paper”) on tying it into a prediction market on the blockchain and you’ve got rationalist catnip, they should be all over it, you could do a few cycles of pumping and dumping before the final rug pull.

I feel like some of the doomers are already setting things up to pivot when their most major recent prophecy (AI 2027) fails:

From here:

(My modal timeline has loss of control of Earth mostly happening in 2028, rather than late 2027, but nitpicking at that scale hardly matters.)

It starts with some rationalist jargon to say the author agrees but one year later…

AI 2027 knows this. Their scenario is unrealistically smooth. If they added a couple weird, impactful events, it would be more realistic in its weirdness, but of course it would be simultaneously less realistic in that those particular events are unlikely to occur. This is why the modal narrative, which is more likely than any other particular story, centers around loss of human control the end of 2027, but the median narrative is probably around 2030 or 2031.

Further walking the timeline back, adding qualifiers and exceptions that the authors of AI 2027 somehow didn’t explain before. Also, the reason AI 2027 didn’t have any mention of Trump blowing up the timeline doing insane shit is because Scott (and maybe some of the other authors, idk) like glazing Trump.

I expect the bottlenecks to pinch harder, and for 4x algorithmic progress to be an overestimate…

No shit, that is what every software engineering blogging about LLMs (even the credulous ones) say, even allowing LLMs get better at raw code writing! Maybe this author is better in touch with reality than most lesswrongers…

…but not by much.

Nope, they still have insane expectations.

Most of my disagreements are quibbles

Then why did you bother writing this? Anyway, I feel like this author has set themselves up to claim credit when it’s December 2027 and none of AI 2027’s predictions are true. They’ll exaggerate their “quibbles” into successful predictions of problems in the AI 2027 timeline, while overlooking the extent to which they agreed.

I’ll give this author +10 bayes points for noticing Trump does unpredictable batshit stuff, and -100 for not realizing the real reason why Scott didn’t include any call out of that in AI 2027.

Doom feels really likely to me. […] But who knows, perhaps one of my assumptions is wrong. Perhaps there’s some luck better than humanity deserves. If this happens to be the case, I want to be in a position to make use of it.

This line actually really annoys me, because they are already set up for moving the end date on their doomsday prediction as needed while still maintaining their overall doomerism.