One of the core beliefs of rationalism is that Intelligence™ is the sole determinant of outcomes, overriding resource imbalances, structural factors, or even just plain old luck. For example, since Elon Musk is so rich, that must be because he is very Intelligent™, despite all of the demonstrably idiotic things he has said over the years. So, even in an artificial scenario like chess, they cannot accept the fact that no amount of Intelligence™ can make up for a large material imbalance between the players.

There was a sneer two years ago about this exact question. I can’t blame the rationalists though. The concept of using external sources outside of their bubble is quite unfamiliar to them.

The dumb strawman protagonist is called “Mr. Humman” and the ASI villain is called “Mr. Assi”. I don’t think any parody writer trying to make fun of rationalist writing could come up with something this bad.

The funniest comment is the one pointing out how Eliezer screws up so many basic facts about chess that even an amateur player can see all the problems. Now, if only the commenter looked around a little further and realized that Eliezer is bullshitting about everything else as well.

Oh boy, another AI doom video popped up on my feed. Time for more morbid curiosity. The topic is about Big Yud and Nate Soares’s new book (“If You Build It, Everyone Dies”) about how AI is gonna kill us all. I have better things to waste 30 minutes on, so I’m not watching the full video, but the thumbnail (“The 7 Minute War”) kinda suggests what the contents are gonna be.

Thankfully, the description of the video has a Google doc with their sources! I’m sure it’s full of hard evidence from careful experiments that logically demonstrate why their doomsday scenario is something to worry about, not just a random assortment of Anthropic blog posts and completely unrelated events.

Somehow, there are a bunch of sources for the first 2 minutes of the video.

“In the New York Times’ best-selling book, which was endorsed by Nobel laureates and the godfathers of AI” Geoffrey Hinton — Personal estimate >50% existential risk.

Geoffrey “All radiologists will be replaced in 5 years” Hinton, Nobel laureate in physics, famous for his work in … physics.

“researchers from the Machine Intelligence Research Institute describe in detail one potential example future” Machine Intelligence Research Institute — The Sable scenario from If Anyone Builds It, Everyone Dies by Yudkowsky & Soares. Fictional narrative illustrating risks, not prediction.

This is not the first we’ve seen from MIRI, and I have a feeling it will not be the last. The monster under my bed is a fictional narrative illustrating risks, not prediction.

“AI researchers have known this has been potentially a very bad idea since at least 2024” Anthropic/Apollo Research — Multiple 2024 papers document deceptive/self-preserving behaviors in controlled evaluations.

They are still trying to flog the Anthropic/Apollo Research claims that chatbots will lie to you if you tell them to lie to you.

“They spin up 200,000 GPUs and let Sable think for 16 hours straight” xAI/NVIDIA — Colossus supercomputer in Memphis scaling toward ~200,000 GPUs for Grok training.

What does this even demonstrate? Some people can do some stuff with some GPUs? I ate some oatmeal today. Now everyone should be thoroughly convinced of my oatmeal-eating abilities.

I watched for a few seconds around the timestamp, and it seems to be the beginning of their scifi story, I mean, AGI scenario. Yes, if you want to convince people that your scenario is plausible, I’m sure this is the part that you need serious amounts of evidence for. Remember, almost half the sources have timestamps for the first two minutes of the video.

“a stunt to see if Sable can crack famous math problems like the Riemann hypothesis” Clay Mathematics Institute — Riemann Hypothesis remains unsolved after 160+ years, considered most famous unsolved problem in pure mathematics.

Again, what does this demonstrate? I tried solving P vs NP with a cheeseburger. That didn’t work either. The only purpose of mentioning this is for narrative window dressing, because Math Is For Smart People.

These are the sources for just the first two minutes. After that, they get a bit sparse.

“Back in 2024, smaller models showed flashes of the same behavior” Multiple Papers — Documented deception/scheming findings in frontier models.

“Claude 3.7 was caught repeatedly cheating on coding tasks even when told to stop”

More Anthropic blog posts and system cards? Come on, I can’t sneer the same thing twice in one post!

“Steal cryptocurrency from weak exchanges just like hackers did to Mt. Gox in 2011” U.S. Department of Justice — Russian nationals charged for 2011 Mt. Gox hack. 647,000-850,000 BTC stolen.

I don’t know what this has to do with supporting the validity of their AI doomsday scenario, but kudos to them for showing why cryptocurrency is also stupid, I guess.

“or Bybit in 2025” Reuters/FBI — Largest cryptocurrency theft to date. FBI attributed to North Korean Lazarus Group.

More? I guess this is hard evidence for showing why cryptocurrency is stupid. I still don’t understand how this demonstrates that AI is scary.

“Reminder, this scenario is based on years of technical research by the Machine Intelligence Research Institute, laid out in the book If Anyone Builds It Everyone Dies” MIRI — Meta-commentary explaining the scenario is illustrative, not predictive.

I knew MIRI would be back. It’s illustrative, not predictive! Please don’t blame us if none of this even remotely happens! But it’s based on years of technical research. An entire graduate student’s worth of output in a decade.

“In 2023, a human gave an LLM access to the internet and created an X account, Terminal of Truths, which gained hundreds of thousands of followers and launched its own crypto meme coin that reached a literal billion dollar market cap” Terminal of Truths — Real-world example of AI agent gaining social media following and wealth.

The link they give references … another one of their own videos. You really are not beating the circular reference allegations here. Even if the purported story is somehow accurate, this again demonstrates how cryptocurrency is stupid. At least they use an LLM as a prop this time.

“Gain of function research. Any one of them could be hijacked to unleash catastrophe.” Science/CIDRAP — Fouchier and Kawaoka created ferret-transmissible H5N1. Controversial GOF research began 2011.

I think Yud is obsessed with this topic in particular. Better than diamondoid bacteria, I guess. Again, the AI just magically comes in and uses this stuff somehow.

“The number one and number two most cited living scientists across all fields think scenarios like this are not only possible but likely to happen. And the average AI researcher thinks there is a 16% chance of AI causing human extinction.”

Okay, let me be completely serious for this one. What would someone do if they truly believed that their work would lead to a horrible disaster, such as the extinction of humanity? Would they continue to work in the field, let alone make enough contributions to rise to the top? Alright I’m done.

Every time I hear a moderate AI argument (e.g. AI will be an aid for searching literature or writing code), it’s like, “Look, it’s impressive that the AI managed to do this. Sure, it took about three dozen prompts over five hours, made me waste another five hours because it generated some completely incorrect nonsense that I had to verify, produced an answer that was much lower quality than if I had just searched it up myself, and boiled two lakes in the process. You should acknowledge that there is something there, even if it did take a trillion dollars of hardware and power to grind the entire internet and all books and scientific papers into a viscous paste. Your objections are invalid because I’m sure things are gonna improve because Progress.”

I am doubly annoyed when I turn my back and they switch back to spouting nonsense about exponential curves and how AI is gonna be smarter than humans at literally everything.

I did not think anything could make me sympathetic to the authors who put 0.1pt white text in their papers so that any reviewer lazy enough to use an LLM would get prompt injected, but here we are.

More AI bullshit hype in math. I only saw this just now so this is my hot take. So far, I’m trusting this r/math thread the most as there are some opinions from actual mathematicians: https://www.reddit.com/r/math/comments/1o8xz7t/terence_tao_literature_review_is_the_most/

Context: Paul Erdős was a prolific mathematician who had more of a problem-solving style of math (as opposed to a theory-building style). As you would expect, he proposed over a thousand problems for the math community that he couldn’t solve himself, and several hundred of them remain unsolved. With the rise of the internet, someone had the idea to compile and maintain the status of all known Erdős problems in a single website (https://www.erdosproblems.com/). This site is still maintained by this one person, which will be an important fact later.

Terence Tao is a present-day prolific mathematician, and in the past few years, he has really tried to take AI with as much good faith as possible. Recently, some people used AI to search up papers with solutions to some problems listed as unsolved on the Erdős problems website, and Tao points this out as one possible use of AI. (I personally think there should be better algorithms for searching literature. I also think conflating this with general LLM claims and the marketing term of AI is bad-faith argumentation.)

You can see what the reasonable explanation is. Math is such a large field now that no one can keep tabs on all the progress happening at once. The single person maintaining the website missed a few problems that got solved (he didn’t see the solutions, and/or the authors never bothered to inform him). But of course, the AI hype machine got going real quick. GPT5 managed to solve 10 unsolved problems in mathematics! (https://xcancel.com/Yuchenj_UW/status/1979422127905476778#m, original is now deleted due to public embarrassment) Turns out GPT5 just searched the web/training data for solutions that have already been found by humans. The math community gets a discussion about how to make literature more accessible, and the rest of the world gets a scary story about how AI is going to be smarter than all of us.

There are a few promising signs that this is getting shut down quickly (even Demis Hassabis, CEO of DeepMind, thought that this hype was blatantly obvious). I hope this is a bigger sign for the AI bubble in general.

EDIT: Turns out it was not some rando spreading the hype, but an employee of OpenAI. He has taken his original claim back, but not without trying to defend what he can by saying AI is still great at literature review. At this point, I am skeptical that this even proves AI is great at that. After all, the issue was that a website maintained by a single person had not updated the status of 10 problems inside a list of over 1000 problems. Do we have any control experiments showing that a conventional literature review would have been much worse?

Alesia was the turning point. With the fall of Gaul, Julius Caesar became the most powerful man in the Roman world.

We conveniently forgot about all the civil wars that happened afterwards that resulted in the dismantling of the Roman republic and Caesar taking absolute power. Oh yeah, I see it now.

Most restaurant origin stories involve someone sharing their favorite taco recipe or whatever. These guys start off with a bad pop-history explanation of the battle of Alesia. That’s how you know their food is great.

There’s more where the founder of the company talks about how he really hated working at his family’s restaurant while growing up (good sign). Knowing that his family came from China adds another layer of weirdness, in my opinion. The characters where the company name comes from (改革) can be read in both Chinese (gǎigé) and Japanese (kaikaku) and mean the same thing (reform) in both languages. It just feels so weird that he talks so much fluff about Julius Caesar, mentions his family from China and then, out of the blue, uses a Japanese name for the company. What is with these people fetishizing ancient Rome and Japan so much?

In the 21st century, the Antichrist is a Luddite who wants to stop all science.

As opposed to the current administration that is destroying science by cutting the NSF’s funding. An administration that Peter Thiel supports. He might want to look into that.

Lately I’ve been mildly annoyed when I just want to relax and watch gaming videos on Youtube and I see recommendations for some AI critihype. Out of morbid curiosity, I decided to click on one of them and of course the “original paper” the video is based on is the stupid Anthropic blog post about how the AI blackmailed someone (after it was told to blackmail someone). I was even more annoyed to find out how popular it is, but at least it shows how the general public has such a negative opinion of AI. Some of the comments are thankfully pushing back against the video and focusing on the real harms.

I thought that by now we would have learned from the tobacco companies to never trust “research” done by a company about their own products.

Oh yeah, he wrote an update saying that the LLM is still great, even if the result is already known, because it saves him time. We have come full circle back to the exact same value proposition as the vibe coders.

After seeing this, I reminded myself that I’ve seen this type of thing happen before. Over the past half year, so many programmers enthusiastically embraced vibe coding after seeing one or two impressive results when trying it out for themselves. We all know how that is going right now. Baldur Bjarnason had some great essays (1, 2) about the dangers of relying on self-experimentation when judging something, especially if you’re already predisposed into believing it. It’s like a mark believing in a psychic after he throws out a couple dozen vague statements and the last one happens to match with something meaningful, after the mark interprets it for him.

Edit: Accidentally hit reply too early.

Pigs will be a true method of space exploration if they can fly to Mars in 1 hour.

Not sure if analog turing machines provide any new capabilities that digital TMs do, but I leave that question for the smarter people in the subject of theorethical computer science

The general idea among computer scientists is that analog TMs are not more powerful than digital TMs. The supposed advantage of an analog machine is that it can store real numbers that vary continuously while digital machines can only store discrete values, and a real number would require an infinite number of discrete values to simulate. However, each real number “stored” by an analog machine can only be measured up to a certain precision, due to noise, quantum effects, or just the fact that nothing is infinitely precise in real life. So, in any reasonable model of analog machines, a digital machine can simulate an analog value just fine by using enough precision.

There aren’t many formal proofs that digital and analog are equivalent, since any such proof would depend on exactly how you model an analog machine. Here is one example.

Quantum computers are in fact (believed to be) more powerful than classical digital TMs in terms of efficiency, but the reasons for why they are more powerful are not easy to explain without a fair bit of math. This causes techbros to get some interesting ideas on what they think quantum computers are capable of. I’ve seen enough nonsense about quantum machine learning for a lifetime. Also, there is the issue of when practical quantum computers will be built.

From the ChatGPT subreddit: Gemini offers to pay me for a developer to fix its mess

Who exactly pays for it? Google? Or does Google send one of their interns to fix the code? Maybe Gemini does have its own bank account. Wow, I really haven’t been keeping up with these advances in agentic AI.

OpenAI claims that their AI can get a gold medal on the International Mathematical Olympiad. The public models still do poorly even after spending hundreds of dollars in computing costs, but we’ve got a super secret scary internal model! No, you cannot see it, it lives in Canada, but we’re gonna release it in a few months, along with GPT5 and Half-Life 3. The solutions are also written in an atrociously unreadable manner, which just shows how our model is so advanced and experimental, and definitely not to let a generous grader give a high score. (It would be real interesting if OpenAI had a tool that could rewrite something with better grammar, hmmm…) I definitely trust OpenAI’s major announcements here, they haven’t lied about anything involving math before and certainly wouldn’t have every incentive in the world to continue lying!

It does feel a little unfortunate that some critics like Gary Marcus are somewhat taking OpenAI’s claims at face value, when in my opinion, the entire problem is that nobody can independently verify any of their claims. If a tobacco company released a study about the effects of smoking on lung cancer and neglected to provide any experimental methodology, my main concern would not be the results of that study.

Edit: A really funny observation that I just thought of: in the OpenAI guy’s thread, he talks about how former IMO medalists graded the solutions in message #6 (presumably to show that they were graded impartially), but then in message #11 he is proud to have many past IMO participants working at OpenAI. Hope nobody puts two and two together!

Hmm, should I be more worried and outraged about genocides that are happening at this very moment, or some imaginary scifi scenario dreamed up by people who really like drawing charts?

One of the ways the rationalists try to rebut this is through the idiotic dust specks argument. Deep down, they want to smuggle in the argument that their fanciful scenarios are actually far more important than real life issues, because what if their scenarios are just so bad that their weight overcomes the low probability that they occur?

(I don’t know much philosophy, so I am curious about philosophical counterarguments to this. Mathematically, I can say that the more they add scifi nonsense to their scenarios, the more that reduces the probability that they occur.)

There’s really no good way to make any statements about what problems LLMs can solve in terms of complexity theory. To this day, LLMs, even the newfangled “reasoning” models, have not demonstrated that they can reliably solve computational problems in the first place. For example, LLMs cannot reliably make legal moves in chess and cannot reliably solve puzzles even when given the algorithm. LLM hypesters are in no position to make any claims about complexity theory.

Even if we have AIs that can reliably solve computational tasks (or, you know, just use computers properly), it still doesn’t change anything in terms of complexity theory, because complexity theory concerns itself with all possible algorithms, and any AI is just another algorithm in the end. If P != NP, it doesn’t matter how “intelligent” your AI is, it’s not solving NP-hard problems in polynomial time. And if some particularly bold hypester wants to claim that AI can efficiently solve all problems in NP, let’s just say that extraordinary claims require extraordinary evidence.

Koppelman is only saying “complexity theory” because he likes dropping buzzwords that sound good and doesn’t realize that some of them have actual meanings.

I study complexity theory and I’d like to know what circuit lower bound assumption he uses to prove that the AI layoffs make sense. Seriously, it is sad that the people in the VC techbro sphere are thought to have technical competence. At the same time, they do their best to erode scientific institutions.

Username called “The Dao of Bayes”. Bayes’s theorem is when you pull the probabilities out of your posterior.

知者不言，言者不知。 He who knows (the Dao) does not (care to) speak (about it); he who is (ever ready to) speak about it does not know it.