• MoonMelon@lemmy.ml
    link
    fedilink
    English
    arrow-up
    0
    ·
    9 days ago

    Reading his response, I think calling it “slop” isn’t being totally fair, but it does sound like he should hand it off again or close the project. Not having test coverage for something is bad, but it happens. It sounds like the alternatives have this issue also. But the sailing comment is kind of tragic. Just go sailing, dude. Unless you have a phylactery under your desk the project will outlive you anyway, and honestly that’s the best compliment a developer can get.

      • MoonMelon@lemmy.ml
        link
        fedilink
        English
        arrow-up
        0
        ·
        9 days ago

        I rewrote the rsync test suite in python from the old shell script design. I did the design for that myself (and I’m really quite pleased with it), but used claude with cross-checks from codex and gemini to do the grunt work. I did not just vibe-code “convert test suite to python”… I used AI tools to do the grunt work because they are good at that. I reviewed every part of it myself and ran through a huge amount of CI time getting it right

        If what he claims is true then he’s using LLMs for test coverage with significant editing by hand. I hate LLMs, but even I have to admit this seems like one of the few, valid use cases of LLM assisted coding. Unless “slop” has become one of those words that’s just lost all meaning.

        • diz@awful.systems
          link
          fedilink
          English
          arrow-up
          0
          ·
          edit-2
          8 days ago

          It’s a perfect example of how “using LLMs for test coverage” can also be harmful. He expected the tests to to prevent introduction of said regressions, probably based on a combination of the quantity of tests and their style (they look like what decent human written tests look like). But the tests are AI slop, and so they give a lot less value per line of code than he expects, hence a significant regression.

          It is literally useful to call these tests AI slop, and the problem is in part caused by not calling them AI slop, and having consequent inflated expectations. LLMs are not any better at writing tests than at writing other code! It is merely that the bar for tests can, legitimately, be a lot lower (in projects where there would otherwise be no tests at all). Making an exception to calling AI generated tests “slop” is thus counter productive, because it leads people to act as if LLMs are actually better at writing tests than at writing other code, and not just because the bar for tests is frequently very low.

        • AnarchistArtificer@slrpnk.net
          link
          fedilink
          English
          arrow-up
          0
          ·
          8 days ago

          On one of the BlueSky threads going over over the test code, one of the things they uncovered was some stuff running as root which in no world should be necessary. He may not have just prompted Claude to “convert test suite to python”, but there’s a lot there that seem like clear red flags in terms of AI slop code.

          Which is no surprise, really, given that properly proof-reading AI code is often much more labour intensive than just writing the code oneself. It’s easy for things like this to slip through the cracks, even if you are trying to check the AI output

          • MoonMelon@lemmy.ml
            link
            fedilink
            English
            arrow-up
            0
            ·
            9 days ago

            I don’t know anything about rsync aside from as a user, but I am pretty experienced with Python and I admit those tests look really bizarre. If he did “slot machine” code it (a term I wasn’t familiar with) then yeah, I agree that’s slop. If he didn’t, I don’t understand why he made these changes. OK yeah, that’s a bad sign.