You can still leverage knowledge from a foundation model in a smaller fine-tuned one.

So the model might have learned general OOP principles from Java but it then drops redundant parameters about specific conventions like AbstractFactoryBuilders when it specialises on a language like Python which has no notion of Interfaces.

Likewise real world knowledge might help distinguish between accounting and database transactions when writing a banking application but you don’t necessarily need your coding assistant to have memorised all the world cup winners since 1966.

These models are unwieldy so I think it makes a lot of sense to try and find ones that are tuned efficiently.