Sooner or later a machine acquires one program that disagrees with it. Everything else is current, and this one thing was built against a library that is two versions behind, and it will not start without exactly that version.
Local AI is where most people meet this now. A speech or vision model ships with the GPU libraries it was compiled against, and the workstation running it is kept current for everything else it does: the display driver, the games, the video editor, the other five things that were installed on purpose. The model wants the machine to go back. Everything else wants it to stay where it is.
The temptation is to change the machine
The obvious move is to downgrade the system library until the model is happy. It works immediately, which is what makes it dangerous. Everything else that expected the newer version is now running against the older one, and the failures that produces do not arrive with a message explaining what you did to cause them.
The second problem arrives later. The next system update puts the library back, at a moment nobody chose, and the model stops working with an error that has no obvious connection to the update that broke it. A fix you have to remember to reapply is not a fix, it is a recurring appointment.
Give the libraries to the program, not to the machine
The alternative is to stop treating the library as a property of the computer. The program gets its own directory holding its own copy of what it was built against, and a small launcher puts that directory ahead of the system one in the library search path, for that process only.
Everything started any other way sees the system libraries and carries on exactly as before. The machine stays current. The one program that needs the past gets the past, scoped to itself, and the two facts stop being in conflict because they are no longer answering the same question.
The part that catches people out
A program has more entry points than you think. There is the command you type, and there is the desktop launcher, and there is the service definition that starts it at boot, and each one builds its own environment from scratch.
Scope the libraries in only one of those, and the result is the worst kind of bug: it works when you start it by hand and fails when the machine starts it, or the other way round, and the difference is invisible in the logs. Every entry point needs the same environment applied, and the way to be sure is to start it each way once and watch it work.
What it costs
Three things, and they are worth naming before choosing this.
- Disk. A second copy of a full GPU library set is comfortably over a gigabyte. If two programs need two different old versions, it is two copies.
- Update work. The scoped copy is outside the package manager, so nothing tells you it is out of date. It gets updated when you decide to update it.
- One more thing to document. An undocumented environment override is a trap for whoever meets it next, including you in eight months.
The reason to accept those costs is the one you get in exchange: the system update that would otherwise have broken the model becomes an ordinary system update.
The general shape
This is not really about GPUs. It is the same answer for a service that needs an old runtime, a build that needs a specific compiler, or a script that needs a library version nothing else will tolerate.
Pin a dependency to the process that needs it, never to the machine. The machine is shared, and anything you change there is a change you made on behalf of every other program on it, whether or not you meant to.