“Desert” can also mean “deserted”, which is what the word in “desert island” means. It’s basically a fossilised older usage of the word that no-one uses on its own any more.