You're absolutely right. Music should be accessible and easy to remember. Anyone who doesn't think so is only putting on airs in order to show their elitist credentials. Who can argue with the following quote about another "modern" composer?:
"This composer goes his own path, and a dreary, eccentric, and tiresome path it is: learning, learning, and nothing but learning, but not a bit of nature or melody. And, after all, it is but a crude and undigested learning, without method or arrangement, a seeking after curious modulations, a hatred of ordinary progressions, a heaping up of difficulties, until all the pleasure and patience are lost."
Except that it's coming from a 19th-century critic who was convinced that Beethoven's music was too difficult for its own sake. It was not a unique opinion of Beethoven's music either; for the next century or so after his death, the Ninth Symhony was considered to be tainted by its composer's deafness and not worth the trouble to perform. Now it's played at the Olympics every four years.
None of us who love Ligeti's music (or Xenakis', or Schoenberg's, or Bartok's, or Messiaen's) believe that their music will be equally beloved someday. But it's false, reverse elitism to claim that no one actually listens earnestly. And we didn't all go to Juilliard, either. I know Kubrick sure didn't.