yes, you're right of course, there are plenty of sorts of music that derive half their interest from modulations, and they wouldn't respond well to a moveable clef. Also those of us brought up in the era of Hymns Not-quite-so-Ancient and Nearly-Modern (the red covered edition) will have horrible memories of everything being moved into the keys of C, G, and F, with the result that church music became very dull. Even those of us without perfect pitch have pitch-memory (in fact I don't know if it's possible to tell the difference between perfect pitch and very good pitch-memory) - we can remember that our key now is different to what it was 10 minutes ago.
Singers also always know a very wrong pitch because the notes feel different to sing (I have no absolute pitch and a poor pitch memory, but I know if an organ is pitched high when I try to sing hymns).
Incidentally, on musicians and absolute pitch: there is an interesting side-line on this. Someone, I wish I could remember who, did some tests using Shepard tones. These are notes made by blending a lot of octaves. As you climb the scale, you de-intensify the upper octaves and increase the intensity of the lower octaves, so that by the time you've climbed an octave, you've arrived back at precisely the same mix as you had at the start. It's a bit like a sound version of Escher's staircase. This allows people in YouTube to make cute videos where strange graphics are accompanied by a tone that is always going upwards but never gets anywhere. Shepard tones can also be used in more interesting ways. If you start with a note, and then play the note that is exactly half an octave away, the question is, did it go up, or down? Every note must be higher or lower than another, right? The trouble with this one is that the notes both ways are exactly the same and exactly the same distance.
When tested, it turns out that some people hear it higher, others lower, and it depends on the absolute pitches involved - and on the listener. Americans generally hear it one way, Europeans the other. It doesn't depend on musical experience - total non-musicians still show the effect. This suggests that absolute pitch is inbuilt to humans and not just a feature of a small subgroup of musicians - though quite what it does and how it works is a mystery (and why it's different in Americans and Brits, who have similar genetics and similar culture).
But I still maintain that for most people, singing notes within a relative scale based on a tonic feels a lot more natural than trying to sing notes based on their position on a fixed stave - and I also maintain that any composer who finds themselves in 4 sharps without having got their by valid and innocent means deserves not to be played. Certainly if they think E major has a special magic colour - because to me, it doesn't. I can't tell E major from D major, or C major for that matter.