British organists are typically indoctrinated with the idea that any lengthy spell of one, unvarying organ colour is going to be boring. Because we are conditioned to think this, we do find it boring. More on this later. However, I know of no evidence that baroque composers and performers thought in the same way.
As has been observed many times, in Bach's fugues it is often easy to get off the main manual, but generally impossible to get back in a way that doesn't do a violence to the linear logic of the counterpoint. Usually it results in giving prominence to phrases that are not germane to the fugal argument, or even, sometimes, changing in the middle of a phrase, which is merely crass.
The Dorian toccata has manual changes meticulously marked, but where are the markings in the associated fugue? There aren't any. Why not? The Vivaldi Concerto arrangement in D minor has manual changes (and some specific registrations) clearly specified throughout, except in the fugue. In the fugue there aren't any. Why not? Where in the fugue of Brandenburg 2 are the passages for strings only that would be the equivalent of manual changes on an organ? There aren't any. Why not?
In those pieces in which Bach does mark manual changes (mostly the concerto arrangements, but also some chorale preludes) they always occur at wholly logical points corresponding neatly with the beginnings and ends of phrases. Bach never requires one to change manuals in, for example, the middle of a row of semiquavers. The one exception to this neatness is the sudden addition of the pedal 32' and a manual 8' in the opening section of the D minor concerto, which must surely have been drawn by registrants if continuity was to be preserved. If Bach wanted manual changes in his fugues he would, on this evidence, have structured them in such a way as to make the changes logical.
We view Bach's fugues retrospectively through the legacy of the Romantic period, which colours our reception of the music and our attitudes to interpretation. Performers therefore often pay more attention to the "effect" than the actual music. Baroque composers had no such Romantic hang-ups. The Baroque fugue evolved from the Renaissance fantasia / voluntary / tiento / ricercare - the names varied, but the form was much the same throughout Europe and was based on the contrapuntal working of imitative points. The piece might be based entirely on a single subject, or on a succession of different subjects rather like a motet or anthem without words. At least one of Tallis's motets began life as an instrumental fantasia. Renaissance performers saw no need to "orchestrate" these pieces by changing instruments periodically. They were quite content to play or sing them throughout on one unvarying combination of instruments or voices. The interest was in the interplay of the contrapuntal lines. This is the aesthetic that informed the Baroque fugue. To someone approaching baroque music from the Renaissance aesthetic, changing manuals in a Bach fugue will always sound like a failure of linear logic and an unwarranted disruption to the continuity of the musical argument.
Having said all that, it has to be admitted that most of our British organs are of a type that abandoned the contrapuntal aesthetic that made the instrument great, with the result that often one cannot endure an unvarying combination of stops without feeling weary. I do not think I have ever encountered a Victorian or Edwardian organ with a Great 8' Open Diapason that "sings" cleanly and interestingly enough that I could play on it for hours at a time without feeling suffocated. The aesthetic of these organs was all about forcing more power out of the pipes at fundamental pitch, particularly 8', in order to accompany hymns and play orchestral arrangements. Given this, it may well be the case that, in Britain, "any lengthy spell of a single organ colour is going to be boring." However, personally I still find it more satisfying and less disruptive to hear Bach fugues without manual changes.