Following the Reformation, a body of music was produced with the specific intention of being NOT-Catholic. These are hymns written for vernacular texts, in a major or minor key, and it is not melismatic but has, typically, one note per syllable. It has a completely different effect on the listener. It arises from a different spirituality.
Bach's church music, such as the cantatas, compilations of cantatas, and organ pieces incorporates Lutheran hymn tunes such as Wachet auf, Eine feste burg (used in Mendelsson's Reformation symphony), Nun danket alle Gott, etc. There is, of course Bach the B-Minor Mass, but it would be extremely unusual to use that setting liturgically.
Mixing music based on Lutheran themes into a Catholic liturgy creates an unhappy and incoherent blend from both a stylistic and spiritual aspect; think of pouring Vindaloo sauce on sushi.
The above comments apply to some extent to organ music when much of this, too, is based on Lutheran themes, and there is another point. There is plenty of Catholic organ music of the finest quality, written on Catholic liturgical themes, which gets squeezed out by the Protestant composers. It deserves performance and is in keeping with the style and spirituality of Gregorian chant. Too much of it is neglected - Frescobaldi, Muffat, Couperin, de Grigny, Duruffle, Messaien. There will always be plenty of opportunity to listen to Bach and Buxtehude at concerts or on the radio or in recordings. It is interesting that at the London Oratory, who set the standard in these matters, Bach is played regularly at Vespers but scarcely at Mass.
What about Mozart or Haydn, whose music is not modal? The line is not a hard and fast one. From the beginning of the eighteenth century and through to the start of the twentieth, European music was generally written in a major or minor key. The key point is "typically modal and melismatic", and of course there are exceptions. A musical setting for a Latin Mass text is obviously Catholic music.
Aside from settings for the Ordinary of the Mass, the sort of music that composers write is naturally going to be influenced by the composers' spiritual orientation, consciously or otherwise. The Lutheran and Calvinist composers of the immediate post-Reformation period were consciously striving to produce music which reflected their theology and attitudes. These were redefining their understanding of the Christian faith in a new and different way. The music was written to promote that aim.
Later Protestant music reflects the subsequent developments - the High Church Anglican music of the seventeenth century, the Nonconformism of the eighteenth, the Oxford movement of the mid-nineteeth and the muscular Christianity of the British Imperial period.
Another issue is the practical one regarding its place in the Mass. Protestant hymns normally find their way into the Mass as replacements for parts of the Proper - ie the Introit, Offertory and Communion antiphons. These form part of the reading. There are specific instructions about this in the General Instruction for the Roman Missal.
there are four options...
- the antiphon from The Roman Missal or the Psalm from the Roman Gradual as set to music there or in another musical setting;
- the seasonal antiphon and Psalm of the Simple Gradual;
- a song from another collection of psalms and antiphons, approved by the Conference of Bishops or the diocesan Bishop, including psalms arranged in responsorial or metrical forms;
- a suitable liturgical song similarly approved by the Conference of Bishops or the diocesan Bishop.
As a practical matter, congregational hymns do not work well at the Offertory or Communion as at both times they are sitting, kneeling, looking for change to put in the collection or queueing for communion.