Do I have to buy a set for 5.1 surround audio, or can I just buy 5 speakers and one subwoofer separately to potentially save money and link them to my AVR?
You can definitely just use any 5 speakers. But typically you would want at least the left, right, and center speaker to be from the same brand and even line to make the sound signature more cohesive. Good idea to have the surrounds from the same line too, but not necessary. And the sub could be from any decent manufacturer.
Potentially even the centre channel may be a bad idea - these are frequently lacklustre and do not share the same frequency response as the rest of the speakers in the series
Good video: https://youtu.be/GZrdsxrcpBw
Otherwise yes, ideally you want speakers to have a similar tonal profile
You can use a different speaker for ever channel. This is not recommended, but you can do it.
From what I’ve heard, it might be a good idea to match the front and centre speakers, while surround and rear matters less. I think the theory is that there shouldn’t be any noticeable change if sounds move around in the scene, but I don’t know if it’s actually a problem in reality.
deleted by creator
They do not have to match, just be compatible with the AVR as far as I’m aware. I just recently bought a 5.1.4 setup and could swap out any speaker if I wanted to
You can mix and match. Most AVRs let you set the crossover frequency for front and rear speakers separately, assuming you’re using the sub/lfe out. Just watch out for the usual speaker stuff like attaching high-impedance speakers to an amp that can’t handle it.
There’s some leeway on the rear speakers since they’re mostly used for environmental effects you can get away with a cheaper set. I’d be less inclined to skimp on the center speaker though. You generally want it to be a close match to the front L/R. Though a 4.1 setup could work if your AVR knows how to do it properly.