The type and quality of the filter can have a pretty dramatic affect on the coffee extraction and the brew. On a simple level, I've noticed that I can taste some paper residue when I use a Chemex natural filter and don't rinse the filter.
There's also an interesting video James Hoffman did on the different V60 filter papers and how they affect flavors. Interesting stuff.