While I agree with the accuracy of your statement, I disagree with the idea that it's worthless.
My car is a toy, a hobby, and is the lowest priority on spending money behind my home/groceries/wife/daughter/pets/entertainment. That said, I tend to start small and build up as money permits. So, a dual exhaust would be one of the first things I buy if I were starting with a car without, as it is relatively inexpensive, easy enough to do in the garage, and is a good "supporting mod" to everything else that comes later.
I am of the opinion that building a good supporting platform first is more important than starting with the big stuff like engine/transmission. So, Exhaust, suspension, wheels/tires/brakes, rear end, driveshaft, transmission/torque converter, then engine is the order I go in. I want to do jobs once, and do the entire job if I can, before moving onto the next thing. So, if I'm going to do an engine (which I'm doing now), I'm going to go as far with it as I can afford, in as short a time as possible so that I don't get tired of not driving my car.
Everyone has different priorities, but putting a free-flowing exhaust on the car can do nothing but help, and helps build a better baseline for when you do more stuff in the future in my opinion.
Its the same as any hobby .. making something your own and the experiences you create with it.
Im exactly the opposite end when it comes to "supporting" mods, I believe they are mostly to enhance after the fact instead of a pre-requisite. Take my SS Chevy for example .. almost 400 horses from the factory - and now I am considering the Borla cat back system. Mostly for the sound aspect to hear that 6.0 Liter motor leave Fords in the dust ( in reality, its been mostly WRX and STI's around here ) .. but its not necessary for the performance - its already there and can only be improved on. In the case of my Tbird, id rather put the money that could be spent on an exhaust, etc .. towards a better engine to have a real starting point.