A reason that is a very close 2nd to the performance difference is the head gasket issue on the 3.8's.
A great running motor however, as I put 206k on my old '88 T-Bird, and I suspect the next owner put quite a bit more on it. But the head gasket issue is reason enough to steer clear of the V6 model.
I have an old buddy that made money buying mid 80's T-Birds and Cougars for $300 with blown head gaskets, then fixing them and re-selling for a good profit.
97 T-Bird LX 4.6 - 80k miles
94 Supra TT Auto - street/strip car
04 CVPI- Brenda's car - 76k miles
95 T-Bird LX 4.6 - fully optioned, owned 15 years, 220k miles
96 Cougar XR-7 4.6- Brenda's car, owned 11 years, 187k miles
88 T-Bird 3.8 - first T-Bird, owned 5 years, 206k miles