Hi @daleshirk, this is really the big question. I’ve always wondered about amp/cable/pre/powerconditioner makers than claim their product has some intangible quality that cannot be measured. But if it cannot be measured, then how do you test to make sure the widget was manufactured correctly? This is important because even Apple will have poor yields when a line is starting up. And if you cannot find the problem quickly through testing, you let defects escape to customers.
And that is the challenge. As you’ve stated, if you hear a problem but you can’t find it through testing, you are stuck. Static tones ignore the temporal components that might come into play. So, perhaps you want to look at amp dynamics? Or IMD? And both of these lead you into TIM.
Finally, it might be possible you are hearing something that isn’t there. A while back I wrote at ABX test app that would take a pristine digital of your choice and allow you introduce a type of distortion (linear or non-linear) and see how well you can pick the distortion out. The hope here was that this would allow you to define a baseline on a reference system, and then learn how a new system impacted your perception. So, you take the best system you have available in the room of your choice, and maybe you can hear a statistical difference between 16 and 14 bit quantization on your chosen source material. And then you move to a new system. Would that system let you hear the difference between 15 and 17 bits of quantization? If so, then it would be “better.” And if it the new system failed to allow you to hear the difference between a 16 and 13 bit quantization, then it would be “worse.”
But the ABX will let you find a baseline for your ears. And it might also suggest a route to pursue based on where your ears are most sensitive.