I recently did some A/B listening tests on amplifiers. After level matching to 1/10th dB and nulling any processing differences, we still had some subtle but real audible differences. They were not necessarily in either the frequency domain (we measured no differences) nor in the time domain. They could be described as differences in “clarity”, “definition”, and similar adjectives. What would be the appropriate metric to measure? I have some ideas, but I’m looking for yours.
@daleshirk- I am fresh off an a/b listening test between two preamps in this case- a $3000 vintage tube preamp (still being made but this one was made in the 90’s) and a $100 preamp made by Fosi. I have done several of these and use 5 questions:
**[u]B[/u] [u]C[/u] [u]About the same[/u]**
(1)Whose bass/ low frequency response did you prefer more? 30% 44% 26%
●
(2)Whose midrange/vocals did you prefer more ? 23% 45% 32%
●
(3)Whose high frequency response did you prefer more ? 14% 48% 38%
●
(4)Whose soundstage/imaging did you prefer more ? 22% 43% 35%
●
(5)Whose overall sound did you prefer more ? 26% 52% 22%
The percentages shown here are NOT the results of that test, but for another one. There is also a place where they can add a comment. No one knows what is being tested other than I may say we are testing two vintage receivers. For the preamp test no one even knew that it was preamps as the gear is hidden under a blanket. I check the levels at 1khz and 440hz to make sure the UUTs are putting out the same voltages into the speakers (for both channels). I tally up the totals at the end and let the numbers speaker for themselves. Yesterdays test had 18 “guinea pigs”. Hope this helps.
●
Hi @daleshirk, this is really the big question. I’ve always wondered about amp/cable/pre/powerconditioner makers than claim their product has some intangible quality that cannot be measured. But if it cannot be measured, then how do you test to make sure the widget was manufactured correctly? This is important because even Apple will have poor yields when a line is starting up. And if you cannot find the problem quickly through testing, you let defects escape to customers.
And that is the challenge. As you’ve stated, if you hear a problem but you can’t find it through testing, you are stuck. Static tones ignore the temporal components that might come into play. So, perhaps you want to look at amp dynamics? Or IMD? And both of these lead you into TIM.
Finally, it might be possible you are hearing something that isn’t there. A while back I wrote at ABX test app that would take a pristine digital of your choice and allow you introduce a type of distortion (linear or non-linear) and see how well you can pick the distortion out. The hope here was that this would allow you to define a baseline on a reference system, and then learn how a new system impacted your perception. So, you take the best system you have available in the room of your choice, and maybe you can hear a statistical difference between 16 and 14 bit quantization on your chosen source material. And then you move to a new system. Would that system let you hear the difference between 15 and 17 bits of quantization? If so, then it would be “better.” And if it the new system failed to allow you to hear the difference between a 16 and 13 bit quantization, then it would be “worse.”
But the ABX will let you find a baseline for your ears. And it might also suggest a route to pursue based on where your ears are most sensitive.
Were these (double) blind tests?
I repair and refurbish tube audio equipment, so I’ve heard a lot of different ones. Different preamps and amps do sound different even with the same program material and speakers. I had a customer who has a PhD in physics/electronics and I discussed this with him. He said that there are subtle harmonics and distortions created in each unit that give it its sound signature. These harmonics and distortions are generated by the unit’s design and components (caps, resistors, tubes, transformers, etc.), and these make each sound signature unique. I asked him if these differences are measurable and he said they are to a small degree, but, overall, the human brain is the thing that sums up the small distortions and harmonics and decides what sounds good. His experience (and mine) is that the super-low-distortion, totally accurate preamps and amps tend to sound sterile and become somewhat taxing over long periods while the not-so-sterile ones are a little softer and more musical. I’ve done blind tests with customers and the differences between units/tubes/components are very real. Even a comparison between a McIntosh MC-30 and a McIntosh MC-240 shows a distinct difference in sound even though they use the same tubes and similar circuits. I have considered building a test jig for tubes that would use the QA403 to test the performance of a single tube like a 12AX7 to see how an RCA compares to a Telefunken or other make.
@MarkT - if you were asking about what I posted, they were just where blind from the point of view that the participants switched between “1” & “2” pushbuttons on an older passive AV switchbox that had both composite and svideo inputs and outputs (I have measured it and it specs really good). It provides an immediate switch between the two devices, which are covered up and no one knows which is which. I am the one who hooked everything up, but lost track of what was what until the test is over. Only 1 person is in the room at a time doing the test, and no one is supposed to discuss what they thought until everyone is done- or peak under the blanket. I will have a video on it in about a month.
@ScottRichardson Interesting in that with the preamp test I just did, one of them was “cleaner” as far as having a better SNR/THD and noise floor and much lower harmonics. There was a statistical preference to the one that was not cleaner, except the cleaner one was preferred for vocals and midrange. When I did an a/b test (blind) between a pair of MacKit 30’s and a Bryston 2BLP (solid state) amp, with vintage Klipsch LaScalas, there was no statistical difference between the two amp speaker pairings using the same five questions above.
That is interesting. I used B&W 803 Nautilus speakers for my tests.
@ScottRichardson Certainly speakers could make a difference. With the recent test with preamps, the speakers used were vintage JBL 250Ti’s (that I wished were mine)- maybe the results would be different with the B&W’s or an electrostatic speaker. I have heard many folks talk about using a tube amp with their Klipsch speakers to “take the edge off”, but I have not found that to be the case with mine.
We didn’t have a setup capable of double blind, but we did a little single blind, and there were quite definite audible differences. This was the consensus of half a dozen well experienced audio professionals, speaker designers, A1s for NFL and MLB, etc., ears unlikely to be fooled.
From my experience, I would expect that. The problem is quantifying what makes those differences.
For testing Amps i would say you also need to look into Slew Rate and Dampingfaktor
for Comparson.
Special the Dampingfaktor could be a huge Faktor on different Speakers for the lower Frequencyrange i guess.
And Tube Amps with Outputtransformer is another Story by sure !
Never heard an Tubeamp that get me happy on lower End, may be because of that bad Dampingfaktors they have.
I had an dual EL34 Tubeamp thats i used only on the Midrange and tweeter Combo that wasn’t Bad @ al
But already on an 2 / 1/2 way Speaker with Audax HM170Z0 Midrangebass driver parallel down to 40Hz, the Tube Amp didn’t done well. No Punch @ al.
As i startet an comparssion AB Amp to an Hypex PWM Amp the first i hear was the dry and controled Bass on the Hypex, even with less Overall Power.
Hypex don’t Use Relais on the Outs so may be thats also an Benefit in Bass Controll and Dampingfactor.
I Use now the OEM Hypex Amps in my aktiv Setup with 250Watt/4Ohm (Bass) and 125 Watt/4Ohm (Midrange and Tweeter) and i am happy with it 
Now i work more on the Speaker and Room Side for Improvements there is a lot thats need to get Improved
And i bet there are more Differences to heare than on different DACS or Amps.
But Sure different DACS sound also Different, but i would say thats only on the last 5% to 10%
Amps around 20% to 30%
And different Speaker in different Rooms 40% to 60% Improvements possible.
The problem is that one person in the room knows which sample is which, everyone can guess from their behaviour (humans are super-tuned-in to non-verbal cues, and most give them out without realizing, which is why they lose at poker to a good player). Ears are very easily fooled in fact, as the McGurk effect testafies, expectation polutes perception.
Using a machine to randomize is a good way to go to eliminate expectation bias. But you have to match the signal levels precisely too since volume has a massive effect on perception (louder is always perceived as better, especially when you are actively struggling to hear the signal in detail). Same for the duration of each sample, they need to be the same or there will be a bias. You also have to worry about things like using a relay to switch between A and B - if the relay makes a different sound on switching A→B from when switching B→A, you may break the blind protocol.
Finally you need to run a lot of samples in a session to get good statistics, unless the differences are blatant. Its no fun doing proper double blind AB testing
In practice good amps that aren’t overloaded with flat FR’s will show very little difference compared to variations in rooms and speakers - headphones are always a good plan for human auditioning of anything but speakers themselves.
@Frunse Speaking of Damping Factor, I plan on posting what I did to improve the measurement with the QA40x here in a day or two (for those who are interested). The tube amps that I have measured typically have low damping factors, but I can’t say they lacked punch on the low end. @MarkT After using Matt’s ABX software, that would be the ideal method and I think more tested items would show results closer to tossing a coin. When I have done amp testing there can be a quick click sound from the relay (switching from either source to the other), but it it very fast and does not seem to affect things. My last a/b test I had a 9, 1 min tracks that went from 1 track to the next very quickly- namely to put a time limit on it so I could get everyone home at a reasonable time. This recent one had a CD with 16 tracks and they could listen to what they wanted to make up their minds, though I tried to put a 10min limit on it. A big factor that is not talked about a lot may be age- though a lot of the folks that attend my events are the kind that show you pictures of their audio gear instead of their grandkids, they are typically 60-70+, and I doubt their hearing is “pristine.”
I’ve read that people tend to prefer tube amps over SS amps because tube amps produce even harmonics vs odd harmonics from SS amps. Perhaps that would be a measurement to look at and see if that affects preference using blind tests.
Yeah i am 55 Years old and could say i can heare stil around 17 kHz, its easey to test
take an Sinusgenarator on an Amp and move slowly from 10kHz to 20kHz and stop moving when you cant hear anything
And Yes Harmonics are an big thing in listening Amps
also Noise to Signal Ratio and what kind of Cable you use from Amp to Speaker Combination
You could get higher or lower Highfrequencys with Different Cable because off Capacitance and different Basssound because of the inductance but on a passive Speaker with Crossover you has always an Big Coild in Front of an Bass Middriver so the Change is not that dramaticaly
But on aktiv Speaker its an different Story depending of how Long the Cable are i guess.
Not every Aktivspeaker need to have the Amp in the Wooden Speaker dEnclousure!
I dont realy like that at all, my Amps are all in an Aluminium Case on the Side off the Speakerenclousures! (care about Microphony effects, specialy on an Tube Amp i Bed)
And sure i never had the chance to listen an Tube Amp that costs more than 2000€ may be they
are better.
On the other Hand on what kind of Studio they do his recordings with Tube Amps?
May be only for Guitar Amp and may be on an Microphone Tube Amp
But thats an other Story to talk about ….LOL
The most studios have they Masters already digitaly saved so also using Analogue Recordings for an true full Analogue Chain isn’t easy to have anymore.
For Comparsion full digital with fully analogue Chains.
Best way, record an, let say acusticaly Guitar or Saxophone with an analogue Tapedeck and do youre Comparsions after copying the Track also to an CD or USB Stick in different Samplerates or so.
Or an big church organ, thats would be even better for deeper low end tones
For my personal taste, when i heared my first CD, around 1985 to 1990 not (sure enymore)
It was Vangelis China, i sold all my Analogue Records and Bought everything new on CD!!!
Even an cheep 500DM CD Player sounds much more Deep, Clean and Dynamic than any Record Player for 1000DM (DUAL or Thorens) @ this time, no cleaning needed @ al.
And for me i have never bought Audio Devices for more than 1500DM per piece, i was never rich
Normaly around 800 to 1000€ this days.
And now i like to build all i could by my self
Using Raspbarrypie 5 with Lyrion Music Server (Logitech Squeezeserver)
Building my own Speaker with efektive OEM Class D Amps and CamillaDSP for FIR Crossover two Audiointerfaces MOTU UL mk 5 with 16 Analogue Out Channels @all with 96kHz Samplefrequncy
waiting for an new MOTU release for an 24AO with that new MOTU 16A Design and latest DAC ships on it for 24 analogue Channels!!!
That would give me my Heaven and Flexibility for Speaker Settup on an full Surround aktiv Setup without using Souround AMP’s thats sounds ugly for my taste.
But thats me and my opinions for listening Music!
Every one has his own Audio Heaven and thats fine.
I own an Panasonic UDP9004 (payed 850€ when it was released) for Souround Movie and Music play
And befor i had different Cambridge Blueray Player around 1150€
So also have an old Onkyo TX-NR 5009 (new 3000€ @ release, got it for around 1500€ new some yeras later) Sorroundamp but thats sounds not good and i used only the AMP section for aktivate my Speaker @ that days 
Now it standing around without any use LOL
Before i had an Audio Analog Pre (Bellini) and stereo AMP (Corelli) with Passive Speakers (selfbuild with 2x AUDAX HM170Z0 and Focal inverse Kalotte Tweeter + Waveguide 2 / 1/2 Way Concept (50Liter Enclousures)) Was not that bad also.
@ScottRichardson I have seen tube amps/preamps with third order harmonics higher than the 2nd order even harmonics and visa versa, and many cases where the harmonics are mixed by channel as to what is predominant. @Frunse I have to take issue with your comment about speaker cables causing the differences you describe unless you are using very thin gauge wire over large distances- at least for audio. But agree with you that “everyone has his own audio heaven”
I have an “Monitor” Koaxial Speaker Kabel 5m 2x2,5mm² and because of that high Capacitance it has, the High’s get damped and you can realy see it on youre Measurement , easey > -3 to -6dB between 15kHz and 20kHz
If i use my Rferenz Speaker Kabel SUPRA Cable Rondo 4x 2,5mm², like on that Video (sorry is german language)
No dampening on the Tweeter are measured on the same Speaker Amp Combo
https://www.youtube.com/watch?v=oBQGyDN99bQ
Even on an cheep 2x 2,5mm² i didnt have this -3dB Problem
But on an aktiv Settup you have much more option using different Kabel for the 3 different Ways.
So keep youre hands away from Coaxial Speaker Kabel LOL
For Bass only may be a good idea. But never on a Tweeter.
If i listening Music with this Coaxial it sounds less Brilliant and less Dynamicaly @ Hights, even i could only hear to max 17kHz you can hear the Differences realy well.
This Cabel on that Video ar very good for that Price and not to Expensive.
I use now this Supra Cable Rondo, bought it already longt time ago and i am very Happy with it
Even usable as an Bi wiring Cable for Mid and High section
Sorry for my bad english …LOL
https://www.dienadel.de/supra-cables-rondo-4-x-2-5qmm-boxenkabel-meterware/a-353632221/
If i have time soon i will made some Measurings with Pictures for you, also with Rectangle Signal and Sleverates, like on that Video for Comparson.