There are two kinds of people who get really riled, really fast online: photographers and audiophiles.
OK, there are lots of people who get really riled online. But photography and audio recording are both black arts, equal parts science and craft, where no expert can ever really be wrong.
So when I reviewed Neil Young’s high-resolution PonoPlayer last week, I went to a lot of trouble to design a fair test.
I was not surprised that my effort to run a fair test was in vain. In emails, in the comments, and on online discussion boards, audiophiles tore into my setup, the conclusion, and even each other.
In this follow-up, I’ll try to address some of the key objections.
“Your test was invalid because you didn’t use audiophile speakers/you didn’t use $900 professionalheadphones/your 15 listener volunteers aren’t trained professionals.”
That’s correct. I don’t review audiophile equipment; I’m not an audiophile.
I reviewed the PonoPlayer only because it’s marketed as a player for everyone who loves music, not as audiophile gear.
Neil Young himself wrote, “We are not an audiophile company.” And the company’s former CEO, Phil Baker, wrote this to me:
“We find that headphones in the $150-$300 range work great with Pono. Examples are the Sennheiser Momentum and equivalents from AKG and Beyerdynamic. Some also like the Beats Studio 2. We’ve used Audeze headphones that cost >$1,000, and they sound fantastic, but they are not needed to appreciate Pono sound. Earbuds from Etymotic, Cardas, Shure, and similar products from $99 work great.”
“Your test was invalid because you used a Radio Shack switch box and RCA cables.”
I had two audio engineers approve my setup. I even asked them about the A/B switch box. One engineer said, “It’s totally fine.” The other wrote, “The switch is just a copper cross-connect. There is literally nothing that can go wrong with it. As I like to say when advising people against buying expensive ‘high-end’ audio cables: copper is copper.”
As for the cables, a third expert wrote, “As an audio/video engineer of 30 years, I can assure you the RCA cable introduces no distortion, assuming the cable is correctly built. Even the cheapie Chinese cables these days are perfectly equal to the most expensive Monster oxygen-free $100 gold-plated scam cables.”
Did I use the right kind of cable? The same expert wrote, “RCA makes a much better connection than 3.5-mm plug, since it has many times the surface area and clamping pressure. The highest AAC audio bandwidth of 192kHz is easily carried by the 10MHz capacity of RCA cables.”
After the fact, lots of other experts strenuously disagreed with those experts.
Some audiophiles suggested eliminating the A/B box entirely. “Just plug and unplug the same headphone cord, back and forth, from the Pono to the phone.”
Yes, that would remove the A/B box as a red herring. But it would also ruin the beauty of the test. Every time you plug or unplug headphones, the player stops playing. So it would be impossible for my volunteer listeners to flip seamlessly between the sources, comparing the sound midnote. There would be silence and a gap and no possibility of synchronizing the two players.
I suspect the critics would find plenty wrong with that arrangement.
"Your test was invalid because even the tiniest volume difference can make music sound better."
I was aware of this. In fact, in my own rehearsing for the test, I discovered that even though I had bought the same three songs — same albums, same recordings — from the Pono store and the iTunes store, they played back at different volumes!
In the end, I therefore had to make volume adjustments on each player before each song played. “Saturday in the Park” required two volume clicks down on the Pono; “There’s a World” required one click up on the iPhone; and so on.
I arrived at these settings by ear, mind you — I made the volume levels identical manually. If there’s a more automated way to do it, I’m open to suggestion.
"Your test was invalid because it wasn’t double blind. You knew which player was which, so your data is tainted.”
The argument here is that I somehow communicated to my 15 volunteers which player I “wanted” to sound better in each test through my “tone of voice, facial expression, or body language” (as one commenter put it).
Say what? I had no horse in this race. It was a test — that was the point. I didn’t want the PonoPlayer to win or to lose; I just wanted to know if it sounded better to real humans. Why would I want it to lose (or win)? What would I get out of stacking the deck?
Anyway. To answer the question: The entire test was filmed. You can watch excerpts of the test here. The listeners and I did not make eye contact during each playback. We weren’t even facing each other.
After each playback, the volunteer would say “I liked A better” or whatever. I wrote down the result and then said the same thing every time: “OK, good. Here’s the next one.”
"Your test was invalid because people these days have come to prefer the compressed sound of phone audio.”
Yes, that occurred to me, too. Maybe, after years of listening to iPod and phone music, that’s the sound consumers have come to know and like.
Still, the question here is, “Which sound do you prefer?”
This critique comes dangerously close to saying, “Your panelists preferred the wrong thing.” I don’t think anyone is qualified to tell anybody else that their preference is incorrect.
A humble proposition
As I write this, there are 449 responses to my column in the comments. Not a single one proposes a test that would be 100 percent valid, and especially not to the other 448 commenters.
I remain convinced that my test is valid for the way most people — not audiophiles — listen to music most of the time. If you have to go to so much trouble to hear the difference between the PonoPlayer and a phone, then I’d argue it’s not worth spending thousands of dollars for the new hardware and for repurchasing all your music. At least for an everyday music fan.
The audiophile version of the test, though, with ultra-high-end equipment and audio wonks as listeners, could very well have different results. Tell you what: If my golden-eared readers can come to a consensus on exactly which gear to use and how the test should be conducted — a foolproof test protocol that won’t trigger further complaints — I’ll be happy to run it. (I’m not getting my hopes up about that consensus thing, though.)
In the meantime, there’s one more piece of feedback I think I should share. It was an email from Neil Young after the review was posted.
“Thanks for the review,” he wrote. “I really learned a lot from it and my communications with you. Good luck … Neil.”
That, my friends, is high-resolution class.