Kemper Profiler MK 2

[Orvillain]

Null tests and LUFS measurement of the results have well-known shortcomings, but I don't think there's much evidence that it's a misleading methodology when used for properly done real-world guitar amp capture comparisons. Most criticisms I've seen are rather theoretical in nature and don't really point to any cases where a properly done null test leads to incorrect or misleading conclusions.

You could argue it's hard to do a null test correctly, and I agree with you, but that's taking issue with the tester, not the test.

I haven't seen anything that improves on using integrated LUFS to objectively measure the results of the test. You want to know the magnitude of the discrepancy and LUFS is a pretty good way to measure that in a way that is meaningful for human hearing.

This is silly. LUFS does not tell you if a model is perceptually close.

And you don't want a 1D measurement for comparing amplifier tonality - models, or otherwise.

You want as many dimensions as possible.

The problem is that a guitar amplifier is not a linear playback device. It is a nonlinear, input-dependent transformation system.

A null test fundamentally answers only one question:
“Are these two signals identical for this exact stimulus?”

That is a very narrow question.

An amp comparison needs to answer a completely different question:
“Do these two systems behave the same when a human plays a guitar through them?”

Those are not equivalent.

Two nonlinear systems can produce extremely small error residuals for one input, and behave radically differently for slightly different inputs. And guitar is nothing but slightly different inputs.

Pick attack
string choice
pickup inductance
note decay
intermodulation between strings
palm muting
dynamic impedance interaction with the speaker

A null test collapses all of that into a single fixed stimulus.
So you are not measuring the amplifier anymore.

You are measuring the amplifier's response to one specific waveform.

LUFS then makes the problem worse, not better. Integrated LUFS is essentially an energy-weighted loudness estimate over time. It is a scalar. But amplifier tone is not a scalar phenomenon.

The perceptual differences players react to live in:
dynamic compression behaviour
attack transient shaping
intermodulation products
harmonic distribution vs input level
bias shift / memory effects
speaker-interaction damping

None of those map cleanly to integrated loudness.

Two amps can have a −45 LUFS null residual and still feel completely different to play because the error is structured, not energetic. And the ear is far more sensitive to structure than magnitude.

A 40 dB-down transient distortion at note onset is perceptually obvious. A −40 dB broadband noise floor is not. LUFS only measures magnitude. Players respond to behaviour.

To evaluate an amp model you need multidimensional measurement:
level-dependent transfer curves, dynamic response, intermodulation behaviour, and time-variant response - essentially system identification, not waveform comparison.

 

[Orvillain]

Oh, and I will go as far to say that the fact someone like Leo Gibson can develop a popular Youtube channel based on his misinformation and propaganda, is a huge indictment against the guitar community.

 

[ian_dissonance]

I suggest you buy a license and find out.

The only time I give a crap about null tests are when Kemper sniffers say that accuracy is in the eye of the gear buyer, lol.

 

[Chocol8]

Null tests and LUFS measurement of the results have well-known shortcomings, but I don't think there's much evidence that it's a misleading methodology when used for properly done real-world guitar amp capture comparisons. Most criticisms I've seen are rather theoretical in nature and don't really point to any cases where a properly done null test leads to incorrect or misleading conclusions.

You could argue it's hard to do a null test correctly, and I agree with you, but that's taking issue with the tester, not the test.

I haven't seen anything that improves on using integrated LUFS to objectively measure the results of the test. You want to know the magnitude of the discrepancy and LUFS is a pretty good way to measure that in a way that is meaningful for human hearing.

You may have drank the kool aide, but you don't seem to understand what is being tested, how, and why the testing and results are so flawed.

It's not THAT hard to do a null test correctly. It's much more difficult to present the results and interpret them in a meaningful way. It's like taking an image with a CAT scan and then interpreting the results. One is done by a tech, who could learn to do it acceptably well with education on the machine and its use that measures in the range of a few hours. The other requires a radiologist who has a 4 year medical degree plus years of additional training in their specialty. Could you imagine posting raw CAT scan images and expecting the average forum user to be able to interpret them? That's what posting raw null test data would be like for most viewers. So you need to simplify it down to something people can understand, but you need to do so in a way that doesn't reduce all meaning. Leo is like a guy who took biology when he was 13 trying to interpret CAT scans and explain them to the public. He doesn't have any idea what he is doing and neither does anyone watching his videos.

LUFS is about as useful as a weighted average brightness for a CAT scan image. It means next to nothing. When looking at LUFS, can you tell which capture is better at replicating transients? Which one is better with high frequency harmonics? Which one has too much compression? Which one has bigger errors in the frequency responses in the lows or low mids? Does it tell you which errors impact tone more and which impact feel more? Nope! LUFS tells you nothing about how a guitarist will perceive the capture, nor how a listener will. No other single score is better because the idea of a single score is so fundamentally wrong here.

 

[Chocol8]

The only time I give a crap about null tests are when Kemper sniffers say that accuracy is in the eye of the gear buyer, lol.

You can use a null test to quickly disprove the "hypothesis" that a Kemper profile is accurate. A few palm mutes and a couple twists of the guitar volume should be enough to prove that any claims of perfection are wildly incorrect to anyone who isn't hearing impaired, but running them through a null test can show it on charts or graphs or whatever for those who are deaf, or simply refuse to hear the obvious. Learning much more takes a lot more knowledge and work.

 

[LP59]

To evaluate an amp model you need multidimensional measurement:

I can produce an even longer list than yours of theoretical shortcomings of integrated LUFS measurement of null test results.

What's missing though, is evidence that a simple integrated LUFS measurement of a null test discrepancy gives incorrect or misleading conclusions in a real-world comparison of guitar amp captures. The closest I've seen are contrived cases where the null test is intentionally performed incorrectly in order to introduce a LUFS difference that you can't hear.

Properly done null tests with integrated LUFS measurements that I've seen will pass the smell test of confirmation by listening, so all I'm saying is: I find it hard to conclude that there is a "need" for a more complex measurement.

 

[Chocol8]

Properly done null tests with integrated LUFS measurements that I've seen will pass the smell test of confirmation by listening, so all I'm saying is: I find it hard to conclude that there is a "need" for a more complex measurement.

Limited and properly CONTRIVED null tests that are setup to mimic preconceptions will give results that mimic your preconceptions. If the Leo's of the world get results that don't match what they hear, they change the test, thereby producing misleading results.

Here is a pertinent adage: It's easier to fool people than to convince them that they have been fooled.

 

[Orvillain]

I can produce an even longer list than yours of theoretical shortcomings of integrated LUFS measurement of null test results.

What's missing though, is evidence that a simple integrated LUFS measurement of a null test discrepancy gives incorrect or misleading conclusions in a real-world comparison of guitar amp captures. The closest I've seen are contrived cases where the null test is intentionally performed incorrectly in order to introduce a LUFS difference that you can't hear.

Properly done null tests with integrated LUFS measurements that I've seen will pass the smell test of confirmation by listening, so all I'm saying is: I find it hard to conclude that there is a "need" for a more complex measurement.

You are wrong.

 

[Tanax]

Ahh... Kemper threads never disappoint

 

[BenIfin]

Ok. Happy to admit when I've totally missed the point of something - consider myself educated :)

So ... is there some sort of way or method to objectively measure and compare the accuracy of one Capture format vs another Capture format -or- is this just a fools errand by Leo and others ?

 

[Orvillain]

Ok. Happy to admit when I've totally missed the point of something - consider myself educated :)

So ... is there some sort of way or method to objectively measure and compare the accuracy of one Capture format vs another Capture format -or- is this just a fools errand by Leo and others ?

You need a multi-variant approach. Not just something based on loudness.

 

[BenIfin]

You need a multi-variant approach. Not just something based on loudness.

Do you think such a process can be "developed" -or- are there just to many human and technical variables that make it pretty much impossible to objectively reconcile ?

 

[Orvillain]

Do you think such a process can be "developed" -or- are there just to many human and technical variables that make it pretty much impossible to objectively reconcile ?

 

[BenIfin]

View attachment 59609

I'll take that as a no.

 

[Chocol8]

Do you think such a process can be "developed" -or- are there just to many human and technical variables that make it pretty much impossible to objectively reconcile ?

I think it would have to be a series of tests and measurements, and then the results would take some knowledge to interpret.

 

[BenIfin]

I think it would have to be a series of tests and measurements, and then the results would take some knowledge to interpret.

..... and I guess that even if it were to be done, you still have the issue of (i) every player "feels" Captured Amps and Modeled Amps and EFX differently and (ii) there are almost and unlimited number of playback / monitoring systems of every style, shape and size.

 

[LP59]

I'll take that as a no.

I wouldn't be so fatalistic. Null test/integrated LUFS is a decent way to answer the question of whether the discrepancy is loud enough to worry about. Perfect is the enemy of good enough.

 

[Chocol8]

I wouldn't be so fatalistic. Null test/integrated LUFS is a decent way to answer the question of whether the discrepancy is loud enough to worry about. Perfect is the enemy of good enough.

No it's not. A null test could go a long ways but a LUFS number doesn't tell you what you seem to think it does.

 

[Chocol8]

..... and I guess that even if it were to be done, you still have the issue of (i) every player "feels" Captured Amps and Modeled Amps and EFX differently and (ii) there are almost and unlimited number of playback / monitoring systems of every style, shape and size.

I think the bigger issue is the range of signals we guitarists can feed into a capture. Various instruments and effects and various playing styles. A great example of this would be my own listening tests of captures years ago was focused on stuff (equipment and music) my friends and I tend to play and we completely missed issues that Tonex v1 and QC v1 had with high gain captures. Also why a country guy may have no issues with Kemper and think it gets really close but anyone who palm mutes might think it's nowhere close. Whatever you don't test you won't know, and that's a big problem with any scoring or ranking.

A hypothetical test signal file that covers everything would be extremely long, which would then burry important results amongst all the noise. That's why I think to do it well would really be best handled by a series of tests, but then you still have the issues of putting all the results together in an easy to consume format and assigning weightings etc. which would always be arbitrary and not right for many players.

The best and most relevant test involves your playing and your equipment, and no YouTuber can do that for you.

 

[Enchilada Jones]

This is sureal. Based on the excellent and detailed explanation of why null testing is a flawed approach to make the determination, we learn the only test that can provide proof is getting real guitarists to play through the test subject gear and deciding if it pleases them in all the multiple and nuanced ways a guitar amp does. And then you will need a large sample size of those guitarist to log their approval or disapproval.

Then, and only then will you have the data to prove to all the guitarists their Kemper sucks or doesn’t suck compared to other gear.
Only then will you have the data to prove Kempers fail the test and the defenders of the sound are idiots for saying they don’t buy into null testing because they like what they hear