To illustrate (and yes, for yourself, if you like) how sensitive we are when it comes to latency: Double a signal in your DAW of choice, pan both hard L/R, add some samples of delay (using track parameters, a 100% wet delay /w 0 feedback or whatever) just to one of the signals. That's how incredibly great at detecting the smallest latencies the combination of our ears and brains is.
It's likely also a matter of evolution. When the leaves of the jungle behind you were rustling back in the days, you simply had to know whether it was slightly left or right because that might've easily become the deciding factor for that approaching sable-toothed fellow to have a nice meal or to stay hungry.
For more scientific explanations, one could always look up the Haas effect.
Note: No, this isn't all too directly related to amp modeler and DAW latencies - it just goes to show that at least in general, our senses are absolute suited to deal with extremely small latencies.