According to the Google, most people can't perceive less than 10-15 ms of latency.
That is not exactly true (and no, let's not get into the Steve Vai <1ms debate again). Those latency perception tests are usually done as kinda "timing tests". Hence, you are to differentiate between a signal being dead on and whatever amount of ms late/early.
But it's a very, very different story once it comes to haptic vs. audible feedback.
And along with that there seems to be a very narrow range of "oh yeah, that feels tight" and "ok, that's somewhat distracted". And, to make matters worse, the start and end points of that very range seem to be quite different for different folks. In addition, apparently it's got not all that much to do with latency tolerance.
As an example: anything up to 7ms might feel tight for you (fwiw, let's suggest we're using headphones - outside of that world, there's yet some more variables kicking in). 9ms however might already feel distracted, with 7-9 being a sort of grey area. And you might then be fine with anything up to, say, 15ms (which is quite a lot under headphones, but these are all just arbitrary numbers).
Another person however might think that anything up to 9ms is fine, but anything over 10 is not even acceptable anymore.
For a while, I've done quite some tests regarding the matter, and once picking (or striking, as with a drum stick) comes in, there seem to be no truly reliable statements regarding exact numbers.
My suspicion would be that there's different ways to perceive sounds (or rather their initial impact/transient). As a guitar player, you could for example basically feel the moment your pick attacks the strings, make that your timing reference and pretty much ignore the resulting audio. That'd possibly a good way to ignore latency. Or you could do it the other way around, as in only listening to the audio output, pretty much trying to line it up against the rest of the music. That'd be a good way to compensate latency. If we could instantly switch between the two, that'd possibly be the best, but life ain't easy, plus there's a lot more variables, especially once we're outside of any controlled environment.
Anyhow, in a nutshell: Whether it matters or not is entirely up to you. But it's certainly the best to avoid additional latency whenever possible. One thing I'm doing to get there is bringing my own little mixer for IEM gigs, simply because I don't trust whatever possibly unknown FOH folks and their methods to tweak my guitar signal anymore. Even if the baseline latency of nowadays digital consoles is extremely small, I had it happening more than once that it was very noticeable, likely because some FOH tool decided to slap a lookahead limiter or whatever onto my signal (or anwhere else and the monitoring wasn't compensating, whatever...).