Can you feel the latency while playing? what kind of music do you play?
It absolutely depends on a lot of things. Monitoring situation, music style, exposure of guitar parts, personal mood, tired/awake, etc.
But that's not even exactly it.
Thing is, I can accept quite some latency, sometimes more, sometimes less. But the "tipping point range" always seems to be pretty small. With my old interface (Zoom UAC-2), I could very often (almost always) tell the difference between 32 and 64 samples, which, IIRC, was kinda like between 4.5 and 6.3 ms (or very similar, it didn't scale with the buffersize for whatever reasons). I even asked a mate once to switch a delay representing that difference on/off while I was just playing.
But, once I raised the baseline latency to, say 128 samples and then added around 2ms with a delay, I usually couldn't tell.
So, apparently, at least for myself, somewhere in the realm between, well, let's say 4-8ms there's something happening making me perceive things differently. I can't even put my fingers onto anything that would exactly feel different. It's not a drama. It's also not even that things necessarily feel worse. And it's also not timing related (I'm no Mr. "In The Pocket").
And that's still not all of it. There seems to be another "tipping point range" where it drifts from "ok, that's still kinda fine" to "yuck". Can't tell you any exact values, either, but it's possibly somewhere between 12 and 15 ms (also depending on all the things mentioned above, such as my mood and what not). And again, compared to the overall latency range I'm more or less happily willing to accept (anything between 0 and, well, maybe 12-15 ms), the range when things all of a sudden start to feel different is astonishingly small.
These things seem to be true for quite some people. I was holding DAW workshops at the local music conservatory some years ago, part of those workshops has been how to monitor yourself in a limited budget (home or project studio) situation, so software monitoring was one of the topics. And while we were at it, I did some similar (admittedly not exactly 100% blind and not exactly scientific) tests, the outcome of those tests has been quite comparable to my own experiences. People have varying ranges of latency acceptance (and it's most often got nothing to do with them being good/bad in terms of time) but there's certain tipping points existing within a pretty small range.
Fwiw: All of the above is only true for headphone usage, things get much more blurried once you monitor with studio monitors, wedges or whatever.
So, TL;DR: Ideally I don't want any plugin to ever push me beyond whatever tipping points. And in case we're dealing with 2ms, given that, at least regarding my personal experience, the range can be pretty small, it's not unlikely that this could happen at one point in time.