Pitch shifting necessarily has latency. You need a history buffer big enough to contain at least one period at the lowest frequency you want to be able to shift. 19ms isn't bad and corresponds to about 53 Hz. Probably actually around 60 Hz as some of that latency is A/D and D/A conversion time and frame buffering.
If you want a "polyphonic shifter" you need a buffer that is large enough to hold an entire "period" of the waveform. This period, which we define as the maximum autocorrelation time, may be much greater than the period of the lowest note played. I.e., if you play an open E chord the period will usually be much less than 1/82. Some chord shapes result in very long periods, particularly those with dissonant intervals, e.g. major 7.
The shorter the buffer the less latency but the more tremolo effect will be introduced due to imperfect correlation at the splice point. The longer the buffer the less tremolo effect but the greater the latency.