I don't think there should be any "side to side" movement. It is a very tight set up (on purpose I believe). Needs a tight sleeve - I had to keep that one black as the yellows I prefer are a bit fatter and would not work.

You can also loosen off the plunger mechanism and angle it, but I'd get that sleeve tight first as that might be it (given the plunger was probably installed 25 years ago and not touched since, less likely to have gone out of alignment).