I think part of the problem is the need/desire to keep costs in check. Look at the three games you mentioned for their innovation - P2K, TZ and HH, and think about the bill of materials for all those games.
I am not saying that something innovative needs such a high BoM, and the guys developing a game with an LCD in the playfield that tracks the ball comes to mind, but cost is most likely one of the big issues.
Having said that, the lack of full-scale competition for Stern for the last fifteen years is more likely the biggest factor IMO. I think they have lifted their game recently in response to most JJP and the possibility that some of the boutique start ups might actually get a game out. But as you say - it is the same with a different twist (most different artwork when its all said and done).
Even when B/W was completely dominant in the 1990's, I firmly believe that the reason B/W came up with so much that was innovative was that they had several independent design groups, who in effect competed with each other to come up with the next game to get green-lighted for the line. That internal competition could very well have been more intense than competing against actual competitor companies.
