Putting 20" rims on.....
What you WILL notice is the "firmer" ride.
I don't doubt this in the slightest, but in my mind I'd prefer the slightly firmer ride, the car will feel tighter and more responsive, but it's not uncomfortable. If it was, those of us with 20's on would be complaining. My XKR has 20's (of course, and staggered wider rears with 295/30s) and the suspension setup on that car is much harder than the XF, you really feel the bumps etc but it's still not uncomfortable. I've never driven an XF on anything other than 20's though.
As John was saying, the bigger the rims the less the tyre profile so the overall rolling diameter of the (correct) tyre stays the same. Stick with genuine Jaguar wheels, you won't have a fitment problem (be cautious you don't get XK wheels as the offset can be different I think) nor any issues with insurance (came fitted, honest gov).
Phil's comment is just as valid, it's person preference.
If you do get another set of wheels, this is a perfect opportunity to keep the 17's and shod them with winter tyres and swap them over with the seasons. You'll extend the life of the summer tyres, and will have a more usable car in the winter. In snow, pretty much any rear-wheel drive car on summer tyres will be almost useless.
Talking of tyres, and this is very personal, just check the threads! But to generalise, Dunlops are known as Dunslips, just avoid. Pirelli seem OK when new but lose grip as they age. Avoid cheap and mid-price tyres, this is a big, heavy and relatively powerful car, no matter which engine. By popular consensus, the tyre of choice is a Michelin MP4S.