I used Avignon as a base for my 3-day visit to the Provence. A good choice, it's a bustling little city with several things to see and do.
The core zone of the WHS occupies just a small area in the north of the walled city. That's where the popes and bishops built their monuments. And there also lies the Bridge of Avignon, unrelated to the papal story as far as my knowledge goes. In the early morning, I spent a few EUR walking on it - not really spectacular though the included audio guide is good.
Another monument that I went into is the Petit Palais, a pretty former bishop's residence at the main square 'Place du Palais'. It's now a museum displaying late medieval French and Italian art. I liked it, though as one of the few visitors I was annoyingly being followed by the museum guards.
I did not get to enter the interior of the cathedral, as it was closed any time that I tried. The Popal Palace next door obviously draws all the visitors. As other reviewers have said: most of it is empty, it looks like an empty castle. It was built during a relatively short period, and size seemed to have been more important than style or exterior decoration. Nice views are to be had from the gardens, down some stairs from the main route.