The Imperial City is the main attraction of Hue and one of the few sights that you would see when you come to the city. Luckily, it's an impressive World Heritage Site that certainly justifies a visit.
The Imperial City is a large complex that takes me more than two hours to explore, wandering through the different sections. Much of it was destroyed during the 'American War' and many of the buildings you'll see have been restored or completely rebuilt. It also means there are large empty sections where no work has been done yet.
But what is here paints a vivid picture of the last dynasty of Vietnam (the one that shaped it in the form we find today). The main entrance gate is the most striking of the buildings, but I enjoy the detail of the Thai Hoa Palace, and the elegant red and gold patterns on the wall of the Can Thanh Palace. In hindsight, I do wish I had hired a guide or brought some reading material, because the explanations of what you're seeing aren't particularly detailed unfortunately.
This part of Vietnam is quite rich with cultural World Heritage Sites. As well as visiting the Hue Imperial City, you could use the city as a base to visit the My Son Sanctuary and to see Hoi An (although it makes for a good place to stay in itself). Within Hue itself, I would also recommend seeing some of the pagodas and tombs of the emperors that are spread along the Perfume River (also part of the WHS), which can be done with a cruise or by bicycle/taxi for a bit more flexibility.