I will say right away: this place isn't my favorite. It looks great on pictures, it's a must-do when in Sri Lanka, it has all the things of an epic day out, I was so looking forward to finally reaching it, or let's say reaching the top, but alas I was disappointed in the end.
The climb is almost a tourist trap. There is a really tiny staircase for most of the way. There are always some tourists trying to get past you, there are always some tourists stopping on the way taking pictures of the not-very-special monkeys and thus EVERYONE needs to wait. This may be more of a rant towards tourists, so how about the rock fortress? The frescoes half-way up are great - I love it. Most people actually go here by mistake judging from their response when they have to turn back towards the stairs. I actually find this the best part of the climb. When you reach the top and look at the all the frustrated faces trying to line up going back down I knew I shouldn't have done it either. What you get is a pretty good view, I wouldn't call it fantastic in anyway, and some much needed wind in the Sri Lankan heat, but from what's left of the ruined fortress is absolutely a waste of time. Instead I should have spent more time in the museum but I figured I should first head up to the top and see it on the way down but who knew it takes 2 hours... and I didn't even spend any time in the gardens yet.
Disregarding my frustration, the fortress' history is something worthy of world heritage status. The planning for the location and work put into it are impressive, and it shouldn't be missed on your tour, I just wish I had lower expectations for the climb.