It's not an easy place to get to but it's worth the effort. I organised a driver from Siem Reap and that was definitely the easiest way to do it. Although there's a lot of military around, there was no hassle getting into the site.
It's a really interesting site and there's a fair amount to see. A few hours is enough time to get around it all properly, though. I would definitely recommend making the side trip from Angkor to also see Preah Vihear.