Xidi and Hongcun are precious gems, getting there is fun! We had our own rented minivan, and left Huangshan very early. We passed the countryside covered in mist and bamboo forests, it looked like Switzerland with the river, the black-and-white ancient villages! I wrote this in the Huangshan section, but it's worth repeating. Try visiting some of the other villages enroute to Hongcun and Xidi, they are quite similar in style, minus the commercial atmosphere.

Having said that, I know that our group blends in with the local Chinese backpackers, so we never have problems with permits. I'm not sure if you don't look Chinese if there's a way around this.