I visited the Giant's Causeway near Christmas of 2001. Not only was there few people around during that time but what made the experience that much more exciting was that it felt that you were the only person there in a mystical experience between you, the ocean and old Gaelic memories.
It is a bit of a walk up and down from the visitor's centre, but the only time I would recommend it would be when the buses are not running full of tourists.