By the way, doctor: Is not washing your face good for your skin?

I wash my face very little because I have heard soaps, no matter how mild, dry out the skin. What do you think – what should I wash my face with?