Removing shower doors?

I want to remove the shower doors in our bathroom. They have a gold trim and they are ugly and they make the small bathroom very difficult to clean. The doors have been there since we purchased the house 11 years ago. The shower wall is the original ceramic tiles and the walls will have holes where the doors are attached (3 holes on each end of the shower wall). I was going to fill in those holes with caulk. My husband said the doors are going to leave marks on the edge of tub as well. I think I can get all the markings off and fill in the holes with caulk. Would it be tacky to fill in the holes with caulk or do you think potential buyers will mind? Should we just leave the shower doors? I think taking the doors out will allow a potential buyer to vision putting their stuff in the bathroom. FYI...we are planning on making some other updates to the bathroom (i.e. tile, vanity top, paint, new light fixtures)