Don’t throw out the baby with the bath water. ~ German Proverb [16286]

Alternate version:
Don't throw the baby out with the bath water. ~ Common English usage

More German Proverbs