Byl otcem pěti dětí, Elizabeth Fenwick Savage (* 1846), Alexander Duncan Savage (* 1848), Thomas Rutherford Savage (* 1852), William Rutherford Savage (* 1854), Jesse Duncan Savage (* 1858).
Savage was born in Middletown, Connecticut to ship owner Josiah Savage and Mary Roberts. The family consisted of wealthy Congregationalists and he went to Yale College and Yale Medical School, receiving an MD in 1833.