The house used by Rancocas Nature Center was built in 1856 by Charles Longstreth Mather and his wife Naomi Passmore McIlvanin Mather. The east end was probably added around 1869-1870. In 1909 when Naomi died the house became the property of Charles Mather and his daughter Susan Mather. She married long-term friend Frank Hilyard after her mother's death. In front of the house was a beautiful Boxwood Hedge for which they were offered $1000 by Longwood Gardens of Kennett Square, PA but they declined. The property was later purchased by William G. McIlvain. In 1939 a portion of the 179 acres including buildings were sold to Seymour W. and Lucretia Ferris for $4,000. In 1966 Mrs. Ferris, widow, sold those acres to the State of New Jersey to be included in the Rancocas State Park.
In 1768 the Rogers Family built a house and grist mill down by Rancocas Creek which at that time was a transportation route to the Delaware River and into Philadelphia. The Mathers lived in that house while building the one that eventually became Rancocas Nature Center.