Edgewater is presumed to have been built by John R. Livingston of the wealthy Hudson River Valley Livingston family for his daughter, Margaret, who married Captain Lowndes Brown of Charleston, South Carolina. Livingston deeded the land to his daughter on November 28, 1824, but there is no evidence to support the local assumption that he also built the house. This was one of four marriages between the Livingstons of the Hudson Valley and the prominent Lowndes family of South Carolina.
The idyllic Livingston/Lowndes Brown life at Edgewater was shattered in 1852 with the coming of the New York to Albany railroad, with land taken by eminent domain on tracks located directly behind the house. Both John R. Livingston and his son-in-law, Lowndes Brown, died the same year the railroad was built. So outraged was the widow Margaret Livingston Brown that she sold the house and moved to London, vowing never to return to the United States. She is buried in Kensal Green Cemetery in London.
Edgewater was purchased from Mrs. Brown in 1852 by Robert Donaldson and his wife, Susan Gaston Donaldson, both native North Carolinians. The Donaldsons had earlier moved to New York, a move made possible by a substantial inheritance Mr. Donaldson received from his bachelor uncle, Samuel Donaldson, a highly regarded Scottish merchant banker in London.
The Donaldsons, who were patrons of architecture and art, added the charming Octagon library to the north side of Edgewater in 1854. The architect is documented to be Alexander Jackson Davis. Donaldson had worked with Davis previously on various projects at the University of North Carolina and on the State Capitol in Raleigh. Donaldson later introduced Davis to Andrew Jackson Downing, the writer and progenitor of romantic landscaping in the Hudson Valley. The Donaldsons were also friends of Asher B. Durand and Thomas Cole, Hudson River School artists who visited Edgewater. The arts certainly flourished at Edgewater under the Donaldsons, whose descendants retained the house until 1902, ending a half-century of Donaldson ownership.