The original building, built in 1764 by Thompson Purdy, and one of the oldest structures in Manchester, now houses the formal living room, the Gathering Room and the Library, Taft, Lincoln and Hildene guest rooms. This portion of the inn still contains (in the basement) one of the earliest jail cells in Manchester (yes, the bars are still there as well as the marble slab on which the prisoners slept) and an all-marble floor. In the foyer of the old entrance is the smoke room where meats were hung and cured. It is believed that Ethan Allen hid in this "room" during the Revolutionary War. The house may have also served as a safe house for the Underground Railroad.

Edward Isham, a prominent Chicago attorney who grew up and went to school at Burr and Burton in Manchester and then to Harvard Law School, purchased the Purdy House in the latter 1800s and added on the rest of the house as you see it today. Isham, who was enthralled by the Revolutionary War and the Green Mountain Boys, named the house Ormsby Hill after Gideon Ormsby, a Revolutionary War hero from Manchester.

After the great Chicago fire, Mr. Isham and Robert Todd Lincoln (son of President Lincoln) formed the law firm known as Isham, Lincoln and Beal. They became the best of friends, and Lincoln was a frequent visitor to Ormsby Hill (along with their friend, President William Howard Taft). Lincoln loved Manchester so much he asked Isham to sell him some land. Isham replied, "You're my best friend and law partner; you're NOT going to be my next-door neighbor." It wasn't until Isham died that Robert Todd Lincoln was able to purchase the land behind Ormsby Hill now known as Hildene.

After the Isham family stopped living at Ormsby Hill, a foundation was created by the family to bring underprivileged boys from New York City to experience country life and attend the local private high school called Burr & Burton Seminary. This was the role of Ormsby Hill until the early 1980s, at which time the foundation abandoned the property.

In 1987 it was purchased and converted into a five-room bed and breakfast. In 1995, five additional rooms were added.

In June of 2012, Yoshio and Diane Endo purchased the Inn at Ormsby Hill, which now consists of ten luxurious bed chambers including one which is completely handicapped accessible.