The Islands in the East River contain various charitable and correctional institutions belonging to the city, permission to visit which may be obtained from the Commissioners of Public Charities, 66 Third Ave...Blackwell's Island, 120 acres in extent, is a long narrow island, extending from about 50th St. to 86th St., and containing the Penitentiary, Female Lunatic Asylum, Workhouse, Alms Houses, and Charity Hospital. To be 'sent to the Island' is the New York euphemism for committal to the Penitentiary

Back in the day, one had to take a boat to the island...today... there is a tram:



Below are images of the Smallpox Hospital (surprisingly not mentioned above), followed by images of what remains of the Asylum, now called the Octagon.