Robert Swain Gifford was born on a small island called Nonamesset, which adjoins Naushon in the chain of Elizabeth Islands off the coast of southeastern Massachusetts. When Gifford was two years old his family moved to Fairhaven where his father was employed as a boatman and fisherman.

Two famous New Bedford artists worked on the Fairhaven waterfront, Albert Van Beest and William Bradford. Gifford had the good fortune to become the friend and student of Van Beest, whom he took on sketching trips in his father's boat. Gifford followed Van Beest to New York and stayed with him there for a short time.

Gifford returned to New Bedford in the late 1850's, where he resided with his friend the sculptor, Walton Ricketson. With Ricketson he made trips to the Adirondack Mountains and to Grand Manan Island, where Gifford obtained subjects for paintings.

In 1865 Gifford moved to New York City and spent the winter painting in Samuel Coleman's studio in the Dodsworth Building. He left the city for the summer and returned to New Bedford and the area islands to make sketches for work to be done the next winter in his studio.

He also supported himself by doing book illustrations for three articles in Picturesque America, a forty-eight part serial edited by William Cullen Bryant and published by D. Appleton & Company in 1872.

In August 1870 Gifford traveled with his friend Louis Comfort Tiffany to Europe and Africa. This trip offered new subjects for Gifford's work; Egypt in particular was one of his favorite themes.

In 1867 he was elected an associate of the National Academy of design, and in 1878 he achieved the standing of academician. In 1873 he married Francis Eliot, the daughter of Thomas Dawes Eliot, a lawyer and congressman from New Bedford. The next year they went to Europe and eventually their travels led them to North Africa. They enjoyed this region very much and they both painted, recorded local customs, and collected local artifacts

In 1875 the Giffords returned to New York and joined the mainstream of American artists. Gifford won a metal of honor at the Centennial Exposition in Philadelphia and in 1876 and was one of the founders of the American Art Association. He became one of the first artists in the country to employ etching as a technique and he helped establish the New York Etching Club. He also became a member of the Royal Society of Painter-Etchers of London. In October 1877 Gifford accepted a position as a teacher of painting in the Cooper Union schools. He began a long and successful career with this institution, eventually becoming head of the Woman's Art School in 1896 and then director of all the art schools from 1903 until his death. He was appointed a member of the Advisory Committee on Fine Arts of the state of New York for the World's Columbian Exposition and was named a judge on the awards committee of the St. Louis World's Fair.

In 1886 Gifford built for his family a large home in Nonquitt in South Dartmouth. From then on the Giffords spent their summers at Nonquitt and their winters in New York. Gifford was a fine sailor who designed his own boats and enjoyed challenging his friends to match races on Buzzards Bay.

Gifford died at his home in New York after a brief illness at the age of sixty-five.