Born in New York City, Ben Goldstein came of age at the beginning of the Great Depression and early in his life formed lasting friendships with older artists and writers of the political left. During the 1930s he worked on Fiorello LaGuardia's mayoral campaign and served as labor adviser on the garment industry to FDR's short-lived but influential National Recovery Administration. He later served as an impartial chairman, adjudicating disputes between manufacturers and needle trade unions such as the International Ladies' Garment Workers' Union (ILGWU).

After World War II, Ben continued his association with the garment industry, but as a manufacturer rather than a labor advocate. He bought a small loft factory providing women's sportswear to the wholesale and retail trade. The business prospered and he began to buy art in earnest. During the Depression he had already purchased, often as a form of financial aid, prints and drawings from such political artists as Robert Minor and Hugo Gellert, and from various artists employed by the Works Progress Administration. He also combed the galleries in lower and midtown Manhattan for images of working people, American industry, politics, and other subjects that few other collectors found desirable. What interested him was "art that related to life" and he objected to abstract art, which he claimed was "not about anything."

He displayed his collection on the walls of his factory and attracted a steady stream of curators, scholars, and collectors drawn by the novelty of an art collection devoted to social and political themes and by the unlikely sight of garment workers toiling beneath their pictorial counterparts. In 1976, at the age of sixty-eight, Ben sold the loft and closed his factory, devoting his energies to the creation of groundbreaking exhibitions on political art and African American history, among other topics, that he continued to do well into his eighties. His collection, unlike most assemblages of artworks, embodies a message. It expresses the enduring dignity of all people across the economic and national spectrum.