The Google Arts & Culture App and the Rise of the “Coded Gaze” (The New Yorker)

The Google Arts & Culture App and the Rise of the “Coded Gaze”

A few days ago, I came across an old post on the Web site Bored Panda
called “10+ Times People Accidentally Found Their Doppelgängers in
Museums and Couldn’t Believe Their Eyes.” The post consisted of around
thirty photos of people posing with portraits whose subjects looked
eerily like them. There was a white-bearded man with reading glasses
hanging around his neck, the spitting image of the pike-carrying soldier
in Jan van Bijlert’s “Mars Vigilant.” (“Same nose!!!!” one commenter
wrote.) There was a young woman in a navy-blue shirt, a dead ringer for
the girl staring forlornly in William-Adolphe Bouguereau’s “The Broken
,” also wearing navy blue. For me, the funniest thing about the
photos was the backstory each one seemed to tell. I imagined van
Bijlert’s Dutch warrior almost cracking a smile as his middle-aged
doppelgänger twisted into position below while a friend, recruited as
the photographer, gamely offered direction. “Put your arm on your waist.
Tilt your chin up—higher, higher, yes!”

The goofiness of the images—in one, a man substitutes a rolled-up museum
guide for a Spanish nobleman’s riding glove—punctured the hushed
reverence that we tend to associate with art-consuming experiences, even
in the age of #MuseumSelfie Day.

The Bored Panda post reminded me, of all things, of a classic work of
Marxist art criticism: John Berger’s “Ways of Seeing,” which aired as a
TV series, in 1972, and was subsequently adapted into a book. For
Berger, the point of European oil paintings, in most cases, was not to
edify viewers but to flatter the wealthy patrons who commissioned them.
Painting was publicity, he contended, which made it akin to contemporary
advertising; the nineteenth-century nude with the come-hither glance
wasn’t so different from the twentieth-century pinup. Whatever the time
period, he wrote, “publicity turns consumption into a substitute for
democracy. The choice of what one eats (or wears or drives) takes the
place of significant political choice. Publicity helps to mask and
compensate for all that is undemocratic within society. And it also
masks what is happening in the rest of the world.” Berger, who hated
what he saw as the élitist tendency to view art museums as reliquaries
for holy objects, might have relished the silly subversiveness of the
art-doppelgänger trend.

If you have spent any time at all on Instagram or Facebook or Twitter in
the past couple of weeks, you have seen this trend going viral. In
December, Google introduced a feature to its Arts & Culture app that
allows you to take a selfie with your phone and use it to generate
results from the company’s image database for your own museum
doppelgänger. Last week, as more and more users discovered the feature,
Arts & Culture briefly became the most downloaded app in the iTunes
store. Social media was flooded with algorithmically generated
diptychs—smartphone selfie on the left, fine-art portrait on the right. In the many celebratory articles that followed, the project was
often framed as a way to democratize art by helping normal people to see
themselves in it. When Amit Sood, the president of Google Arts &
Culture, discussed the app with Bloomberg Technology, on Monday, he spoke of it in downright Bergerian terms. Growing up in Mumbai, India,
Sood said, he had seen art “as a posh experience, and not something that
was for me or for my people.” He added, “If you want to reach people
like me, or at least how I used to be before, you have to find a reason
for them to want to engage.”

When I fired up the app, I was initially less interested in finding my
doppelgänger than in what the result might say about the current state
of facial-recognition algorithms, which are famously bad at parsing
nonwhite features. My last foray into face-matching involved an app
called Fetch, which purported to tell users which breed of dog they most
resembled; many of my Asian friends and I were told we looked like Shih
Tzus. Joy Buolamwini, a black technologist and the founder of the
Algorithmic Justice League, which seeks to raise awareness of
algorithmic bias, calls this phenomenon the “coded gaze.” Just as the
male gaze sees the world on its own terms, as a place made for men’s
pleasure, the coded gaze sees everything according to the data sets on
which its creators trained it. Typically, those data sets skew white.
Not long ago, Buolamwini highlighted the story of a New Zealander of
Asian descent whose passport photo was rejected by government
authorities because a computer thought his eyes were closed.

As it happened, the Arts & Culture app did better than I’d expected. The
portrait I got, a painting by the twentieth-century Japanese artist
Ryusei Kishida , showed a handsome, rugged-looking man named Koya
Yoshio, with a tuft of brilliantly black hair. Not too shabby, I
thought. Still, others had plenty of complaints. For TechCrunch, the
writer Catherine Shu reported that nonwhite users were being confronted with “stereotypical tropes
that white artists often resorted to when depicting people of color:
slaves, servants or, in the case of many women, sexualized novelties.” In response, Google cast the blame not on its facial-analysis algorithm
but on art history. The app, the company told Siu, was “limited by the
images we have on our platform. Historical artworks often don’t reflect
the diversity of the world. We are working hard to bring more diverseartworks online.”

I’ll leave it to computer scientists to judge Google’s algorithm. But
the rise of the coded gaze has implications that go beyond the details
of any one app. As my social-media timelines filled with images of my friends with their doppelgängers, I was struck more by their
photos than the matches. They were often alone, poorly lit, looking
straight into the camera with a blank expression. In my own match image,
I have a double chin and look awkwardly at a point below the camera.
Next to my washed-out face, Ryusei’s painting looks shockingly vivid,
right down to the vein in his subject’s forehead. Unlike the
well-composed selfies and cheerful group shots that people usually
share, these images were not primarily intended for human consumption.
They were meant for the machine, useful only as a collection of data
points. The resulting images of photographed face next to painted face,
with a percentage score indicating how good the match was, seemed
vaguely diagnostic, as if the painting had materialized like the pattern
of bands in a DNA test. Judging by the flattered or insulted reactions
on social media, many people saw their matches as revealing something
about themselves.

In one way, the art selfie app might be seen as a fulfillment of
Berger’s effort to demystify the art of the past. As an alternative to
museums and other institutions that reinforce old hierarchies, Berger
offered the pinboard hanging on the wall of an office or living room,
where people stick images that appeal to them: paintings, postcards,
newspaper clippings, and other visual detritus. “On each board all the
images belong to the same language and all are more or less equal within
it, because they have been chosen in a highly personal way to match and
express the experience of the room’s inhabitant,” Berger writes.
“Logically, these boards should replace museums.” (As the critic Ben
Davis has noted,
today’s equivalent of the pinboard collage might be Tumblr or

Yet in Berger’s story this flattening represents the people prying away
power from “a cultural hierarchy of relic specialists.” Google Arts &
Culture is overseen by a new cadre of specialists: the programmers and
technology executives responsible for the coded gaze. Today the Google
Cultural Institute, which released the Arts & Culture app, boasts more
than forty-five thousand art works scanned in partnership with over
sixty museums. What does it mean that our cultural history, like
everything else, is increasingly under the watchful eye of a giant
corporation whose business model rests on data mining? One dystopian
possibility offered by critics in the wake of the Google selfie app was
that Google was using all of the millions of unflattering photos to
train its algorithms. Google has denied this. But the training goes both
ways. As Google scans and processes more of the world’s cultural
artifacts, it will be easier than ever to find ourselves in history, so
long as we offer ourselves up to the computer’s gaze.

via The New Yorker

January 26, 2018 at 03:23PM

No tags for this post.