README

Paint Your Face Away addresses the issues of privacy and surveillance concerning the data economy of faces online in association with the technology of facial recognition.

The project develops and makes accessible to the public a browser-based application of an online face painter that creates portrait pictures that can resist the technology and operation of face scraping that powers some facial recognition systems.

How the painter works

You are invited to use this tool to paint over the image of your face until the site’s face detection system can no longer detect it in the image. The finished portraits could potentially provide the following effects, depending on how the face is painted:

1. Face detection obfuscation (though the detection proofing tested only with face-api.js - SSD Mobilenet V1 used and not necessarily working with other models).
2. Quasi-anonymisation (if the face and ears are thoroughly painted out).
3. Potential misclassification of facial recognition systems and/or object classification systems (an experimental feature).

Also, please see Disclaimer.

The painter allows you to decide whether your facial biometric identity is opaquely painted out by layering the paint, or left partially visible. Even if your facial features are not completely painted out, the process could still enmesh your biometric data with the useless noise of paint so that it could potentially resist the automated data harvesting of faces.

Why is the tool designed in this way?

The painter is both informed and inspired by the technology and concept of adversarial attacks, which are used to trick Machine Learning models. In particular, the painter explores various data manipulation methods to hack algorithmic categorisation of AI systems [1].

The painter uses some methods of the adversarial attacks and explores the juxtaposing potential of paint as something irrelevant for the biometric data economy of faces.

Furthermore, the finished portraits can be circulated to help spread the word about the issues of face scraping and the availability of this painter. The more biometrically ruined images we provide, the more contaminated the online harvesting ground of faces could become. Though when you share your portraits, be sure to paint them thoroughly so that your portrait is not identifiable.

[1 - Transmediale Symposium 2020, Neural Network Cultures.]

Web-scraping of faces

If you have uploaded pictures of your face on some websites or social media platforms, chances are your faces could have been “scraped”, or algorithmically gathered. They may then have been used to train facial recognition systems to identify you, possibly without your consent. This kind of practice has already been used socially to build datasets and real facial recognition systems.

A face scraper could be built by combining face detection, object recognition (to look for the categories of images that may contain faces), and the hashtag sorting of relevant categories (such as #selfie, #portrait, #face) in order to make the process more efficient for gathering a large number of faces.

Paint Your Face Away started in mid-2019 before the above practice of face scraping became known to the public fairly widely. Its initiation was rather speculative and in response to the technological possibility of automatically collecting a vast number of faces to build a dataset to train facial recognition systems.

Disclaimer

This is primarily an art project, and the testing capacity of the functionalities are limited by means of research resources and scientific expertise compared to laboratory-run scientific research.

This painting tool is a beta version, and not optimised for mobile devices.

The painting tool does not always guarantee perfect anonymisation.

The face detection proofing does not necessarily work on other systems.

Some cases of misclassification against facial recognition and object classification systems have been observed with some painted faces (examples (1) | (2) | (3) ), but these are experimental features and the effects are not laboratory proven.

Credits

Project by Shinji Toya. The original version of the online application of Paint Your Face Away was commissioned by and co-produced with Fotomuseum Winterthur in 2020 for the exhibition SITUATIONS/Strike.
(This is version 1.2 beta.)


Production support:
Alejandro Daniel Ball from Agorama (Full stack development support)
Ashwin D’Cruz (Machine Learning expertise and advice)
Tobias Stenberg (Coding contribution)


Research advice and inspiration by:
Agorama
Simon Crowe
Alexander Fefegha from Comuzi
Marco De MuTiis


Special thanks:
Jonathan Brantschen | Compiler (Tanya Boyarkina & Oscar Cass-Darweish) | Inês Costa | Rebecca Edwards | Max Dovey | Claudel Goy | Trisan Deschamps-Lange | Sam Mercer | Mona Schubert | Jon Uriarte | Nimrod Vardi | Anna Viani | Nadine Wietlisbach | Fotomuseum Winterthur


The tools used include:
Javascript: P5.js, ml5.js , face-api.js (TensorFlow.js) / Max MSP Jitter.