Avoid using confrontational t-shirts for testing


Learn about the latest research behind the latest T-shirt patterns, the model spoofing an advanced body detection system

By param Raval
Compile Flin
Source: to ward data science

How can a person wearing a certain type of T-shirt make him invisible to body detection and human surveillance systems? Well, researchers have discovered and exploited the Achilles’ heel of deep neural networks – the framework behind some of the best object detectors (yolov2, faster, r-cnn, hrnetv2, etc.).

Earlier approach

In [1], the author managed to obtain 57% benchmark deception accuracy in practical use cases. However, this is not the first attempt to cheat the object detector. In [2], the author designed a method for their model to learn and generate patches that may cheat the detector. When the patch is worn on the cardboard (or any flat surface), it can avoid the human body detector successfully with the accuracy of 18%.

From [2]. Left: successful detection of people without patches. Correct: people with patches will be ignored.

Such “obfuscation” or “deception” neural networks are called physical countermeasures or real-world countermeasures. These attacks are initially based on complex changing pixel values, which make the network (based on its training data) mark the object as “unknown” or just ignore it.

[2] In this paper, the authors transform the images from the training data, apply the initial patch, and then input the resulting images into the detector. The target loss is used to change the pixel value in the patch to minimize the target score.

However, in addition to the low accuracy of 18%, this method is limited to rigid carriers such as cardboard, and the effect is not good when the captured frame is deformed or tilted. Moreover, when printed on T-shirts, the effect is certainly not good.

“A person’s movement may cause the wrinkles in his clothes to continue to change significantly (also known as deformation)” [1]. Therefore, the task of developing a general countermeasure patch becomes more difficult.

New methods

[1] In the new method, thin plate spline mapping is used to simulate cloth deformation. These deformations simulate the practical problems faced by the confrontational mode in the past. Taking care of different deformations will greatly improve the performance of the system because it will not be able to detect patterns in more frames.

Understanding the spline itself is enough to get a general idea of what they are going to do with this method.

Spline curve

You can see more formal mathematical definitions here

But for a simpler understanding, I think this article is the best.

In an intuitive sense, spline is helpful to draw any function smoothly, especially those that need interpolation. Splines help to model missing data: when modeling cloth deformation, we can see the deformation of patch shape in continuous frames. We can use an advanced form of polynomial spline called thin plate spline function (TPS).

Take a look at this article from Colombia, which explains TPS regression very well.

Then, these changes or displacements in the patch frame timeout are simply modeled as regression problems (because we only need to predict the TPS parameters of future frames).

Generate T-shirt pattern

The above mode is just an example of antagonism – a patch that goes against the purpose of the target detector. The author uses the expected over transformation (EOT) algorithm, which helps to generate such adversarial examples on a given transformation distribution.

Here, the transform distribution is made up of TPS transform, because we want to copy real-time wrinkle, slight twist and change of fabric contour.

In addition to the TPS transform, they also use the physical color transform and the conventional physical transform within the person’s bounding box. Therefore, this leads to the equation of modeling pixel values for disturbed images.

EOT formula based on all these complex formulas can finally calculate the attack loss, so as to achieve the purpose of deceiving the target detector.

So far, the simplest explanation of this process is for single target detector. The author also proposes a multi-target detector strategy, including applying min max optimization to the single target detector equation.


After training and testing our own data set, the results are impressive.

The use of TPS has also been greatly improved

What is the future

  • In an article by Northeastern University, Xue Lin, one of the authors of [1], clarified that their goal was not to make a T-shirt secretly undetected by the detector.

“The ultimate goal of our research is to design a secure deep learning system,… But the first step is to benchmark its vulnerabilities.” – Xue Lin

  • Of course, the authors recognize that there is a lot of room for improvement in their results, and mention that further research will be carried out to achieve this goal.


[1]: Xu, Kaidi, et al. Adversarial t-shirt! evading person detectors in a physical world (2019), arXiv preprint. arXiv-1910.11099

PDF: https://arxiv.org/pdf/1910.11099.pdf

[2]: Thys, Simen, Wiebe Van Ranst, and Toon Goedemé, Fooling automated surveillance cameras: adversarial patches to attack person detection (2019), Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.

PDF: https://arxiv.org/pdf/1904.08653.pdf

[3]: Athalye, Anish, and Ilya Sutskever, Synthesizing robust adversarial examples (2017), arXiv preprint arXiv:1707.07397.

PDF: https://arxiv.org/pdf/1707.07397.pdf

Link to the original text:https://towardsdatascience.com/avoiding-detection-with-adversarial-t-shirts-bb620df2f7e6

Welcome to panchuang AI blog:

Sklearn machine learning official Chinese document:

Welcome to pancreato blog Resource Hub: