Models

This attack involves injecting malicious functionality into a model's architecture and/or weights. It exploits vulnerabilities in the serialization formats of models of popular machine learning platforms, enabling arbitrary code execution [9].

Examples include an attacker adding layers to the model architecture to steal input data or encoding malware into the model weights, potentially with self-learning capabilities.

To evade detection, the poisoned model must maintain performance quality comparable to that of a clean model.

This is a type of poisoning attack where the attacker embeds hidden logic into the model, such as misclassifying certain objects or failing to detect specific object classes which activates only when a particular trigger is present in the input data. Without the trigger, the model functions as intended by its developers.

The backdoor can be introduced into the model via weights - poisoning the training dataset by injecting malicious samples and then retraining the model on the entire dataset [11], or fine-tuning the model exclusively on the poisoned samples [12]. In the latter case, the trigger synthesis algorithm analyzes neuron responses within the targeted model itself.

Another way to deliver a backdoor is to change the model architecture. A subnet (payload) is added to the model, trained to detect the trigger in input data, along with a block that determines the behavior (output) of the model when the trigger is present [10, 13]. This approach is less resource-intensive because the model doesn't need to be trained on poisoned data; it also doesn't degrade the model's performance on clean data and offers better transferability across models.

It's important to note that triggers can include both synthetic objects, such as adversarial patches (textures in images that are not interpretable by humans), and real-world objects, such as a traffic light or sign in a road scene. This significantly expands the practical applications of this type of attack type in real-world scenarios.

In this attack, the attacker uses legitimate access to the targeted model's outputs to extract its knowledge and transfer it into their own model[1, 2]. As a result, the attacker obtains a functional replica without physically stealing the model, bypassing traditional information security measures.

One method used in this attack is "knowledge distillation" [3]. This method is widely used for model compression. A compact "student" model is trained using the outputs of a larger "teacher" model, which serves as an "oracle" by providing data labeling and confidence scores in this labeling (so-called "soft labeling"). This supervised learning process guides the student model to replicate the teacher model's behavior. In an extraction attack, the attacked model acts as the teacher, and the attacker's model is the student. One example involved the extraction of knowledge from a model with 193 million parameters trained on a proprietary dataset of 1 billion images [2].

A straightforward way to prevent these attacks is to limit the number of queries allowed to the model. However, attackers can overcome this by leveraging advanced techniques like active learning [5, 6, 8] or semi-supervised learning [7], which require fewer labelled examples to train models effectively.

A successful model extraction attack, conducted in a partial (grey-box) or zero (black-box) knowledge mode, enables attackers to perform more sophisticated. With full (white-box) knowledge of the model, they can carry out adversarial attacks, data poisoning, or explore for other vulnerabilities in the model more effectively.