Attacks on Federated Learning: Protecting Privacy
Attacks on Model Updates
In federated learning, each participant submits model updates instead of raw training data during the training process. However, recent research has demonstrated that it’s often possible to extract raw training data from model updates. One early example came from the work of Hitaj et al., who showed that it was possible to train a second AI model to reconstruct training data based on model updates.
Figure 1: Data extracted from model updates by the attack developed by Hitaj et al. [1]. The top row contains original training data; the bottom row contains data extracted from model updates.
Figure 2: Data extracted from model updates by the attack developed by Zhu et al. [2]. Each row corresponds to a different training dataset and AI model. Each column shows data extracted from model updates during training; columns with higher values for "Iters" represent data extracted later in the training process.
How to fix it
Attacks on model updates suggest that federated learning alone is not a complete solution for protecting privacy during the training process. Many defenses against such attacks focus on protecting the model updates during training, so that the organisation that aggregates the model updates does not have access to individual updates.
Attacks on Trained Models
The second major class of attacks target the trained AI model after training has finished. The model is the output of the training process, and often consists of model parameters that control the model’s predictions. This class of attacks attempts to reconstruct the training data from the model’s parameters, without any of the additional information available during the training process.
Figure 3: Training data extracted from a trained AI model using the attack developed by Haim et al. [3]. The top portion of the figure (a) shows extracted data; the bottom portion (b) shows corresponding images from the original training data.
Figure 4: Training data extracted from a diffusion model using the attack developed by Carlini et al. [4]. Diffusion models are designed for generating images; one popular example is OpenAI’s DALL-E.
Figure 5: Training data extracted from a large language model (LLM) using the attack developed by Carlini et al. [5]. This example is from GPT-2, the predecessor of ChatGPT.
How to fix it
Attacks on trained models show that trained models are vulnerable, even when the training process is completely protected. Defenses against such attacks focus on controlling the information content of the trained model itself, to prevent it from revealing too much about the training data.
Conclusion
In this post, we have discussed the two major classes of attacks on federated learning: attacks on model updates and attacks on trained models. We have also discussed the importance of protecting privacy during the training process and the need for defenses against such attacks. In the next post, we will introduce one of the key issues for federated learning: distribution of the data among the participating entities.
FAQs
Q: What are the two major classes of attacks on federated learning?
A: The two major classes of attacks on federated learning are attacks on model updates and attacks on trained models.
Q: How do attacks on model updates work?
A: Attacks on model updates involve extracting raw training data from the model updates shared during the training process.
Q: How do attacks on trained models work?
A: Attacks on trained models involve reconstructing the training data from the trained model’s parameters.
Q: What are some common defenses against attacks on model updates?
A: Some common defenses against attacks on model updates include protecting the model updates during training and using cryptography to prevent unauthorized access to the model updates.
Q: What are some common defenses against attacks on trained models?
A: Some common defenses against attacks on trained models include controlling the information content of the trained model itself and using differential privacy to prevent the model from revealing too much about the training data.
