From Domain Adaptation to Federated Learning

Sammanfattning: Data-driven methods have been gaining increasing attention; however, along with the benefits they offer, they also present several challenges, particularly concerning data availability, accessibility, and heterogeneity, the three factors that have shaped the development of this thesis. Data availability is the primary consideration when employing data-driven methodologies. Suppose we consider a system for which we aim to develop a Machine Learning (ML) model. Gathering labeled samples, particularly in the context of real-world problem-solving, consistently poses challenges. While collecting raw data may be feasible in certain situations, the process of labeling them is often difficult, leading to a shortage of labeled data. However, historical (outdated) data or labeled data may occasionally be available from different yet related systems. A feasible approach would be to leverage data from different but related sources to assist in situations in which data is scarce. The challenge with this approach is that data collected from various sources may exhibit statistical differences even if they have the same features, i.e., data heterogeneity. Data heterogeneity impacts the performance of ML models. This issue arises because conventional machine learning algorithms assume what’s known as the IID (Independently and Identically Distributed) assumption; training and test data come from the same underlying distribution and are independent and identically sampled. The IID assumption may not hold when data comes from different sources and can result in a trained model performing less effectively when used in another system or context. In such situations, Domain Adaptation (DA) is a solution. DA enhances the performance of ML models by minimizing the distribution distance between samples originating from diverse resources. Several factors come into play within the DA context, each necessitating distinct DA methods. In this thesis, we conduct an investigation and propose DA methods while considering various factors, including the number of domains involved, the quantity of data available (both labeled and unlabeled) within these domains, the task at hand (classification or regression), and the nature of statistical heterogeneity among samples from different domains, such as covariate shift or concept shift. It is crucial to emphasize that DA techniques work by assuming that we access the data from different resources. Data may be owned by different data owners, and data owners are willing to share their data. This data accessibility enables us to adapt data and optimize models accordingly. However, privacy concerns become a significant issue when addressing real-world problems, for example, where the data owners are from industry sectors. These privacy considerations necessitate the development of privacy-preserving techniques, such as Federated Learning (FL). FL is a privacy-preserving machine learning technique that enables different data owners to collaborate without sharing raw data samples. Instead, they share their ML models or model updates. Through this collaborative process, a global machine learning model is constructed, which can generalize and perform well across all participating domains. This approach addresses privacy concerns by keeping individual data localized while benefiting from collective knowledge to improve the global model. Among the most widely accepted FL methods is Federated Averaging (FedAvg). In this method, all clients connect with a central server. The server then computes the global model by aggregating the local models from each client, typically by calculating their average. Similar to DA, FL encounters issues when data from different domains exhibit statistical differences, i.e., heterogeneity, that can negatively affect the performance of the global model. A specialized branch known as Heterogeneous FL has emerged to tackle this situation. This thesis, alongside DA, considers the heterogeneous FL problem. This thesis examines FL scenarios where all clients possess labeled data. We begin by conducting experimental investigations to illustrate the impact of various types of heterogeneity on the outcomes of FL. Afterward, we perform a theoretical analysis and establish an upper bound for the risk of the global model for each client. Accordingly, we see that minimizing heterogeneity between the clients minimizes this upper bound. Building upon this insight, we develop a method aimed at minimizing this heterogeneity to personalize the global model for the clients, thereby enhancing the performance of the federated system. This thesis focuses on two practical applications that highlight the relevant challenges: Predictive Maintenance and Network Security. In predictive maintenance, the focus is on fault identification using both DA and FL. Additionally, the thesis investigates predicting the state of health of electric bus batteries using DA. Regarding network security applications, the thesis addresses network traffic classification and intrusion detection, employing DA. ©Zahra Taghiyarrenani.