Split Learning

The core mechanism where a neural network is split by layer between participants. Typically, the client holds the initial layers (the input and feature extraction layers), processes its raw data, and sends only the output of the last client-side layer (called the smashed data or activations) to the server, which completes the forward pass through the remaining layers.

The primary advantage. Since only non-raw activations or gradients are exchanged, the client's original training data never leaves its device. This protects sensitive information (e.g., medical images, personal messages) and reduces the risk of data reconstruction attacks compared to federated learning, which shares model updates that can sometimes be inverted.

Clients only need to compute a subset of the model, significantly lowering their computational overhead and memory footprint. This makes Split Learning feasible for training large models (like deep CNNs or transformers) on resource-constrained devices such as mobile phones or IoT sensors, where running the full model locally would be impossible.

Training occurs in a strict, alternating sequence rather than in parallel:

1. Client Forward Pass: Client computes to the cut layer.
2. Server Forward Pass: Server completes the forward pass and computes the loss.
3. Server Backward Pass: Server computes gradients back to the cut layer.
4. Client Backward Pass: Server sends gradients to the client, which completes backpropagation. This creates a communication bottleneck but simplifies synchronization.

While it reduces client compute, Split Learning can incur high communication latency. Each training step requires at least two rounds of communication (activations forward, gradients backward). For large batch sizes or complex models, the volume of smashed data transferred can be significant, making network bandwidth a potential bottleneck compared to some federated learning approaches.

Ideal for scenarios with highly sensitive data and asymmetric compute resources. Common applications include:

• Healthcare: Training on distributed medical records or MRI scans across hospitals.
• Mobile Computing: On-device learning for next-word prediction or activity recognition.
• Edge IoT: Collaborative anomaly detection across industrial sensors.
• Finance: Fraud detection using transaction data from multiple banks.

Enables hospitals and research institutions to collaboratively train diagnostic models on patient data without exposing sensitive Protected Health Information (PHI). Key applications include:

• Training medical imaging models (e.g., for tumor detection) across multiple institutions.
• Developing predictive models for patient outcomes using distributed electronic health records.
• Complying with strict regulations like HIPAA and GDPR by keeping raw data localized.

Allows banks and financial institutions to build more robust fraud detection systems by pooling knowledge without sharing transactional data. This is critical because:

• Fraud patterns are often sparse and distributed; a single institution may not have enough examples.
• Sharing raw transaction logs between competitors is prohibited and risky.
• Models can learn from a broader set of anomalous patterns while keeping customer financial data private on the institution's premises.

A foundational technique for Federated Learning, particularly on mobile and IoT devices with limited resources. The model is split so the heavy computation stays on the server.

• The device computes on its local data and sends only the activations or gradients from a cut layer to the central server.
• This reduces bandwidth usage and preserves battery life compared to sending raw data or full model updates.
• Used for applications like next-word prediction on smartphones or activity recognition on wearables.

Facilitates vertical federation where different organizations (e.g., a manufacturer and a retailer) hold different features about the same entities. Split Learning allows them to build a joint model.

• Each party holds a portion of the model that processes their unique feature set.
• Only intermediate outputs are exchanged, never the raw input data or the complete model.
• This enables applications like improved supply chain forecasting or enhanced customer lifetime value models without merging proprietary datasets.

Serves as a core privacy-enhancing technology in trusted execution environments (TEEs) and multi-party computation setups. The split architecture adds an extra layer of security.

• Sensitive data can be processed within a secure enclave, with only non-sensitive model fragments exposed.
• It mitigates risks of model inversion or membership inference attacks by obfuscating the data flow.
• Used in scenarios requiring stringent data sovereignty, such as government analytics or confidential business intelligence.

The sequential nature of forward and backward passes between the client and server creates a significant communication bottleneck. Unlike federated learning, where clients compute full gradients locally, split learning requires constant, synchronous communication for each training step, leading to high latency. This is particularly challenging with a large number of clients or slow network connections, drastically increasing total training time.

The smashed data (intermediate activations) and gradient updates exchanged during training can be exploited for privacy reconstruction. Key attack vectors include:

• Model Inversion Attacks: Reconstructing raw input data from the smashed data.
• Membership Inference Attacks: Determining if a specific data sample was part of the training set.
• Property Inference Attacks: Inferring global properties of the client's dataset. Defenses like adding noise (differential privacy) or using secure multi-party computation (SMPC) add further computational cost.

Clients must perform the forward pass for their segment of the model and the backward pass for their local layers. This requires maintaining a partial model and sufficient compute resources (GPU/CPU memory, processing power). This burden can exclude resource-constrained devices (e.g., many IoT sensors or older mobile phones) from participating, creating a bias in the training cohort and limiting the system's applicability.

Managing the training process across heterogeneous clients with varying speeds and availability is complex. The central server must orchestrate the sequential training steps, handle client dropouts, and ensure model consistency. This requires sophisticated scheduling logic and fault-tolerant protocols, making the system architecture more complex than traditional centralized training or simpler federated averaging approaches.

The sequential dependency between client and server computations eliminates the possibility of parallelizing the forward/backward passes for a single data sample. While clients can be processed in parallel, each individual client's training step is a blocking operation. This limits throughput and makes inefficient use of server-side resources (e.g., powerful GPUs) that sit idle while waiting for client computations.

Choosing the optimal cut layer (split point) is a critical but non-trivial hyperparameter. It involves a trade-off:

• Early Split: Pushes more computation to the client, increasing their burden but potentially enhancing privacy.
• Late Split: Reduces client compute but sends more information-rich smashed data, increasing privacy risk and communication costs. The choice is often dataset- and model-specific, requiring careful empirical evaluation.

The primary privacy benefit of Split Learning is that raw training data never leaves the data owner's device. Instead, only the intermediate activations (or smashed data) and gradients from a cut layer are shared. This prevents direct exposure of sensitive input data, making it suitable for applications in healthcare (medical images) and finance (transaction records).

While raw data is protected, the shared activations can be vulnerable to model inversion or membership inference attacks. Adversaries may attempt to reconstruct training data or determine if a specific record was in the training set. Defenses include:

• Adding differential privacy noise to the activations.
• Using homomorphic encryption for secure forward/backward passes.
• Implementing gradient clipping to limit information leakage.

Split Learning operates under a semi-honest (honest-but-curious) adversary model, where participants follow the protocol but may try to learn extra information. It typically requires a trusted aggregator or parameter server to coordinate the learning process. For stronger security, protocols can be enhanced with secure multi-party computation (MPC) or trusted execution environments (TEEs) to protect the model logic and gradients.

Privacy comes at a cost. Split Learning requires significant communication rounds between clients and the server for each forward and backward pass, which can be slower than centralized training. The choice of the cut layer impacts both privacy and efficiency—a later cut exposes more abstract features (better privacy) but transfers larger tensors (higher bandwidth).

Unlike Federated Learning (FL), where clients train full local models and share only weight updates, Split Learning clients compute only a portion of the forward/backward pass. This makes Split Learning more suitable for clients with limited compute (e.g., IoT devices) but can be less private than FL with Secure Aggregation, as the server sees individual client activations.

Ensuring the correctness of computations in a split setup is challenging. Malicious participants might submit incorrect activations or gradients to poison the model. Techniques to mitigate this include:

• Proof-of-learning schemes for verification.
• Redundant computations across different nodes.
• Robust aggregation methods to filter out outliers, similar to defenses in federated learning.

A distributed ML paradigm where multiple clients train a shared model locally and send only model updates (e.g., gradients) to a central server for aggregation. Unlike Split Learning, the entire model is typically present on each client device. Key characteristics include:

• Horizontal Federated Learning: Clients share the same feature space but different data samples.
• Vertical Federated Learning: Clients share the same data samples but different feature spaces.
• Federated Averaging (FedAvg): The core algorithm for aggregating local model updates.

A form of encryption that allows computations to be performed directly on ciphertext, generating an encrypted result that, when decrypted, matches the result of operations performed on the plaintext. This is a cryptographic alternative to Split Learning for privacy, enabling secure model training and inference on encrypted data without exposing raw inputs or intermediate values.

A cryptographic protocol that enables multiple parties to jointly compute a function over their private inputs while keeping those inputs concealed from each other. MPC can be used to implement privacy-preserving ML, often providing stronger cryptographic guarantees than the architectural separation used in Split Learning, but with higher computational and communication overhead.

The core technical mechanism of Split Learning, where a neural network is divided into segments. The client-side model (or bottom model) processes raw data locally and sends the output (called smashed data or activations) to the server. The server-side model (or top model) completes the forward pass and computes the loss, sending gradients back. This partition is the cut layer.

A system for publicly sharing information about a dataset by describing patterns of groups within the dataset while withholding information about individuals. It is often used in conjunction with Split Learning or Federated Learning by adding calibrated noise to gradients or updates, providing a rigorous, mathematical guarantee of privacy against membership inference attacks.

Describes how data is distributed among participants in a Split Learning setup, analogous to federated learning.

• Horizontal Split (Sample-based): Different entities hold different data samples with the same feature set (e.g., different hospitals with similar patient records).
• Vertical Split (Feature-based): Different entities hold different features for the same set of data samples (e.g., a bank and a retailer holding financial and purchase data for the same customers). The split architecture must be adapted to this data alignment.