Onboarding the federated learning into multi-cluster environment(ACM)

[yanmxa]

Containerize the Collaborator and Aggregator Client

Training a neural network often involves third-party packages like torch, which significantly increase the image size. In many cases, the image size can exceed 1GB.

For example:

299M    /Users/yanmeng/anaconda3/envs/fl/lib/python3.11/site-packages/torch

To simplify the initial setup, I used the sklearn package to build a lightweight Logistic Regression model for the startup.

Other issues when containerize the application:

• openfl: unable to find user openfl: no matching entries in passwd file securefederatedai/openfl#1211
• flower discussion: https://friendly-flower.slack.com/archives/C01RM6LMKQA/p1734316514339399?thread_ts=1734238174.554119&cid=C01RM6LMKQA

Federated Learning

• Operand: Namespaced or Cluster
• Samples: https://github.com/yanmxa/federated-learning/tree/main/controller/config/samples
• Different with the proposal

  • Layer configuration is disabled

  • Clients -> Client: Use a single "client" configuration instead of defining individual details for each client. This ensures scalability and flexibility by avoiding repetitive definitions.

  • Remove client replicas:

    • Each client should not have replicas. Clients are split based on data distribution, not replication.
    • For a single data source, there should be exactly one client (model). Creating unnecessary replicas wastes training resources and can degrade accuracy during aggregation.

  • Use the of placement to schedule the client to specific clusters(key of the Label or ClusterClaim from the managed cluster)

  • Use the value of the Label or ClusterClaim from the managed cluster to locate the data metadata for the client

[yanmxa]

TODO:

1. addon cluster -> fl - client

2. controller -> manifestwork resplicasset -> cluster namespace

3. namespaced scope -> clusterset, placement

4. result -> model save(s3, kubernetes ...)

[yanmxa]

Jan 5, 2025

• The model with more parameters
• Data Privacy -> Case
• Detail the case before the demo

[yanmxa]

Feb 11, 2025

1. Multilayer

• Does the customer require this design?
  Review different Federated Learning frameworks—none of them currently support this kind of architecture.

• Gaps:

  • Federated Learning: Service B needs to function as both an aggregator and collaborator, but FL frameworks don’t support this dual functionality at the moment. This will require integrating these separate features, which may involve delving into the details and implementation of each FL framework.
  • OCM: The Placement API needs to be able to schedule workloads on managed clusters. Alternatively, we may need to treat the managed cluster as a hub cluster to support this functionality, similar to the global hub scenarios.

2. Leaf nodes setup - is node -> hub FL process?

Note: Currently, we lack a secure connection between the client and server. If we allow external nodes or devices to run the client or collaborator, we’ll need to enable a TLS connection to secure the communication.

3. Reference other FL to involve the multicluster environment: Standardized Interface for FL Frameworks to Support Multi-Cluster Environments #3

4. Lack of security connection

5. Local Model Metrics: Add Observability and Metrics for Federated Learning API #4

6. Add more framework: [GSoC 2025] Privacy-preserving and efficient AI model training across multi-clusters open-cluster-management-io/ocm#825

7. Give a interface to use the pretrained model: eeb13ac

[yanmxa]

Answer 7: Add pre-trained model

1. Configure the init model

cat <<EOF | oc apply -f -
apiVersion: federation-ai.open-cluster-management.io/v1alpha1
kind: FederatedLearning
metadata:
  name: federated-learning-sample
spec:
  framework: flower
  server:
    ...
    storage:
      type: PersistentVolumeClaim
      name: model-pvc
      path: /data/models/model_round_1_2025-02-25-14-27-46.pth
      size: 2Gi
  client:
    ...
EOF

2. Verification