From e04f9338040ecd617ae2e234a318edb65aa199b5 Mon Sep 17 00:00:00 2001 From: Niklas Date: Fri, 4 Oct 2024 07:35:37 +0200 Subject: [PATCH] Docs/SK-1085 | Update introduction page (#721) --- docs/aggregators.rst | 1 - docs/apiclient.rst | 2 +- docs/architecture.rst | 2 +- docs/auth.rst | 2 +- docs/conf.py | 1 + docs/developer.rst | 2 +- docs/faq.rst | 2 +- docs/fedn.network.api.rst | 2 -- docs/fedn.network.clients.rst | 2 -- docs/fedn.network.combiner.aggregators.rst | 2 -- docs/fedn.network.combiner.rst | 2 -- docs/fedn.network.controller.rst | 2 -- docs/fedn.network.loadbalancer.rst | 2 -- docs/fedn.network.rst | 2 -- docs/fedn.network.statestore.rst | 2 -- docs/fedn.network.storage.models.rst | 2 -- docs/fedn.network.storage.rst | 2 -- docs/fedn.network.storage.s3.rst | 2 -- docs/fedn.network.storage.statestore.rst | 2 -- docs/fedn.rst | 2 -- docs/fedn.utils.helpers.plugins.rst | 2 -- docs/fedn.utils.helpers.rst | 2 -- docs/fedn.utils.plugins.rst | 2 -- docs/fedn.utils.rst | 2 -- docs/helpers.rst | 2 +- ...ine_learning_centralized_decentralized.svg | 1 + docs/index.rst | 2 +- docs/introduction.rst | 36 ++++++++++++------- docs/modules.rst | 2 +- docs/projects.rst | 2 +- docs/quickstart.rst | 1 - 31 files changed, 35 insertions(+), 57 deletions(-) create mode 100644 docs/img/machine_learning_centralized_decentralized.svg diff --git a/docs/aggregators.rst b/docs/aggregators.rst index bab17f045..dba34b6b4 100644 --- a/docs/aggregators.rst +++ b/docs/aggregators.rst @@ -117,5 +117,4 @@ This extension can then simply be called as such: .. meta:: :description lang=en: Aggregators are responsible for combining client model updates into a combiner-level global model. During a training session, the combiners will instantiate an Aggregator and use it to process the incoming model updates from clients. - :robots: index, follow :keywords: Federated Learning, Aggregators, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems diff --git a/docs/apiclient.rst b/docs/apiclient.rst index 4676bd375..e0416150c 100644 --- a/docs/apiclient.rst +++ b/docs/apiclient.rst @@ -139,4 +139,4 @@ For more information on how to use the APIClient, see the :py:mod:`fedn.network. :description lang=en: FEDn comes with an APIClient - a Python3 library that can be used to interact with FEDn programmatically. In this tutorial we show how to use the APIClient to initialize the server-side with the compute package and seed models, run and control training sessions and to retrieve models and metrics. :keywords: Federated Learning, APIClient, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems - :robots: index, follow + diff --git a/docs/architecture.rst b/docs/architecture.rst index 0060dbd05..4eabe3585 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -56,4 +56,4 @@ many different possible outcomes can be achieved. Good starting configurations a :description lang=en: Architecture overview - Constructing a federated model with FEDn amounts to a) specifying the details of the client-side training code and data integrations, and b) deploying the federated network. :keywords: Federated Learning, Architecture, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems - :robots: index, follow + diff --git a/docs/auth.rst b/docs/auth.rst index e4222318b..7e56d01fc 100644 --- a/docs/auth.rst +++ b/docs/auth.rst @@ -109,4 +109,4 @@ You can use `--token` flags in the FEDn CLI to set the access token. :description lang=en: Authentication and Authorization (RBAC) - FEDn supports Role-Based Access Control (RBAC) for controlling access to the FEDn API and gRPC endpoints. The RBAC system is based on JSON Web Tokens (JWT) and is implemented using the `jwt` package. :keywords: Federated Learning, Authentication and Authorization, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems - :robots: index, follow + diff --git a/docs/conf.py b/docs/conf.py index 3dc218da5..efcf2d2ea 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -42,6 +42,7 @@ html_theme_options = { "logo_only": True, + "display_version": False, } # Add any paths that contain custom static files (such as style sheets) here, diff --git a/docs/developer.rst b/docs/developer.rst index 36f418923..697649442 100644 --- a/docs/developer.rst +++ b/docs/developer.rst @@ -269,4 +269,4 @@ You can use `--token` flags in the FEDn CLI to set the access token. :description lang=en: During development on FEDn, and when working on own extentions including aggregators and helpers, it is useful to have a local development setup of the core FEDn server-side services (controller, combiner etc). We provide Dockerfiles and docker-compose template for an all-in-one local sandbox :keywords: Federated Learning, Developer guide, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems - :robots: index, follow \ No newline at end of file + \ No newline at end of file diff --git a/docs/faq.rst b/docs/faq.rst index a9c651eb8..7a27bbcd0 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -91,4 +91,4 @@ with the Scaleout team. :description lang=en: How do you approach the question of output privacy? We take security in (federated) machine learning seriously. Federated learning is a foundational technology that improves input privacy in machine learning by allowing datasets to stay local and private, and not copied to a server. :keywords: Federated Learning, FAQ, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems - :robots: index, follow + diff --git a/docs/fedn.network.api.rst b/docs/fedn.network.api.rst index 9f3db0391..5c0eb3e88 100644 --- a/docs/fedn.network.api.rst +++ b/docs/fedn.network.api.rst @@ -41,5 +41,3 @@ fedn.network.api.tests module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.clients.rst b/docs/fedn.network.clients.rst index fe98e345f..4be362177 100644 --- a/docs/fedn.network.clients.rst +++ b/docs/fedn.network.clients.rst @@ -49,5 +49,3 @@ fedn.network.clients.test\_client module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.combiner.aggregators.rst b/docs/fedn.network.combiner.aggregators.rst index 2e399d491..9bb760344 100644 --- a/docs/fedn.network.combiner.aggregators.rst +++ b/docs/fedn.network.combiner.aggregators.rst @@ -41,5 +41,3 @@ fedn.network.combiner.aggregators.fedopt module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.combiner.rst b/docs/fedn.network.combiner.rst index bcd563825..b3461224a 100644 --- a/docs/fedn.network.combiner.rst +++ b/docs/fedn.network.combiner.rst @@ -65,5 +65,3 @@ fedn.network.combiner.roundhandler module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.controller.rst b/docs/fedn.network.controller.rst index 84fdecf89..c3af9d103 100644 --- a/docs/fedn.network.controller.rst +++ b/docs/fedn.network.controller.rst @@ -25,5 +25,3 @@ fedn.network.controller.controlbase module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.loadbalancer.rst b/docs/fedn.network.loadbalancer.rst index db88bc8be..387cb0bc0 100644 --- a/docs/fedn.network.loadbalancer.rst +++ b/docs/fedn.network.loadbalancer.rst @@ -33,5 +33,3 @@ fedn.network.loadbalancer.loadbalancerbase module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.rst b/docs/fedn.network.rst index 43311ea12..ee7a9bd73 100644 --- a/docs/fedn.network.rst +++ b/docs/fedn.network.rst @@ -38,5 +38,3 @@ fedn.network.state module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.statestore.rst b/docs/fedn.network.statestore.rst index ce983a31d..8c1908294 100644 --- a/docs/fedn.network.statestore.rst +++ b/docs/fedn.network.statestore.rst @@ -25,5 +25,3 @@ fedn.network.statestore.statestorebase module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.storage.models.rst b/docs/fedn.network.storage.models.rst index e7450668f..09133d89e 100644 --- a/docs/fedn.network.storage.models.rst +++ b/docs/fedn.network.storage.models.rst @@ -33,5 +33,3 @@ fedn.network.storage.models.tempmodelstorage module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.storage.rst b/docs/fedn.network.storage.rst index 5fb804959..5bf9f3964 100644 --- a/docs/fedn.network.storage.rst +++ b/docs/fedn.network.storage.rst @@ -16,5 +16,3 @@ Subpackages fedn.network.storage.s3 fedn.network.storage.statestore -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.storage.s3.rst b/docs/fedn.network.storage.s3.rst index 3def5cf1d..7d1f7cd91 100644 --- a/docs/fedn.network.storage.s3.rst +++ b/docs/fedn.network.storage.s3.rst @@ -33,5 +33,3 @@ fedn.network.storage.s3.repository module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.network.storage.statestore.rst b/docs/fedn.network.storage.statestore.rst index 16690c022..895ad7ce4 100644 --- a/docs/fedn.network.storage.statestore.rst +++ b/docs/fedn.network.storage.statestore.rst @@ -25,5 +25,3 @@ fedn.network.storage.statestore.statestorebase module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.rst b/docs/fedn.rst index 2f6a9d2dd..c7ec895cb 100644 --- a/docs/fedn.rst +++ b/docs/fedn.rst @@ -15,5 +15,3 @@ Subpackages fedn.network fedn.utils -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.utils.helpers.plugins.rst b/docs/fedn.utils.helpers.plugins.rst index ff6abac29..bd52e0214 100644 --- a/docs/fedn.utils.helpers.plugins.rst +++ b/docs/fedn.utils.helpers.plugins.rst @@ -17,5 +17,3 @@ fedn.utils.helpers.plugins.numpyhelper module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.utils.helpers.rst b/docs/fedn.utils.helpers.rst index 46b540a71..33799044f 100644 --- a/docs/fedn.utils.helpers.rst +++ b/docs/fedn.utils.helpers.rst @@ -33,5 +33,3 @@ fedn.utils.helpers.helpers module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.utils.plugins.rst b/docs/fedn.utils.plugins.rst index 4629e0918..349d14c4d 100644 --- a/docs/fedn.utils.plugins.rst +++ b/docs/fedn.utils.plugins.rst @@ -41,5 +41,3 @@ fedn.utils.plugins.pytorchhelper module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/fedn.utils.rst b/docs/fedn.utils.rst index 09073b4ec..6f10aa3a1 100644 --- a/docs/fedn.utils.rst +++ b/docs/fedn.utils.rst @@ -41,5 +41,3 @@ fedn.utils.process module :undoc-members: :show-inheritance: -.. meta:: - :robots: index, follow \ No newline at end of file diff --git a/docs/helpers.rst b/docs/helpers.rst index 2a696aa14..4b33d2d5e 100644 --- a/docs/helpers.rst +++ b/docs/helpers.rst @@ -32,4 +32,4 @@ for further details. :description lang=en: Model marshalling - In federated learning, model updates need to be serialized and deserialized in order to be transferred between clients and server/combiner. There is also a need to write and load models to/from disk, for example to transiently store updates during training rounds. :keywords: Federated Learning, Model marshalling, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems - :robots: index, follow + diff --git a/docs/img/machine_learning_centralized_decentralized.svg b/docs/img/machine_learning_centralized_decentralized.svg new file mode 100644 index 000000000..084ed6263 --- /dev/null +++ b/docs/img/machine_learning_centralized_decentralized.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 5e1ab7ccf..23800fc86 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -34,4 +34,4 @@ Indices and tables :description lang=en: FEDn is a federated learning platform that is secure, scalable and easy-to-use. FEDn supports the full journey from early testing/exploration, through pilot projects, to real-world deployments and integration. :keywords: Federated Learning, Machine Learning, Federated Learning Framework, Federated Learning Platform, FEDn, Scaleout Systems - :robots: index, follow \ No newline at end of file + \ No newline at end of file diff --git a/docs/introduction.rst b/docs/introduction.rst index b57888285..16eeebca2 100644 --- a/docs/introduction.rst +++ b/docs/introduction.rst @@ -1,17 +1,30 @@ What is Federated Learning? =========================== -Federated Learning is a novel approach to address challenges related to data privacy, security, -and decentralized data distribution. In contrast to traditional machine learning setups where data is collected and stored centrally, -Federated Learning allows for collaborative model training while keeping data local with the data owner or device. This is particularly advantageous in -scenarios where data cannot be easily shared due to privacy regulations, network limitations, or ownership concerns. +Federated learning (FL) is a decentralized approach that tackles the issues of centralized machine learning by allowing models to be trained on data distributed across various locations without moving the data. Instead, FL moves the computation to where the data exists, enabling the creation of strong, globally-informed models while preserving data privacy and security. -At its core, Federated Learning orchestrates model training across distributed devices or servers, referred to as clients or participants. -These participants could be diverse endpoints such as mobile devices, IoT gateways, or remote servers. Rather than transmitting raw data to a central location, -each participant computes gradients locally based on its data. These gradients are then communicated to a server, often called the aggregator. -The server aggregates and combines the gradients from multiple participants to update a global model. -This iterative process allows the global model to improve without the need to share the raw data. +Traditional machine learning +----------------------------- +Traditionally, machine learning follows a centralized model. This approach involves collecting data from various sources into a single, centralized repository, typically in a cloud environment or a dedicated data center. Then, advanced algorithms are applied to this consolidated dataset, training predictive models that can be deployed to make decisions based on new incoming data. + +However, the centralized machine learning paradigm is facing growing challenges. The explosion of connected devices, sensors, and distributed data sources has led to an exponential increase in the volume and complexity of data being generated. At the same time, concerns around privacy, security, and regulatory compliance have made it increasingly difficult to freely move and consolidate data from different sources. + +The data needed to train effective machine learning models is often distributed across organizations, devices, or clients, making centralization challenging due to privacy risks and high transfer costs. + +.. image:: img/machine_learning_centralized_decentralized.svg + +How federated learning works +----------------------------- + +In federated learning, AI models are trained across multiple devices or servers (called client nodes) without needing to move the data off those devices. Here’s a simplified breakdown of how it works: + +1. **Starting the global model -** The process begins with a global model on a central server. This could be any type of machine learning model, like a neural network or decision tree. +2. **Sending the model to clients -** The server sends the global model’s parameters to a group of selected client nodes. Each client uses its own local dataset, which stays securely on the device. +3. **Local training -** Each client trains the model using its local data, adjusting the model’s parameters based on what it learns from the data. This training process is repeated for several rounds, rather than continuing until full accuracy is achieved. +4. **Combining the updates -** The updated models from each client are sent back to the central server, where they are combined. A common approach is called Federated Averaging, where the server takes a weighted average of the updates from each client. + +At last, the improved global model is sent back to the clients for further training. This cycle continues until the model reaches a satisfactory level of accuracy. The FEDn framework -------------------- @@ -71,11 +84,10 @@ For professionals / Enteprise, we offer `Dedicated support