Merge branch 'main' into enhance_json_serialization_for_numpy

docs/quickstart.rst

@@ -101,6 +101,17 @@ Note on branches:
 
 * The 2.1, 2.2, 2.3, 2.4, 2.5, etc. branches are the branches for each major release and there are tags based on these with a third digit for minor patches
 
+Install NVFlare from source
+----------------------------
+
+Navigate to the NVFlare repository and use pip install with development mode (can be useful to access latest nightly features or test custom builds for example):
+
+.. code-block:: shell
+
+  $ git clone https://github.com/NVIDIA/NVFlare.git
+  $ cd NVFlare
+  $ pip install -e .
+
 
 .. _containerized_deployment:
 
@@ -197,11 +208,30 @@ Production mode is secure with TLS certificates - depending the choice the deplo
 
 Using non-HA, secure, local mode (all clients and server running on the same host), production mode is very similar to POC mode except it is secure.
 
-Which mode should I choose for running NVFLARE?
-
-  - For a quick research run, use the FL Simulator
-  - For simulating real cases within the same machine, use POC or production (local, non-HA, secure) mode. POC has convenient ``nvflare poc`` commands for ease of use.
-  - For all other cases, use production mode.
+Which mode should I choose for running NVFLARE? (Note: the same jobs can be run in any of the modes, and the same project.yml deployment options can be run in both POC mode and production.)
+
+.. list-table:: NVIDIA FLARE Modes
+   :header-rows: 1
+
+   * - **Mode**
+     - **Documentation**
+     - **Description**
+   * - Simulator
+     - :ref:`fl_simulator`
+     - | The FL Simulator is a light weight simulation where the job run is automated on a 
+       | single system. Useful for quickly running a job or experimenting with research 
+       | or FL algorithms.
+   * - POC
+     - :ref:`poc_command`
+     - | POC mode establishes and connects distinct server and client "systems" which can 
+       | then be orchestrated using the FLARE Console all from a single machine. Users can 
+       | also experiment with various deployment options (project.yml), which can be used 
+       | in production modes.
+   * - Production
+     - :ref:`provisioned_setup`
+     - | Real world production mode involves a distributed deployment with generated startup 
+       | kits from the provisioning process. Provides provisioning tool, dashboard, and 
+       | various deployment options.
 
 .. _starting_fl_simulator:
 
@@ -376,4 +406,18 @@ To start the server and client systems without an admin console:
 
   nvflare poc start -ex [email protected]
 
+We can use the :ref:`job_cli` to easily submit a job to the POC system. (Note: We can run the same jobs we ran with the simulator in POC mode. If using the :ref:`fed_job_api`, simply export the job configuration with ``job.export_job()``.)
+
+.. code-block::
+
+  nvflare job submit -j NVFlare/examples/hello-world/hello-numpy-sag/jobs/hello-numpy-sag
+
+.. code-block::
+
+  nvflare poc stop
+
+.. code-block::
+
+  nvflare poc clean
+
 For more details, see :ref:`poc_command`.

docs/user_guide/nvflare_cli/poc_command.rst

@@ -7,6 +7,8 @@ Proof Of Concept (POC) Command
 
 The POC command allows users to try out the features of NVFlare in a proof of concept deployment on a single machine.
 
+Different processes represent the server, clients, and the admin console, making it a useful tool in preparation for a distributed deployment.
+
 Syntax and Usage
 =================
 
@@ -292,7 +294,7 @@ will start ALL clients (site-1, site-2) and server as well as FLARE Console (aka
 
 .. note::
 
-    If you prefer to have the FLARE Console on a different terminal, you can start everything else with: ``nvflare poc start -ex admin``.
+    If you prefer to have the FLARE Console on a different terminal, you can start everything else with: ``nvflare poc start -ex admin@nvidia.com``.
 
 Start the server only
 ----------------------
@@ -356,6 +358,59 @@ If there is no GPU, then there will be no assignments. If there are GPUs, they w
 
            nvidia-smi --list-gpus
 
+Operating the System and Submitting a Job
+==========================================
+After preparing the poc workspace and starting the server, clients, and console (optional), we have several options to operate the whole system.
+
+First, link the desired job directory to the admin's transfer directory:
+
+.. code-block:: none
+
+    nvflare poc prepare-jobs-dir -j NVFlare/examples
+
+FLARE Console
+--------------
+After starting the FLARE console with:
+
+.. code-block:: none
+
+    nvflare poc start -p [email protected]
+
+Login and submit the job:
+
+.. code-block:: none
+
+    submit_job hello-world/hello-numpy-sag/jobs/hello-numpy-sag
+
+Refer to :ref:`operating_nvflare` for more details.
+
+FLARE API
+---------
+To programmatically operate the system and submit a job, use the :ref:`flare_api`:
+
+.. code-block:: python
+
+    import os
+    from nvflare.fuel.flare_api.flare_api import new_secure_session
+
+    poc_workspace = "/tmp/nvflare/poc"
+    poc_prepared = os.path.join(poc_workspace, "example_project/prod_00")
+    admin_dir = os.path.join(poc_prepared, "[email protected]")
+    sess = new_secure_session("[email protected]", startup_kit_location=admin_dir)
+    job_id = sess.submit_job("hello-world/hello-numpy-sag/jobs/hello-numpy-sag")
+
+    print(f"Job is running with ID {job_id}")
+
+
+Job CLI
+-------
+The :ref:`job_cli` also provides a convenient command to submit a job:
+
+.. code-block:: none
+
+    nvflare job submit -j NVFlare/examples/hello-world/hello-numpy-sag/jobs/hello-numpy-sag
+
+
 Stop Package(s)
 ===============
 
@@ -381,3 +436,9 @@ There is a command to clean up the POC workspace added in version 2.2 that will
 .. code-block::
 
     nvflare poc clean
+
+Learn More
+===========
+
+To learn more about the different options of the POC command in more detail, see the 
+:github_nvflare_link:`Setup NVFLARE in POC Mode Tutorial <examples/tutorials/setup_poc.ipynb>`.

examples/advanced/finance-end-to-end/README.md

@@ -225,6 +225,8 @@ Please refer to the scripts:
 - [graph_construct.py](./nvflare/graph_construct.py) and [graph_construct_job.py](./nvflare/graph_construct_job.py) for graph construction
 - [gnn_train_encode.py](./nvflare/gnn_train_encode.py) and [gnn_train_encode_job.py](./nvflare/gnn_train_encode_job.py) for GNN training and encoding
 
+The resulting GNN encodings will be merged with the normalized data for enhancing the feature.
+
 ## Step 3: Federated Training of XGBoost
 Now we have enriched / encoded features, the last step is to run federated XGBoost over them. 
 Below is the xgboost job code
@@ -285,7 +287,7 @@ it anywhere in a real deployment.
 
 Assuming you have already downloaded the credit card dataset and the creditcard.csv file is located in the current directory:
 
-* prepare data
+* Prepare data
 ```
 python ./utils/prepare_data.py -i ./creditcard.csv -o /tmp/nvflare/xgb/credit_card
 ```
@@ -302,21 +304,21 @@ python ./utils/prepare_data.py -i ./creditcard.csv -o /tmp/nvflare/xgb/credit_ca
 > * 'XITXUS33_Bank_10' 
 > Total 10 banks
 
-* enrich data
+* Enrich data
 ```
 cd nvflare
 python enrich_job.py -c 'ZNZZAU3M_Bank_8' 'SHSHKHH1_Bank_2' 'FBSFCHZH_Bank_6' 'YMNYFRPP_Bank_5' 'WPUWDEFF_Bank_4' 'YXRXGB22_Bank_3' 'XITXUS33_Bank_10' 'YSYCESMM_Bank_7' 'ZHSZUS33_Bank_1' 'HCBHSGSG_Bank_9' -p enrich.py  -a "-i /tmp/nvflare/xgb/credit_card/ -o /tmp/nvflare/xgb/credit_card/"
 cd ..
 ```
 
-* pre-process data
+* Pre-process data
 ```
 cd nvflare
 python pre_process_job.py -c 'YSYCESMM_Bank_7' 'FBSFCHZH_Bank_6' 'YXRXGB22_Bank_3' 'XITXUS33_Bank_10' 'HCBHSGSG_Bank_9' 'YMNYFRPP_Bank_5' 'ZHSZUS33_Bank_1' 'ZNZZAU3M_Bank_8' 'SHSHKHH1_Bank_2' 'WPUWDEFF_Bank_4' -p pre_process.py -a "-i /tmp/nvflare/xgb/credit_card  -o /tmp/nvflare/xgb/credit_card/"
 cd ..
 ```
 
-* construct graph
+* Construct graph
 ```
 cd nvflare
 python graph_construct_job.py -c 'YSYCESMM_Bank_7' 'FBSFCHZH_Bank_6' 'YXRXGB22_Bank_3' 'XITXUS33_Bank_10' 'HCBHSGSG_Bank_9' 'YMNYFRPP_Bank_5' 'ZHSZUS33_Bank_1' 'ZNZZAU3M_Bank_8' 'SHSHKHH1_Bank_2' 'WPUWDEFF_Bank_4' -p graph_construct.py -a "-i /tmp/nvflare/xgb/credit_card  -o /tmp/nvflare/xgb/credit_card/"
@@ -330,6 +332,10 @@ python gnn_train_encode_job.py -c 'YSYCESMM_Bank_7' 'FBSFCHZH_Bank_6' 'YXRXGB22_
 cd ..
 ```
 
+* Add GNN embeddings to the normalized data
+```
+python3 utils/merge_feat.py
+```
 
 * XGBoost Job 
 
@@ -343,7 +349,7 @@ cd ..
 Below is the output of last round of training (starting round = 0) 
 ```
 ...
-[9]	eval-auc:0.67596	train-auc:0.70582
+[9]	eval-auc:0.69383	train-auc:0.71165
 ```
 For GNN embeddings, we run the following command
 ```
@@ -354,7 +360,6 @@ cd ..
 Below is the output of last round of training (starting round = 0) 
 ```
 ...
-[9]	eval-auc:0.53788	train-auc:0.61659
+[9]	eval-auc:0.72318	train-auc:0.72241
 ```
-For this example, the normalized data performs better than the GNN embeddings. This is expected as the GNN embeddings are produced with randomly generated transactional information, which adds noise to the data.
-
+As shown, GNN embeddings help to promote the model performance by providing extra features beyond the hand-crafted ones.

examples/advanced/finance-end-to-end/utils/merge_feat.py

@@ -0,0 +1,95 @@
+# Copyright (c) 2024, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import argparse
+import os
+
+import pandas as pd
+
+files = ["train", "test"]
+
+bic_to_bank = {
+    "ZHSZUS33": "Bank_1",
+    "SHSHKHH1": "Bank_2",
+    "YXRXGB22": "Bank_3",
+    "WPUWDEFF": "Bank_4",
+    "YMNYFRPP": "Bank_5",
+    "FBSFCHZH": "Bank_6",
+    "YSYCESMM": "Bank_7",
+    "ZNZZAU3M": "Bank_8",
+    "HCBHSGSG": "Bank_9",
+    "XITXUS33": "Bank_10",
+}
+
+original_columns = [
+    "UETR",
+    "Timestamp",
+    "Amount",
+    "trans_volume",
+    "total_amount",
+    "average_amount",
+    "hist_trans_volume",
+    "hist_total_amount",
+    "hist_average_amount",
+    "x2_y1",
+    "x3_y2",
+]
+
+
+def main():
+    args = define_parser()
+    root_path = args.input_dir
+    original_feat_postfix = "_normalized.csv"
+    embed_feat_postfix = "_embedding.csv"
+    out_feat_postfix = "_combined.csv"
+
+    for bic in bic_to_bank.keys():
+        print("Processing BIC: ", bic)
+        for file in files:
+            original_feat_file = os.path.join(root_path, bic + "_" + bic_to_bank[bic], file + original_feat_postfix)
+            embed_feat_file = os.path.join(root_path, bic + "_" + bic_to_bank[bic], file + embed_feat_postfix)
+            out_feat_file = os.path.join(root_path, bic + "_" + bic_to_bank[bic], file + out_feat_postfix)
+
+            # Load the original and embedding features
+            original_feat = pd.read_csv(original_feat_file)
+            embed_feat = pd.read_csv(embed_feat_file)
+
+            # Select the columns of the original features
+            original_feat = original_feat[original_columns]
+
+            # Combine the features, matching the rows by "UETR"
+            out_feat = pd.merge(original_feat, embed_feat, on="UETR")
+
+            # Save the combined features
+            out_feat.to_csv(out_feat_file, index=False)
+
+
+def define_parser():
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument(
+        "-i",
+        "--input_dir",
+        type=str,
+        nargs="?",
+        default="/tmp/nvflare/xgb/credit_card",
+        help="output directory, default to '/tmp/nvflare/xgb/credit_card'",
+    )
+
+    return parser.parse_args()
+
+
+if __name__ == "__main__":
+    main()