Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement secure horizontal scheme for federated learning #10231

Merged
merged 84 commits into from
Jun 18, 2024
Merged

Implement secure horizontal scheme for federated learning #10231

merged 84 commits into from
Jun 18, 2024

Commits on Jan 31, 2024

  1. Add additional data split mode to cover the secure vertical pipeline

    @ZiyueXu77
    ZiyueXu77 committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    8570ba5 View commit details
    Browse the repository at this point in the history

  2. Add IsSecure info and update corresponding functions

    @ZiyueXu77
    ZiyueXu77 committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    2d00db6 View commit details
    Browse the repository at this point in the history

  3. Modify evaluate_splits to block non-label owners to perform hist comp… 

    …ute under secure scenario
    @ZiyueXu77
    ZiyueXu77 committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    ab17f5a View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. Continue using Allgather for best split sync for secure vertical, equ… 

    …valent to broadcast
    @ZiyueXu77
    ZiyueXu77 committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    fb1787c View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. Modify histogram sync scheme for secure vertical case, can identify g… 

    …lobal best split, but need to further apply split correctly
    @ZiyueXu77
    ZiyueXu77 committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    7a2a2b8 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Sync cut informaiton across clients, full pipeline works for testing … 

    …case
    @ZiyueXu77
    ZiyueXu77 committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    3ca3142 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. Code cleanup, phase 1 of alternative vertical pipeline finished

    @ZiyueXu77
    ZiyueXu77 committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    22dd522 View commit details
    Browse the repository at this point in the history

  2. Code clean

    @ZiyueXu77
    ZiyueXu77 committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    52e8951 View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. change kColS to kColSecure to avoid confusion with kCols

    @ZiyueXu77
    ZiyueXu77 committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    e9eef15 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. Replace allreduce with allgather, functional but inefficient version

    @ZiyueXu77
    ZiyueXu77 committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    91c8a2f View commit details
    Browse the repository at this point in the history

  2. Update AllGather behavior from individual pair to bulk by adopting ne… 

    …w histogram transmission data structure of a flat vector
    @ZiyueXu77
    ZiyueXu77 committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    8340c26 View commit details
    Browse the repository at this point in the history

  3. comment out the record printing

    @ZiyueXu77
    ZiyueXu77 committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    42a9df1 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. fix pointer bug for histsync with allgather

    @ZiyueXu77
    ZiyueXu77 committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    41e5abb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dmlc:master' into SecureBoostP2

    @ZiyueXu77
    ZiyueXu77 authored Feb 20, 2024
    Configuration menu
    Copy the full SHA
    ea5dc98 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. Merge branch 'dmlc:master' into SecureBoostP2

    @ZiyueXu77
    ZiyueXu77 authored Feb 23, 2024
    Configuration menu
    Copy the full SHA
    5d542f8 View commit details
    Browse the repository at this point in the history

  2. identify the HE adding locations

    @ZiyueXu77
    ZiyueXu77 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    d91be10 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. revise and simplify template code

    @ZiyueXu77
    ZiyueXu77 committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    dd60317 View commit details
    Browse the repository at this point in the history

  2. revise and simplify template code

    @ZiyueXu77
    ZiyueXu77 committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    8da824c View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. prepare aggregator for gh broadcast

    @ZiyueXu77
    ZiyueXu77 committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    fb9f4fa View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. prepare histogram for histindex and row index for allgather

    @ZiyueXu77
    ZiyueXu77 committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    e77f8c6 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. Merge branch 'vertical-federated-learning' into SecureBoostP2

    @ZiyueXu77
    ZiyueXu77 authored Mar 15, 2024
    Configuration menu
    Copy the full SHA
    7ef48c8 View commit details
    Browse the repository at this point in the history

  2. fix conflicts

    @ZiyueXu77
    ZiyueXu77 committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    8405791 View commit details
    Browse the repository at this point in the history

  3. fix conflicts

    @ZiyueXu77
    ZiyueXu77 committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    db7d518 View commit details
    Browse the repository at this point in the history

  4. fix format

    @ZiyueXu77
    ZiyueXu77 committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    dd6adde View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. fix allgather logic and update unit test

    @ZiyueXu77
    ZiyueXu77 committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    9567e67 View commit details
    Browse the repository at this point in the history

  2. fix linting

    @ZiyueXu77
    ZiyueXu77 committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    53800f2 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. fix linting and other unit test issues

    @ZiyueXu77
    ZiyueXu77 committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    b7e70f1 View commit details
    Browse the repository at this point in the history

  2. fix linting and other unit test issues

    @ZiyueXu77
    ZiyueXu77 committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    49e8fd6 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. integration with interface initial attempt

    @ZiyueXu77
    ZiyueXu77 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    da0f7a6 View commit details
    Browse the repository at this point in the history

  2. integration with interface initial attempt

    @ZiyueXu77
    ZiyueXu77 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    406cda3 View commit details
    Browse the repository at this point in the history

  3. integration with interface initial attempt

    @ZiyueXu77
    ZiyueXu77 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    f6c63aa View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. functional integration with interface

    @ZiyueXu77
    ZiyueXu77 committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    f223df7 View commit details
    Browse the repository at this point in the history

  2. remove debugging prints

    @ZiyueXu77
    ZiyueXu77 committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    d881d84 View commit details
    Browse the repository at this point in the history

  3. remove processor from another PR

    @ZiyueXu77
    ZiyueXu77 committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    2997cf7 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. Update the processor functions according to new processor implementation

    @ZiyueXu77
    ZiyueXu77 committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    3a1f9ac View commit details
    Browse the repository at this point in the history

  2. Move processor interface init from learner to communicator

    @ZiyueXu77
    ZiyueXu77 committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    1107604 View commit details
    Browse the repository at this point in the history

  3. Move processor interface init from learner to communicator functional

    @ZiyueXu77
    ZiyueXu77 committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    30b7ed5 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. switch to allgatherV for encrypted message with varying lenghts

    @ZiyueXu77
    ZiyueXu77 committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    a3ddf7d View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. consolidate with processor interface PR

    @ZiyueXu77
    ZiyueXu77 committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    3123b51 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. remove prints and fix format

    @ZiyueXu77
    ZiyueXu77 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    73225a0 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. fix linting over reference pass

    @ZiyueXu77
    ZiyueXu77 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    e85b1fb View commit details
    Browse the repository at this point in the history

  2. fix undefined symbol issue

    @ZiyueXu77
    ZiyueXu77 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    57750b4 View commit details
    Browse the repository at this point in the history

  3. fix processor test

    @ZiyueXu77
    ZiyueXu77 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    fa2665a View commit details
    Browse the repository at this point in the history

  4. secure vertical relies on processor, move the unit test

    @ZiyueXu77
    ZiyueXu77 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    87d2fdb View commit details
    Browse the repository at this point in the history

  5. type correction

    @ZiyueXu77
    ZiyueXu77 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    9941293 View commit details
    Browse the repository at this point in the history

  6. type correction

    @ZiyueXu77
    ZiyueXu77 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    dd4f440 View commit details
    Browse the repository at this point in the history

  7. extra linting from last change

    @ZiyueXu77
    ZiyueXu77 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    5b2dfe6 View commit details
    Browse the repository at this point in the history

  8. Added Windows support

    @nvidianz
    nvidianz committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    80d3b89 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. Merge pull request #4 from nvidianz/processor-windows-support 

    Processor windows support
    @ZiyueXu77
    ZiyueXu77 authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    184b67f View commit details
    Browse the repository at this point in the history

  2. fix for cstdint types

    @ZiyueXu77
    ZiyueXu77 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    3382707 View commit details
    Browse the repository at this point in the history

  3. fix for cstdint types

    @ZiyueXu77
    ZiyueXu77 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    2a8f19a View commit details
    Browse the repository at this point in the history

  4. Added support for horizontal secure XGBoost

    @nvidianz
    nvidianz committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    9ff2935 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #5 from nvidianz/processor-horizontal-support 

    Added support for horizontal secure XGBoost
    @ZiyueXu77
    ZiyueXu77 authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    38e9d3d View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. update with mock plugin

    @ZiyueXu77
    ZiyueXu77 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    38c176c View commit details
    Browse the repository at this point in the history

  2. secure horizontal fully functional with mock plugin

    @ZiyueXu77
    ZiyueXu77 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    a5ce92e View commit details
    Browse the repository at this point in the history

  3. linting fix

    @ZiyueXu77
    ZiyueXu77 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    5e824ac View commit details
    Browse the repository at this point in the history

  4. linting fix

    @ZiyueXu77
    ZiyueXu77 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    81db216 View commit details
    Browse the repository at this point in the history

  5. linting fix

    @ZiyueXu77
    ZiyueXu77 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    15c211a View commit details
    Browse the repository at this point in the history

  6. fix type

    @ZiyueXu77
    ZiyueXu77 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    f7341cd View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Merge branch 'vertical-federated-learning' into SecureHorizontal

    @ZiyueXu77
    ZiyueXu77 authored Apr 29, 2024
    Configuration menu
    Copy the full SHA
    a579205 View commit details
    Browse the repository at this point in the history

  2. change loader and proc params input pattern to align with std map

    @ZiyueXu77
    ZiyueXu77 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    35d8c15 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. update with secure vertical incorporation

    @ZiyueXu77
    ZiyueXu77 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    3d31905 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Merge branch 'vertical-federated-learning' into SecureHorizontal

    @ZiyueXu77
    ZiyueXu77 authored May 16, 2024
    Configuration menu
    Copy the full SHA
    7f86787 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Update mock_processor to enable nvflare usage

    @ZiyueXu77
    ZiyueXu77 authored May 28, 2024
    Configuration menu
    Copy the full SHA
    bdcb6e2 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. [backport] Fix compiling with the latest CTX. (dmlc#10263)

    @trivialfis
    trivialfis committed May 29, 2024
    Configuration menu
    Copy the full SHA
    a8205d3 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'ZiyueXu77/SecureHorizontal' into Secure… 

    …Horizontal
    @trivialfis
    trivialfis committed May 29, 2024
    Configuration menu
    Copy the full SHA
    ae77f2d View commit details
    Browse the repository at this point in the history

  3. fix secure horizontal inference

    @ZiyueXu77
    ZiyueXu77 committed May 29, 2024
    Configuration menu
    Copy the full SHA
    cc13605 View commit details
    Browse the repository at this point in the history

  4. initialized aggr context only once

    @ZiyueXu77
    ZiyueXu77 committed May 29, 2024
    Configuration menu
    Copy the full SHA
    d7a6da6 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Added support for multiple plugins in a single lib

    @nvidianz
    nvidianz committed May 30, 2024
    Configuration menu
    Copy the full SHA
    032b14d View commit details
    Browse the repository at this point in the history

  2. Merge pull request #7 from nvidianz/support-multi-processors 

    Support multiple plugins
    @ZiyueXu77
    ZiyueXu77 authored May 30, 2024
    Configuration menu
    Copy the full SHA
    ea9b298 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. remove redundant condition

    @ZiyueXu77
    ZiyueXu77 committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    7f3472e View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Added support for boolean in proc_params

    @nvidianz
    nvidianz committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    5569e78 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #10 from nvidianz/support-bool-params-2nd-try 

    Support bool params in proc_params.
    @ZiyueXu77
    ZiyueXu77 authored Jun 4, 2024
    Configuration menu
    Copy the full SHA
    71e578c View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. free buffer

    @ZiyueXu77
    ZiyueXu77 committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    454f69d View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Merge branch 'vertical-federated-learning' into SecureHorizontal

    @trivialfis
    trivialfis authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    e2f77e2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'vertical-federated-learning' into SecureHorizontal

    @trivialfis
    trivialfis authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    6e4a3fb View commit details
    Browse the repository at this point in the history

  3. CUDA.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    61c8f47 View commit details
    Browse the repository at this point in the history

  4. Fix clean build.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    05be1e8 View commit details
    Browse the repository at this point in the history

  5. Fix include.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    e0795cf View commit details
    Browse the repository at this point in the history

  6. tidy.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    fd4f331 View commit details
    Browse the repository at this point in the history

  7. lint.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    074c63b View commit details
    Browse the repository at this point in the history

  8. nolint.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    2a8fd72 View commit details
    Browse the repository at this point in the history

  9. disable.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    ac02279 View commit details
    Browse the repository at this point in the history

  10. disable sanitizer.

    @trivialfis
    trivialfis committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    1ef69ea View commit details
    Browse the repository at this point in the history