Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Experimental Workflow interface (#632)
* Remove CLA from OpenFL Repo * Experimental workflow interface initial commit * Added Vertical FL Example * WIP * LocalRuntime: Ray support with fallback to single process. Request GPUs through decorator. UI Code needs debugging for jupyter notebook (currently disabled). Bug fixes * Fixed optimizer state in primary example * Changes to support Flowgraph with Jupyter notebook * Changes to support Flowgraph with Jupyter notebook * Test case for validating subset of collaborator works correctly. * Test to validate references. * renamed the file * Clean up next function * Refactored next function working as expected * Added a mutex in MetaflowInterface.create_task * Test case for validating internalloops works correctly. * Test case for validating internalloops works correctly. * Included graph code in metaflow_utils * added pathspec to open graph on browser * Update workflow interface quickstart notebook * Add files via upload * Add files via upload * Cleaned up Workflow Interface 101 tutorial. Added tutorials 102 (aggregator validation), and 201 (requesting exclusive GPUs) * Updated with instructions for Google Colab. Cleared tutorial output * Minor fix to workflow interface GPU tutorial * Fixed after one round filter attributes issue Fix After one round filter attributes does not work properly * Documentation WIP * corrected the spelling mistake * Initial Version of Watermarking Tutorial * Delete Workflow_Interface_MNIST.ipynb Unnecessary changes committed * Remove out of date example * Remove old documentation from flow interface description. Section needs comprehensive descriptions still of interface / runtimes / features * Add metaflow to 3rd party licenses * Adds CI tests for PRs tagged with 'workflow_interface' * CI Pipeline fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * passing input attribute to checkpoint Making changes to save attribute "input" in datastore. * Removing commented code Removing commented code * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * CI Fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fixed issues with test cases Signed-off-by: Patrick Foley <psfoley@gmail.com> * Remove unused dependencies * Revert changes to metaflow utils Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fixing lint errors * Resolving lint suggestions Signed-off-by: Patrick Foley <psfoley@gmail.com> * updated the test case for testflow reference with include and exclude * Test Cases for Include and exclude works correctly. * Global DP Workflow Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * ML Privacy Meter Workflow Co-authored-by: Hongyan Chang <hongyan.chang@intel.com> Co-authored-by: Anindya S. Paul <anindya.s.paul@intel.com> Co-authored-by: Brandon Edwards <brandon.edwards@intel.com> Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Initial version of the Watermarking notebook in Tutorial format * GPU OOM fix. Co-authored-by: Hongyan Chang <hongyan.chang@intel.com> Co-authored-by: Anindya S. Paul <anindya.s.paul@intel.com> Co-authored-by: Brandon Edwards <brandon.edwards@intel.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> * Fix optimizer memory leak. Co-authored-by: Anindya S. Paul <anindya.s.paul@intel.com> Co-authored-by: Brandon Edwards <brandon.edwards@intel.com> Co-authored-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Fix testing function to load tensors into device. Co-authored-by: Hongyan Chang <hongyan.chang@intel.com> Co-authored-by: Anindya S. Paul <anindya.s.paul@intel.com> Co-authored-by: Brandon Edwards <brandon.edwards@intel.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Update Summary 09/12/2022: 1. Generate Watermark dataset in notebook instead of downloading the entire WAFFLE archive 2. Installation of imagen from github source (required for Watermark dataset generation) 3. Updates to Tutorial text 4. Modularized Train function as it is being used in multiple places in the Workflow * Reset and cleared outputs * Delete dataset and dataloader attributes after each round. Co-authored-by: Hongyan Chang <hongyan.chang@intel.com> Co-authored-by: Anindya S. Paul <anindya.s.paul@intel.com> Co-authored-by: Brandon Edwards <brandon.edwards@intel.com> Co-authored-by: Patrick Foley patrick.foley@intel.com Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Resolve ray object store memory leak. Makes single process default option for LocalRuntime * Update readme.md Update image links * Pin requirements for workflow tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * Updated testflow_internalloop.py to fix errors * Updated to fix import errors and Lint issues * Added copyright to Test cases * Fix imports for tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * enable single_process or ray backend in local_runtime Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Update flow_interface.rst * Fix lint warnings in openfl/experimental/placement/placement.py - ./openfl/experimental/placement/placement.py:6:1: F401 'gc' imported but unused - ./openfl/experimental/placement/placement.py:14:1: E302 expected 2 blank lines, found 1 * Fix lint warnings in openfl/experimental/utilities/utils.py - ./openfl/experimental/utilities/utils.py:6:1: E302 expected 2 blank lines, found 1 * Fix lint warnings in openfl/experimental/runtime/runtime.py - ./openfl/experimental/runtime/runtime.py:24:100: E501 line too long (102 > 99 characters) - ./openfl/experimental/runtime/runtime.py:35:26: E711 comparison to None should be 'if cond is not None:' - ./openfl/experimental/runtime/runtime.py:60:16: F541 f-string is missing placeholders * Fix lint warnings in openfl/experimental/interface/fl_spec.py - ./openfl/experimental/interface/fl_spec.py:19:1: F401 'threading.Lock' imported but unused - ./openfl/experimental/interface/fl_spec.py:20:1: F401 'time.sleep' imported but unused - ./openfl/experimental/interface/fl_spec.py:255:13: F841 local variable 'func_refs' is assigned to but never used - ./openfl/experimental/interface/fl_spec.py:256:13: F841 local variable 'remote_functions' is assigned to but never used - ./openfl/experimental/interface/fl_spec.py:257:13: F841 local variable 'remote_ctx_refs' is assigned to but never used * Fix lint warnings in openfl/experimental/utilities/metaflow_utils.py - ./openfl/experimental/utilities/metaflow_utils.py:111:1: E800 Found commented out code - ./openfl/experimental/utilities/metaflow_utils.py:128:24: C402 Unnecessary generator - rewrite as a dict comprehension. - ./openfl/experimental/utilities/metaflow_utils.py:135:1: E800 Found commented out code - ./openfl/experimental/utilities/metaflow_utils.py:152:1: E800 Found commented out code - ./openfl/experimental/utilities/metaflow_utils.py:156:1: E800 Found commented out code - ./openfl/experimental/utilities/metaflow_utils.py:260:61: FS001 '%' operator used - ./openfl/experimental/utilities/metaflow_utils.py:264:21: F841 local variable 'e' is assigned to but never used - ./openfl/experimental/utilities/metaflow_utils.py:276:60: FS001 '%' operator used - ./openfl/experimental/utilities/metaflow_utils.py:280:25: F841 local variable 'e' is assigned to but never used - ./openfl/experimental/utilities/metaflow_utils.py:282:21: F841 local variable 'e' is assigned to but never used - ./openfl/experimental/utilities/metaflow_utils.py:394:42: FS002 '.format' used - ./openfl/experimental/utilities/metaflow_utils.py:476:32: C408 Unnecessary dict call - rewrite as a literal. - ./openfl/experimental/utilities/metaflow_utils.py:497:21: C408 Unnecessary dict call - rewrite as a literal. - ./openfl/experimental/utilities/metaflow_utils.py:535:32: C408 Unnecessary dict call - rewrite as a literal. - ./openfl/experimental/utilities/metaflow_utils.py:536:22: C408 Unnecessary dict call - rewrite as a literal. * fix lint errorrs with black reformatting. Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * fix minor typo Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Update flow_interface.rst * Update flow_interface.rst * Refactored LocalRuntime and updated the runtime package file structure as follows: 1. Created seperate files for each Runtime 2. Refacoted excute_task method in LocalRuntime from FLSpec 3. Methods used commonly in both functions next, and execute_task are moved to utilities package runtime_utils.py file 4. Updated Watermarking Tutorial to correct the checkpoint usage (flflow.checkpoint to flflow._checkpoint) 5. Resolved flake8 suggestions in openfl/experimental package * 1. Updated Watermarking Tutorial to correct the checkpoint usage (flflow.checkpoint to flflow._checkpoint) * Add standardized requirements for workflow interface. Upgrade protobuf minor version Signed-off-by: Patrick Foley <psfoley@gmail.com> * Install common workflow requirements. Small updates to Privacy Meter Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update global db requirements Signed-off-by: Patrick Foley <psfoley@gmail.com> * Vertical FL Examples Signed-off-by: Patrick Foley <psfoley@gmail.com> * Remove extra lines from notebook (no checkpointing for 2 party example) Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fixed issue of aggregator privates are visible in collaborator steps via runtime. * Update flow_interface.rst * Modified collaborator to be a private attribute of LocalRuntime so that collaborator private variables are not accessible to aggregators * Collaborator is made a private attribute of local runtime. Due to this local_runtime._collaborators is no longer accessible. Updated Workflow Interface testcases and Tutorials with this change * Added a new Test case to validate accessibility of private attributes - Aggregator private attributes are accessible only in Aggregator steps - Collaborator private attributes are accessible only in Collaborator steps * Addressing lint failures (WIP) Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Fixed Lint issues * Lint fix * Include more tests, validate both runtime backends Signed-off-by: Patrick Foley <psfoley@gmail.com> * Create Base Runtime Specification Signed-off-by: Patrick Foley <psfoley@gmail.com> * Added back readme.md Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Update readme.md Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * add versions for packages Signed-off-by: Hongyan chang <hongyan@comp.nus.edu.sg> * Addressing review comments * fixing review comments Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * Added type hints, docstrings. Distinguishing private methods from public * Fix circular dependencies resulting from type annotations * Add type annotations * Address lint naming errors * More docstrings and type hints * flake8 fixes Signed-off-by: Patrick Foley <psfoley@gmail.com> * Added cyclic institutional incremental learning example * Resolve review comments - add docstrings to tests * using Generator class for fixing seed Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * Make flake8 exceptions * added doscstrings to several functions for better user understanding about the functionality Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> * Fix jupyter notebook links. Update documentation / create links from README * More README updates --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Paul, Anindya S <anindya.s.paul@intel.com> Signed-off-by: Hongyan chang <hongyan@comp.nus.edu.sg> Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Keerti Talwar <keertix.talwar@intel.com> Co-authored-by: Greesh-N <greeshmax.greeshma@intel.com> Co-authored-by: ParthM-GitHub <parthx.mandaliya@intel.com> Co-authored-by: Sachin Gupta <saching.gupta.dsp@gmail.com> Co-authored-by: Paul, Anindya S <anindya.s.paul@intel.com> Co-authored-by: Hongyan Chang <hongyan.chang@intel.com> Co-authored-by: Brandon Edwards <brandon.edwards@intel.com> Co-authored-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> Co-authored-by: Hongyan chang <hongyan@comp.nus.edu.sg>
- Loading branch information
11 people
authored
Jan 11, 2023
1 parent
b1e1d39
commit 942fccc