Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to bare Windows GPU VMs and fix build issues #475

Merged
merged 14 commits into from
Jan 23, 2020
Merged

Move to bare Windows GPU VMs and fix build issues #475

merged 14 commits into from
Jan 23, 2020

Conversation

youngpark
Copy link
Contributor

Description

This PR addresses the following issues:

  • Lack of stability on the Windows build/test infrastructure
    • Fix:
      • Move the test infrastructure to use bare Windows GPU VMs
      • Add a Azure DevOps pipeline yaml file specifically for Windows
  • ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (unknown location)
  • TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
    • Fix:
      • We need to use a fork of the cocodataset/cocoapi to support windows builds, so we have the option of choosing between a) pinning numpy < 1.18 or b) or taking the fix from commit: cocodataset/cocoapi@6c3b394
      • Pinning the version of numpy is only a temporary hack as numpy stopped accepting a floating point value in numpy.linspace() since version 1.12 and will not be supported in future releases.
      • Hence, I created a PR to the fork with the required changes: Fix numpy.float64 cannot be safely interpreted as an integer issue philferriere/cocoapi#9
      • Since it looks like the fork isn't being actively maintained, so I updated the environment.yml file to use a different fork that has the changes merged to unblock us for now.
      • Also added the required fix to utils_cv/detection/plot.py as well.

Related Issues

[BUG] Error occurred: User program failed with TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer

Checklist:

  • I have followed the contribution guidelines and code style for this project.
  • I have added tests covering my contributions.
  • I have updated the documentation accordingly.
  • This PR is being made to staging and not master

@youngpark youngpark requested a review from PatrickBue as a code owner January 23, 2020 16:00
@youngpark youngpark merged commit 4cc09a2 into microsoft:staging Jan 23, 2020
youngpark added a commit that referenced this pull request Jan 23, 2020
* Updating AzureDevOps pipeline agent pool to use a barebones Windows GPU VM.

* Added a AzureDevOps pipeline for windows.

* Added a conda init step for powershell.

* Adding conda to PATH as a separate step

* Pinning pillow to 6.1 to fix issues listed in: python-pillow/Pillow#4130

* chained powershell commands

* Updated AzureDevOps yml file to use the inline powershell script syntax.

* Moved to using inline commands + Invoke-Expression syntax

* Adding indentation for chained commands.

* Moving to the literal block style indicator for yaml for chained commands.

* Using call activate instead of source activate when activating the conda environment through powershell.

* Switching to script steps instead of using powershell steps for conda operations.

* Fixed the numpy.float64 issue by moving to a fork with the upstream fixes for the issue.

* Fixed the numpy.float64 issue in utils/cv/detection/plot.py
youngpark added a commit that referenced this pull request Jan 23, 2020
* Removed submodules.

* Add back submodules using https://

* simplified plotting functions

* fixed most tests

* fixed test

* fixed unit test

* small text edits to the 02 notebook

* added fct description

* tiny cleanup on notebook

* Add pretrained keypoint model (#453)

* Add pretrained keypoint model

* Fix bugs in tests

* Add 03 notebook in conftest.py

* Minor revision

* Reformat code using black

* if folder exists, remove (#448)

* Add mask annotation tool (#447)

* Add mask annotation tool

* Update mask annotation explanation and add converion scripts

* Add screenshots of Labelbox annotation

* Rearrange screenshots

* Move convertion script into functions in data.py

* Point out annotation conversion scripts clearly in notebook

* Refine annotation conversion scripts

* Fix bugs

* Add tests for labelbox format conversion methods

* Move r2p1d from contrib to scenarios.

* Update .gitignore.

* Add README.md

* Remove the folder /scenario/action_recognition/data/samples; update notebook to use web url for sample data.

* Move data split files to data/misc; update notebook accordingly.

* Update data path.

* Add keypoint detection with tuned model (#454)

* Add keypoint detetion with tuned model

* Add tests

* Minor revision

* Update tests

* Fix bugs in tests

* Use GPU device if available

* Update tests

* Fix bug: 'not idx' will be 'True' if 'idx=0'

* Fix bugs

* Move toy keypoint meta into notebook

* Fix bugs

* Fix bugs

* Fix bugs in notebook

* Add descriptions for keypoint meta data

* Raise exception when RandomHorizontalFlip is used without specifying hflip_inds

* Add NOTICE file.

* Add keypoint detection model tuning with top and bottom keypoints (#456)

* Add keypoint detection model tuning with top and bottom keypoints

* Fix undefined unzip_url

* Resolved undefined od_urls

* Add annotation tool to scenarios.

* Plot keypoints as round dots to make them noticeable (#458)

* Plot keypoints as dots

* Change variable naming

* Resolve test machine failure (#460)

This is due to the latest PyTorch (version 1.3) from conda is built on
CUDA 10.1 while the version on the test machine is CUDA 10.0.

* Remove unused imports in 02_mask_rcnn.ipynb (#463)

* Remove unused imports in 02_mask_rcnn.ipynb

* Add missing imports

* Simplify binary_mask() (#464)

* clean up of keypoint detection notebook

* minor clean-up

* remove conflict code (#471)

* Update README.md (#472)

* updated readmes

* added images

* updated readmes

* added intro figure

* modified new picture

* update figure

* Move to bare Windows GPU VMs and fix build issues (#475)

* Updating AzureDevOps pipeline agent pool to use a barebones Windows GPU VM.

* Added a AzureDevOps pipeline for windows.

* Added a conda init step for powershell.

* Adding conda to PATH as a separate step

* Pinning pillow to 6.1 to fix issues listed in: python-pillow/Pillow#4130

* chained powershell commands

* Updated AzureDevOps yml file to use the inline powershell script syntax.

* Moved to using inline commands + Invoke-Expression syntax

* Adding indentation for chained commands.

* Moving to the literal block style indicator for yaml for chained commands.

* Using call activate instead of source activate when activating the conda environment through powershell.

* Switching to script steps instead of using powershell steps for conda operations.

* Fixed the numpy.float64 issue by moving to a fork with the upstream fixes for the issue.

* Fixed the numpy.float64 issue in utils/cv/detection/plot.py

Co-authored-by: PatrickBue <[email protected]>
Co-authored-by: Simon Zhao <[email protected]>
Co-authored-by: Miguel González-Fierro <[email protected]>
Co-authored-by: JS <[email protected]>
@gautamchitnis
Copy link

@PatrickBue Take a look at this fork for cocoapi updates towards windows.

fboylu added a commit that referenced this pull request Jun 16, 2020
* Syncing staging <> master branches (#476)

* Removed submodules.

* Add back submodules using https://

* simplified plotting functions

* fixed most tests

* fixed test

* fixed unit test

* small text edits to the 02 notebook

* added fct description

* tiny cleanup on notebook

* Add pretrained keypoint model (#453)

* Add pretrained keypoint model

* Fix bugs in tests

* Add 03 notebook in conftest.py

* Minor revision

* Reformat code using black

* if folder exists, remove (#448)

* Add mask annotation tool (#447)

* Add mask annotation tool

* Update mask annotation explanation and add converion scripts

* Add screenshots of Labelbox annotation

* Rearrange screenshots

* Move convertion script into functions in data.py

* Point out annotation conversion scripts clearly in notebook

* Refine annotation conversion scripts

* Fix bugs

* Add tests for labelbox format conversion methods

* Move r2p1d from contrib to scenarios.

* Update .gitignore.

* Add README.md

* Remove the folder /scenario/action_recognition/data/samples; update notebook to use web url for sample data.

* Move data split files to data/misc; update notebook accordingly.

* Update data path.

* Add keypoint detection with tuned model (#454)

* Add keypoint detetion with tuned model

* Add tests

* Minor revision

* Update tests

* Fix bugs in tests

* Use GPU device if available

* Update tests

* Fix bug: 'not idx' will be 'True' if 'idx=0'

* Fix bugs

* Move toy keypoint meta into notebook

* Fix bugs

* Fix bugs

* Fix bugs in notebook

* Add descriptions for keypoint meta data

* Raise exception when RandomHorizontalFlip is used without specifying hflip_inds

* Add NOTICE file.

* Add keypoint detection model tuning with top and bottom keypoints (#456)

* Add keypoint detection model tuning with top and bottom keypoints

* Fix undefined unzip_url

* Resolved undefined od_urls

* Add annotation tool to scenarios.

* Plot keypoints as round dots to make them noticeable (#458)

* Plot keypoints as dots

* Change variable naming

* Resolve test machine failure (#460)

This is due to the latest PyTorch (version 1.3) from conda is built on
CUDA 10.1 while the version on the test machine is CUDA 10.0.

* Remove unused imports in 02_mask_rcnn.ipynb (#463)

* Remove unused imports in 02_mask_rcnn.ipynb

* Add missing imports

* Simplify binary_mask() (#464)

* clean up of keypoint detection notebook

* minor clean-up

* remove conflict code (#471)

* Update README.md (#472)

* updated readmes

* added images

* updated readmes

* added intro figure

* modified new picture

* update figure

* Move to bare Windows GPU VMs and fix build issues (#475)

* Updating AzureDevOps pipeline agent pool to use a barebones Windows GPU VM.

* Added a AzureDevOps pipeline for windows.

* Added a conda init step for powershell.

* Adding conda to PATH as a separate step

* Pinning pillow to 6.1 to fix issues listed in: python-pillow/Pillow#4130

* chained powershell commands

* Updated AzureDevOps yml file to use the inline powershell script syntax.

* Moved to using inline commands + Invoke-Expression syntax

* Adding indentation for chained commands.

* Moving to the literal block style indicator for yaml for chained commands.

* Using call activate instead of source activate when activating the conda environment through powershell.

* Switching to script steps instead of using powershell steps for conda operations.

* Fixed the numpy.float64 issue by moving to a fork with the upstream fixes for the issue.

* Fixed the numpy.float64 issue in utils/cv/detection/plot.py

Co-authored-by: PatrickBue <[email protected]>
Co-authored-by: Simon Zhao <[email protected]>
Co-authored-by: Miguel González-Fierro <[email protected]>
Co-authored-by: JS <[email protected]>

* fix to load from saved mask-rcnn model

* fix to load from saved mask-rcnn model

* fix to load from saved mask-rcnn model

Co-authored-by: Young Park <[email protected]>
Co-authored-by: PatrickBue <[email protected]>
Co-authored-by: Simon Zhao <[email protected]>
Co-authored-by: Miguel González-Fierro <[email protected]>
Co-authored-by: JS <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants