[WIP] 6054 Integrate azureml for autorunner training#6088

Draft

peterhessey wants to merge 76 commits into

Project-MONAI:devfrom

peterhessey:6054-integrate-azureml-for-autorunner-training

peterhessey commented Mar 1, 2023 •

edited

Loading

Finishes first task in #6054 .

Description

Introduces changes necessary to make it possible to train the Auto3dSeg AutoRunner training in AzureML with no/minimal impact to the existing user experience. Requires changes made in this (currently-open) PR for the hi-ml-azure package.

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

peterhessey and others added 22 commits

February 20, 2023 11:04


          ➕ Add minimum requirements for triggering AzureML

76f3737


          Set AzureML to SDK v1 only

c762d9e

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          🚧 Set up AzureML environment file

6288cdc


          ✨ Enable job submission to run correct command

9d0b98f


          📌 Add hi-ml to AzureML env

dc8101b


          📌 AzureML environment working

2b88637


          🚧 Enable configuring training using task.yaml

136ec66


          🚧 Pass in azureml args as dict

49dbb04


          🚧 Fix AzureML training arg parsing

c4b8d4e


          🐛 🚧 Fix training to work with AzureML dataroot

ce16775


          🚧 Attempt to run on Prostate data

f5ea9d6


          🔇 Remove old logs + configs

218cb8b


          🙈 Update gitignore for azureml_configs

f547930


          🐛 Fix creating local work_dir when running in Azure

9a1e7c7


          🙈 🐛 Update gitignore and fix work dir creation

4c024bc


          🚨 Pass linting tests

e6e3456


          Merge branch 'Project-MONAI:dev' into 6054-integrate-azureml-for-auto…

b7cf860

…runner-training


          [pre-commit.ci] auto fixes from pre-commit.com hooks

d35d6e7

for more information, see https://pre-commit.ci


          🐛 Update optional imports

2ed7ab0


          Merge branch '6054-integrate-azureml-for-autorunner-training' of http…

1d40819

…s://github.com/peterhessey/MONAI into 6054-integrate-azureml-for-autorunner-training


          DCO Remediation Commit for Peter Hessey <a-phessey@microsoft.com>

a3342e4

I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 76f3737
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 6288cdc
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 9d0b98f
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: dc8101b
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 2b88637
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 136ec66
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 49dbb04
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: c4b8d4e
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: ce16775
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: f5ea9d6
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 218cb8b
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: f547930
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 9a1e7c7
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 4c024bc
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: e6e3456
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 2ed7ab0

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Fix mypy

91d7320

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>

peterhessey changed the title ~~6054 integrate azureml for autorunner training - WORK IN PROGRESS~~ [WIP] 6054 Integrate azureml for autorunner training

peterhessey and others added 7 commits

March 6, 2023 09:29


          📌 Pin packages to remove Azure warnings

244cf81


          ✅ 🚧 Add azureml auto3dseg test for heart data

d7d2af8


          🚧 Start building AzureML test using synthetic data

dcfee01


          [pre-commit.ci] auto fixes from pre-commit.com hooks

4145a78

for more information, see https://pre-commit.ci


          Merge branch 'Project-MONAI:dev' into 6054-integrate-azureml-for-auto…

aeec9f4

…runner-training


          🎨 Restructure auto3dseg synth data functions

e3eacf4

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          ✅ Add synthetic data test

9c0cc4d

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>

peterhessey and others added 26 commits

March 23, 2023 12:01


          ✅ Pass Auto3dSeg AzureML test with subclass

354fde1


          📝 Add docstrings

3a60f6d


          📝 Add more docstrings

a800423


          Add typing and docstrings

662c6b4

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          DCO Remediation Commit for Peter Hessey <a-phessey@microsoft.com>

468f893

I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 354fde1
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 3a60f6d
I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: a800423

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

34f51b1


          [pre-commit.ci] auto fixes from pre-commit.com hooks

45e482f

for more information, see https://pre-commit.ci


          Styling fixes

838abb7

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Merge branch '6054-integrate-azureml-for-autorunner-training' of http…

364f311

…s://github.com/peterhessey/MONAI into 6054-integrate-azureml-for-autorunner-training


          Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

b7292f4


          Remove num_fold arg

a073b76

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Remove need for --azureml flag

229a48e

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          🚧 Reduce AzureML environment complexity

a43bd9b


          Update env again

fe49bee

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          DCO Remediation Commit for Peter Hessey <a-phessey@microsoft.com>

55d2aaf

I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: a43bd9b

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Add fire to AzureML environment

2b642a5

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Remove num_fold arg from AMLAutoRunner

d66c65f

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          🚧 🐛 Bugfix and env update


          DCO Remediation Commit for Peter Hessey <a-phessey@microsoft.com>

345c191

I, Peter Hessey <a-phessey@microsoft.com>, hereby add my Signed-off-by to this commit: 0967115

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…

b8644ff

…ml-for-autorunner-training

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          mypy fixes

ff5c4f4

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…

4c275b3

…ml-for-autorunner-training

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

510b141

for more information, see https://pre-commit.ci


          Black fixes

643d713

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

d81bf97


          Merge branch 'dev' into 6054-integrate-azureml-for-autorunner-training

86adbdf

lapellaniz reviewed

View reviewed changes

environment-azureml.yml

+                    - matplotlib==3.7.1
+                    - mlflow==2.2.2
+                    - mlflow-skinny==2.2.2
+                    - git+https://github.com/peterhessey/MONAI.git@6054-integrate-azureml-for-autorunner-training

lapellaniz Apr 12, 2023

This needs to be reviewed and changed back to MONAI repo during PR approval

peterhessey added 2 commits

April 13, 2023 10:05


          Merge remote-tracking branch 'upstream/dev' into 6054-integrate-azure…

2e9c6a1

…ml-for-autorunner-training

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>


          Minor fixes

4833e13

Signed-off-by: Peter Hessey <a-phessey@microsoft.com>

wyli mentioned this pull request

Refactor multi-node running command into dedicated functions #6623

Merged

7 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

At least 1 approving review is required to merge this pull request.

Labels

None yet