Skip to content

Azure pipeline #262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 44 commits into from
Jul 4, 2019
Merged

Azure pipeline #262

Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
b30ac22
Azure pipeline
Jun 26, 2019
658bae8
Changes from PR review #262
Jun 27, 2019
5377dd9
Cannot provide variables without a stage definition
Jun 27, 2019
cd28a47
A world of new possibilities
Jun 27, 2019
5942dd6
Try with templates.
Jun 27, 2019
da8c55c
May work!!!
Jun 27, 2019
a5f381c
Test
Jun 27, 2019
4d98086
Rename the template file.
Jun 27, 2019
2fd38ba
Another test for displayNames to be pretty
Jun 27, 2019
a17e3da
Forgot some quotes.
Jun 27, 2019
54534d3
I may be loosing my mind slowly
Jun 27, 2019
9bb3535
Another test
Jun 27, 2019
bd08e8f
Last ty
Jun 27, 2019
a639ff5
Should work but not super pretty
Jun 27, 2019
0cdfd74
Instead of running step on condition, do not add it on condition.
Jun 27, 2019
f35d489
forgot a $
Jun 27, 2019
d0170c0
revert the changes which were not working, but kept some docs.
Jun 27, 2019
322abe9
Triggered only for PRs, not every branch for every commit.
Jun 28, 2019
4c13fb4
Remove publishing steps from the `whole build`. Instead, copy the cre…
Jun 28, 2019
9f16bf4
Publish all the created artifacts if succeeded.
Jun 28, 2019
7f5dd9f
Trigger on tags starting with the letter v
Jun 28, 2019
625a973
Why is it building master???
Jun 28, 2019
68a3614
Is it the PR?
Jun 28, 2019
e6eebd9
Stage to have the publishing one depending on the building one.
Jun 28, 2019
e11c639
Need to publish artifacts per job.
Jun 28, 2019
6ddecc8
Update back the publish artifact step
Jun 28, 2019
65651f0
Let's try the release pipelines.
Jun 28, 2019
f477e86
Add some documentation.
Jun 28, 2019
f511546
ignore folders `vnv`.
Jun 28, 2019
b8d7e99
Publish test results even if tests failed.
Jul 1, 2019
f47fe0c
Build every tag
Jul 1, 2019
687a107
Version v0.13.1-b2
Jul 1, 2019
24cd5ec
Version 0.13.1-b3
Jul 1, 2019
f91870e
Forgot a star
Jul 1, 2019
fc34b07
Was the space really important? And version beta up
Jul 1, 2019
ddcb2b2
Build is `Automated` for every tag, not only the one starting with `v`.
Jul 1, 2019
15ab9f2
Add a MANIFEST.in file to include the read me and the license files i…
Jul 1, 2019
e0e04b2
Bump beta version for final test
Jul 1, 2019
7ac1925
Update some docs.
Jul 1, 2019
a14e8a0
MANIFEST is not the issue.
Jul 1, 2019
f7241df
It seems building a wheel from a windows machine is complicated with …
Jul 1, 2019
5c3adb9
The published packaged version seems to include the readme.
Jul 1, 2019
7ba0bc2
Should be the final version bump for test.
Jul 1, 2019
0c97b1c
Merge branch 'master' of github.com:data61/clkhash into feature-azure…
Jul 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 104 additions & 0 deletions azurePipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
jobs:
- job: 'Test'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment this job appears to test and package and publish... and just on Windows.

pool:
vmImage: 'vs2017-win2016'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
vmImage: 'vs2017-win2016'
vmImage: $(imageName)

Consider adding support for OSX as well as Windows. https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started-multiplatform?view=azure-devops#add-additional-platforms

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not originally part of the spec, but easy change :)

strategy:
matrix:
Python27-x64:
python.version: '2.7'
architecture: 'x64'
# Python34-x64:
# python.version: '3.4'
# architecture: 'x64'
Python35-x64:
python.version: '3.5'
architecture: 'x64'
Python36-x64:
python.version: '3.6'
architecture: 'x64'
Python37-x64:
python.version: '3.7'
architecture: 'x64'
Python27-x86:
python.version: '2.7'
architecture: 'x86'
# Python34-x86:
# python.version: '3.4'
# architecture: 'x86'
Python35-x86:
python.version: '3.5'
architecture: 'x86'
Python36-x86:
python.version: '3.6'
architecture: 'x86'
Python37-x86:
python.version: '3.7'
architecture: 'x86'

steps:
# I would not have found on my own such a step. Thanks https://dvlup.com/2019/01/03/using-powershell-to-installing-msi-in-a-devops-build/
# The solution I found was not working: it hanged forevever, using "msiexec /i"
- powershell: Invoke-WebRequest -Uri https://download.microsoft.com/download/7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/VCForPython27.msi -OutFile VCForPython27.msi
displayName: 'Download Microsoft Visual C++ 9.0 from https://download.microsoft.com/download/7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/VCForPython27.msi'
condition: eq(variables['python.version'], 2.7)
- powershell: Start-Process VCForPython27.msi -ArgumentList "/q" -Wait
displayName: 'Install Microsoft Visual C++ 9.0.'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required? I'm guessing for installing bitarray? #153

If so I think we should add a note in the docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may indeed be related to bitarray but I cannot confirm it is the only requirements requiring it:

Building wheels for collected packages: bashplotlib, bitarray, future, functools32, pefile, pycparser, tornado, simplegeneric, win-unicode-console
  Building wheel for bashplotlib (setup.py): started
  Building wheel for bashplotlib (setup.py): finished with status 'done'
  Stored in directory: C:\Users\VssAdministrator\AppData\Local\pip\Cache\wheels\39\f0\8b\6e42f59f24ca2115413dc1a8195a2fb82f9daeed52839aa631
  Building wheel for bitarray (setup.py): started
  Building wheel for bitarray (setup.py): finished with status 'error'
  Running setup.py clean for bitarray
  ERROR: Complete output from command 'C:\hostedtoolcache\windows\Python\2.7.14\x64\python.exe' -u -c 'import setuptools, tokenize;__file__='"'"'c:\\users\\vssadm~1\\appdata\\local\\temp\\pip-install-omoklv\\bitarray\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'c:\users\vssadm~1\appdata\local\temp\pip-wheel-wal369' --python-tag cp27:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-2.7
  creating build\lib.win-amd64-2.7\bitarray
  copying bitarray\test_bitarray.py -> build\lib.win-amd64-2.7\bitarray
  copying bitarray\__init__.py -> build\lib.win-amd64-2.7\bitarray
  running build_ext
  building 'bitarray._bitarray' extension
  error: Microsoft Visual C++ 9.0 is required. Get it from http://aka.ms/vcpython27

condition: eq(variables['python.version'], 2.7)
# - script: |
# wget -O VCForPython27.msi https://download.microsoft.com/download/7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/VCForPython27.msi
# msiexec.exe /i VCForPython27.msi
# displayName: 'Install Microsoft Visual C++ 10.0 from https://download.microsoft.com/download/A/6/A/A6AC035D-DA3F-4F0C-ADA4-37C8E5D34E3D/winsdk_web.exe'
# condition: eq(variables['python.version'], 3.4)
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
architecture: '$(architecture)'

# Add additional tasks to run using each Python version in the matrix above
- script: |
python --version
python -c "import struct; print(struct.calcsize('P') * 8)"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can remove this whole task - the struct check isn't required anymore and the UsePythonVersion@0 prints the Python version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

displayName: 'Python version and architecture'
env:
INCLUDE_CLI: 1
- script: |
python -m pip install --upgrade pip
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for consistency with the following commands use -U instead of --upgrade

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

python -m pip install -U wheel setuptools codecov
python -m pip install -U -r requirements.txt
python -m pip install .
displayName: 'Pip installs'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
displayName: 'Pip installs'
displayName: 'Install dependencies'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

env:
INCLUDE_CLI: 1
- script: python -m pytest --cov=clkhash --junitxml=testResults.xml --cov-report=xml:coverageReport.xml
displayName: 'Run the tests'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
displayName: 'Run the tests'
displayName: 'Test with pytest for Python $(python.version) ($(architecture))'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is useful, as this step is already part of the job stating the operating system and all.
But note that the test result include them, making the test reports aggregator easy to read.

env:
INCLUDE_CLI: 1
- task: PublishTestResults@2
condition: succeeded()
displayName: 'Publish test results'
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'testResults.xml'
testRunTitle: 'Test results python$(python.version), $(architecture)'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
testRunTitle: 'Test results python$(python.version), $(architecture)'
testRunTitle: 'Publish test results for Python $(python.version) ($(architecture))'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will also include the operating system, but I will not add "Publish" at the beginning: this name us used for the test result report name, not for the task of publishing. This name is then shown in the test tab of the build: https://dev.azure.com/data61/Anonlink/_build/results?buildId=40&view=ms.vss-test-web.build-test-results-tab To see it, you need to include the passed tests as by default it shows only the failed tests.

failTaskOnFailedTests: true
- script: python -m codecov --token $(CODECOV_TOKEN) --file coverageReport.xml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can also render codecov in azure - https://docs.microsoft.com/en-us/azure/devops/pipelines/languages/python?view=azure-devops#publish-code-coverage-results

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

displayName: 'Send coverage to codecov'
condition: and(eq(variables['python.version'], '3.7'), eq(variables['architecture'], 'x86'))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we send codecov reports every version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

- script: |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get the feeling that this should be a new deployment stage?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I thought about it but creating a new job or stage would imply re-installing all the dependencies (as each job uses a different VM). So I chose speed over prettiness of stages/jobs/tasks. But happy to change if preferred.

pyinstaller cli.spec
.\dist\clkutil.exe --version
displayName: 'Build clkutil.exe'
condition: and(eq(variables['python.version'], '3.7'), eq(variables['architecture'], 'x86'))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this condition needs to check that Agent.OS is windows

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

- script: python setup.py bdist_wheel
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- script: python setup.py bdist_wheel
- script: python setup.py sdist bdist_wheel

(Build the source distribution too)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

displayName: 'Build wheel'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
displayName: 'Build wheel'
displayName: 'Package'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

condition: and(eq(variables['python.version'], '3.7'), eq(variables['architecture'], 'x86'))
- script: |
pip install twine
twine upload -u $(PYPI_LOGIN) -p $(PYPI_PASSWORD) --skip-existing dist/*.whl
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note there is a builtin twine auth task: https://docs.microsoft.com/en-us/azure/devops/pipelines/languages/python?view=azure-devops#authenticate-with-twine

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure when the auth task is useful as we still need to run the twine commands after...

displayName: 'Send artifact to Pypi'
condition: and(eq(variables['python.version'], '3.7'), eq(variables['architecture'], 'x86'))
- task: PublishPipelineArtifact@0
inputs:
artifactName: 'Windows artifacts'
targetPath: dist/
condition: and(eq(variables['python.version'], '3.7'), eq(variables['architecture'], 'x86'))