Skip to content

Commit ea99c5c

Browse files
gusmithgusmith
authored
Azure pipeline (#262)
* Azure pipeline Can replace appveyor. Release to pypi for every new tag on github.
1 parent dae4497 commit ea99c5c

File tree

6 files changed

+192
-5
lines changed

6 files changed

+192
-5
lines changed

‎.azurePipeline/wholeBuild.yml

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
parameters:
2+
pythonVersions: []
3+
operatingSystems: []
4+
architectures: []
5+
6+
jobs:
7+
- job: 'test_and_publish'
8+
strategy:
9+
matrix:
10+
${{ each py in parameters.pythonVersions }}:
11+
${{ each os in parameters.operatingSystems }}:
12+
${{ each architecture in parameters.architectures }}:
13+
${{ format('{0}_Python_{1}_{2}', os, py, architecture) }}:
14+
pythonVersion: ${{ py }}
15+
operatingSystem: ${{ os }}
16+
architecture: ${{ architecture }}
17+
18+
pool:
19+
vmImage: $(operatingSystem)
20+
21+
displayName: 'Test and publish for '
22+
23+
steps:
24+
# I would not have found on my own such a step. Thanks https://dvlup.com/2019/01/03/using-powershell-to-installing-msi-in-a-devops-build/
25+
# The solution I found was not working: it hanged forevever, using "msiexec /i"
26+
# These steps are required because of at least the bitarray dependency (but maybe more). See issue #153 tracking if bitarray is still used.
27+
- powershell: Invoke-WebRequest -Uri https://download.microsoft.com/download/7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/VCForPython27.msi -OutFile VCForPython27.msi
28+
displayName: 'Download Microsoft Visual C++ 9.0 from https://download.microsoft.com/download/7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/VCForPython27.msi'
29+
condition: eq(variables['pythonVersion'], 2.7)
30+
- powershell: Start-Process VCForPython27.msi -ArgumentList "/q" -Wait
31+
displayName: 'Install Microsoft Visual C++ 9.0.'
32+
condition: and(eq(variables['operatingSystem'], 'vs2017-win2016'), eq(variables['pythonVersion'], 2.7))
33+
34+
- task: UsePythonVersion@0
35+
inputs:
36+
versionSpec: '$(pythonVersion)'
37+
architecture: '$(architecture)'
38+
39+
- script: |
40+
python -m pip install -U pip
41+
python -m pip install -U wheel setuptools codecov
42+
python -m pip install -U -r requirements.txt
43+
displayName: 'Install requirements'
44+
45+
- script: python setup.py sdist bdist_wheel
46+
displayName: 'Package'
47+
48+
- script: python -m pytest --cov=clkhash --junitxml=testResults.xml --cov-report=xml:coverageReport.xml --cov-report=html:htmlcov
49+
displayName: 'Test with pytest'
50+
env:
51+
INCLUDE_CLI: 1
52+
53+
- task: PublishTestResults@2
54+
displayName: 'Publish test results in Azure'
55+
condition: succeededOrFailed()
56+
inputs:
57+
testResultsFormat: 'JUnit'
58+
testResultsFiles: 'testResults.xml'
59+
testRunTitle: 'Test results on a vm $(operatingSystem) ($(architecture)) for Python $(pythonVersion)'
60+
failTaskOnFailedTests: true
61+
62+
- task: PublishCodeCoverageResults@1
63+
displayName: 'Publish code coverage in Azure'
64+
# If the previous stage fail, we still want to run this one as the previous stage may fail because of a failing test.
65+
condition: succeededOrFailed()
66+
inputs:
67+
codeCoverageTool: Cobertura
68+
summaryFileLocation: 'coverageReport.xml'
69+
# Seems to create warning as this step is already creating its own html pages.
70+
# reportDirectory: 'htmlcov'
71+
failIfCoverageEmpty: true
72+
73+
- script: python -m codecov --token $(CODECOV_TOKEN) --file coverageReport.xml
74+
displayName: 'Send coverage to codecov'
75+
condition: succeededOrFailed()
76+
77+
- script: |
78+
pyinstaller cli.spec
79+
.\dist\clkutil.exe --version
80+
displayName: 'Build clkutil.exe'
81+
condition: and(eq(variables['operatingSystem'], 'vs2017-win2016'), eq(variables['pythonVersion'], '3.7'), eq(variables['architecture'], 'x86'))
82+
83+
- task: PublishPipelineArtifact@0
84+
displayName: 'Publish artifacts'
85+
condition: and(eq(variables['pythonVersion'], '3.7'), eq(variables['architecture'], 'x86'))
86+
inputs:
87+
artifactName: Artifacts
88+
targetPath: 'dist/'

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ lib64/
2222
parts/
2323
sdist/
2424
var/
25+
venv/
2526
*.egg-info/
2627
.installed.cfg
2728

‎azurePipeline.yml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
trigger:
2+
branches:
3+
include:
4+
- master
5+
exclude:
6+
- '*'
7+
tags:
8+
include:
9+
- '*'
10+
pr:
11+
branches:
12+
include:
13+
- '*'
14+
15+
jobs:
16+
- template: .azurePipeline/wholeBuild.yml # Template reference
17+
parameters:
18+
pythonVersions: ['3.7', '3.6', '3.5', '2.7']
19+
# operatingSystems: ['ubuntu-16.04', 'macos-10.13', 'vs2017-win2016']
20+
operatingSystems: ['vs2017-win2016']
21+
architectures: ['x64', 'x86']
22+
23+
- job:
24+
steps:
25+
# In this step, if this build is triggered by a tag, it will add a tag 'Automated' to the current build.
26+
- script: echo "##vso[build.addbuildtag]Automated"
27+
condition: startsWith(variables['Build.SourceBranch'], 'refs/tags/')

‎docs/devops.rst

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
Devops
2+
===========
3+
4+
5+
.. toctree::
6+
:maxdepth: 2
7+
8+
clkhash
9+
10+
Azure Pipeline
11+
--------------
12+
13+
``clkhash`` is automatically built and tested using Azure Pipeline
14+
for Windows environment, in the project `Anonlink <https://dev.azure.com/data61/Anonlink>`
15+
16+
Two pipelines are available:
17+
- `Build pipeline <https://dev.azure.com/data61/Anonlink/_build?definitionId=2>`,
18+
- `Release pipeline <https://dev.azure.com/data61/Anonlink/_release?definitionId=1>`.
19+
20+
The build pipeline is described by the script `azurePipeline.yml`
21+
which is using resources from the folder `.azurePipeline`.
22+
Mainly, a number of builds and tests are started for different
23+
version of python and system architecture.
24+
Only the packages created with ``Python 3.7`` and the ``x86``
25+
architecture are then published (in Azure).
26+
27+
The build pipeline is triggered for every pushes on the master branch,
28+
for every tagged commit, and for every pushes part of a pull
29+
request. We are not building on every push and
30+
pull requests not to build twice the same code. For every tagged commit,
31+
the build pipeline will also add the Azure tag `Automated` which will trigger
32+
automatically the release pipeline.
33+
34+
The build pipeline does:
35+
- install the requirements,
36+
- package ``clkhash``,
37+
- run `pytest`,
38+
- publish the test results,
39+
- publish the code coverage (on Azure and codecov),
40+
- publish the artifacts from the build using ``Python 3.7``
41+
with a ``x86`` architecture (i.e. a whl, a tar.gz and an exe).
42+
43+
The build pipeline requires one environment variable provided by Azure environment:
44+
- `CODECOV_TOKEN` which is used to publish the coverage to codecov.
45+
46+
47+
The release pipeline can either be triggered manually, or automatically from
48+
a successful build on master where the build is tagged `Automated`
49+
(i.e. if the commit is tagged, cf previous paragraph).
50+
51+
The release pipeline consists of two steps:
52+
- asking for a manual confirmation that the artifacts from the
53+
triggering build should be released,
54+
- uses ``twine`` to publish the artifacts.
55+
56+
The release pipeline requires two environment variables provided by Azure environment:
57+
- `PYPI_LOGIN`: login to push an artifact to ``clkhash`` ``Pypi`` repository,
58+
- `PYPI_PASSWORD`: password to push an artifact to ``clkhash`` ``Pypi`` repository
59+
for the user `PYPI_LOGIN`.
60+

‎docs/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ Table of Contents
3333
cli
3434
schema
3535
development
36+
devops
3637
rest_client
3738
references
3839

‎setup.py

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
from setuptools import setup, find_packages
2-
import codecs
2+
import sys
3+
if sys.version_info[0] < 3:
4+
import codecs
5+
36

47
requirements = [
58
"bitarray>=0.8",
@@ -16,13 +19,20 @@
1619
"typing>=3.6; python_version < '3.5'", # Backport from Py3.5
1720
"bashplotlib>=0.6.5"
1821
]
19-
20-
with codecs.open('README.md', 'r', 'utf-8') as f:
21-
readme = f.read()
22+
if sys.version_info[0] < 3:
23+
# Python 2.7 does not accept the parameter `encoding` to open a file
24+
with codecs.open('README.md', 'r', 'utf-8') as f:
25+
readme = f.read()
26+
else:
27+
# But on Windows, something wrong happens when creating the package
28+
# and using codecs.open. related issues are https://github.com/di/markdown-description-example/issues/4
29+
# But this is not exactly the same either...
30+
with open('README.md', 'r', encoding='utf-8') as f:
31+
readme = f.read()
2232

2333
setup(
2434
name="clkhash",
25-
version='0.13.1b1',
35+
version='0.13.1-b6',
2636
description='Encoding utility to create Cryptographic Linkage Keys',
2737
long_description=readme,
2838
long_description_content_type='text/markdown',

0 commit comments

Comments
 (0)