Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds changes for one branch pipelines #945

Merged
merged 20 commits into from
Jul 28, 2022
Merged

Adds changes for one branch pipelines #945

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ddaae4e
Adds changes for one branch pipelines
gurkanindibay Jul 25, 2022
0893987
Reflects changes into script files
gurkanindibay Jul 25, 2022
a60e914
Adds sudo and change parameter for one branch
gurkanindibay Jul 25, 2022
5e93913
Removes 'u' bash flag to handle undefined parameter
gurkanindibay Jul 25, 2022
0f63517
Removes 'u' bash flag to handle undefined parameter
gurkanindibay Jul 25, 2022
32e0e2c
Adds ACR environment variables
gurkanindibay Jul 25, 2022
a033d61
Adds ACR environment variables
gurkanindibay Jul 25, 2022
16d5379
Adds tagging
gurkanindibay Jul 25, 2022
28b7062
changes parameter name
gurkanindibay Jul 25, 2022
c413fa1
Changes tools version in pipelines
gurkanindibay Jul 25, 2022
c9a56ad
Updates script files from latest template
gurkanindibay Jul 25, 2022
fa9416e
Rollbacks test removal changes
gurkanindibay Jul 25, 2022
2c2e719
Merge branch 'develop' into one_branch_image_changes
gurkanindibay Jul 25, 2022
ef914d2
Fixes auto merge problems
gurkanindibay Jul 26, 2022
d6971ec
Fixed undefined parameter check
gurkanindibay Jul 28, 2022
cc62f33
Changes tools branch name into one_branch image
gurkanindibay Jul 28, 2022
53893b6
Fixes review comments
gurkanindibay Jul 28, 2022
cf85c71
Activates u flag after changing empty parameter
gurkanindibay Jul 28, 2022
952e177
Updates script files
gurkanindibay Jul 28, 2022
bdc2623
Updates tools versions
gurkanindibay Jul 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/build-package-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ env:
PACKAGING_PASSPHRASE: "${{ secrets.PACKAGING_PASSPHRASE }}"
DOCKERHUB_PASSWORD: "${{ secrets.DOCKERHUB_PASSWORD }}"
DOCKERHUB_USER_NAME: "${{ secrets.DOCKERHUB_USER_NAME }}"
ACR_PASSWORD: "${{ secrets.ACR_PASSWORD }}"
ACR_SERVICE_PRINCIPAL_ID: "${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}"
CURRENT_BRANCH: ${GITHUB_REF##*/}
TEST: true
on:
Expand Down Expand Up @@ -73,7 +75,7 @@ jobs:
POSTGRES_VERSION: ${{ matrix.POSTGRES_VERSION }}

- name: Clone tools repo for test
run: git clone -b v0.8.15 --depth=1 https://github.com/citusdata/tools.git tools
run: git clone -b one_branch_additions --depth=1 https://github.com/citusdata/tools.git tools

- name: Execute packaging tests
run: |
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/build-package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ env:
PACKAGING_PASSPHRASE: "${{ secrets.PACKAGING_PASSPHRASE }}"
DOCKERHUB_PASSWORD: "${{ secrets.DOCKERHUB_PASSWORD }}"
DOCKERHUB_USER_NAME: "${{ secrets.DOCKERHUB_USER_NAME }}"
ACR_PASSWORD: "${{ secrets.ACR_PASSWORD }}"
ACR_SERVICE_PRINCIPAL_ID: "${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}"
CURRENT_BRANCH: "${GITHUB_REF##*/}"
TEST: false
on:
Expand Down Expand Up @@ -74,7 +76,8 @@ jobs:
POSTGRES_VERSION: ${{ matrix.POSTGRES_VERSION }}

- name: Clone tools repo for test
run: git clone -b v0.8.15 --depth=1 https://github.com/citusdata/tools.git tools
run: git clone -b one_branch_additions --depth=1 https://github.com/citusdata/tools.git tools


- name: Execute packaging tests
run: |
Expand Down
6 changes: 6 additions & 0 deletions ci/push_images
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,10 @@ done <"${topdir}/os-list.csv"

echo -e "${args}" | xargs -t -L1 -P "${nprocs}" docker

docker tag citus/"${image_name}":"${tag}" citusengine.azurecr.io/citus/"${image_name}":"${tag}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACR Push code block

echo "${ACR_PASSWORD}" | docker login citusengine.azurecr.io -u "${ACR_SERVICE_PRINCIPAL_ID}" --password-stdin
ONEBRANCH_ACR_REGISTRY_NAME="citusengine.azurecr.io/citus/"
ACR_COMMAND=$(echo "${args}" | sed "s#citus/#${ONEBRANCH_ACR_REGISTRY_NAME}#")
echo -e "${ACR_COMMAND}" | xargs -t -L1 -P "${nprocs}" docker


13 changes: 12 additions & 1 deletion dockerfiles/centos-7-pg10/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When initializing the docker image in OneBranch, docker run process fall into this block so it does not die

# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-7-pg11/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-7-pg12/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-7-pg13/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-7-pg14/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-8-pg10/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-8-pg11/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-8-pg12/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-8-pg13/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
13 changes: 12 additions & 1 deletion dockerfiles/centos-8-pg14/scripts/fetch_and_build_rpm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down Expand Up @@ -150,6 +159,8 @@ mkdir -p "${rpmbuilddir}"
pkgsrcdir="${builddir}/${pkgname}-${packageversion}"
mkdir "${pkgsrcdir}"



download=$(mktemp)
tarballurl="https://api.github.com/repos/${repopath}/tarball/${gitsha}"
curl -sL "${tarballurl}" -o "${download}"
Expand Down
1 change: 1 addition & 0 deletions dockerfiles/debian-bullseye-all/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ RUN echo 'deb http://apt.postgresql.org/pub/repos/apt/ bullseye-pgdg main' > /et
liblz4-1 \
libzstd1 \
libzstd-dev \
sudo \
&& rm -rf /var/lib/apt/lists/*

RUN pip3 install sphinx
Expand Down
11 changes: 10 additions & 1 deletion dockerfiles/debian-bullseye-all/scripts/fetch_and_build_deb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down
1 change: 1 addition & 0 deletions dockerfiles/debian-buster-all/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ RUN echo 'deb http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main' > /etc/
liblz4-1 \
libzstd1 \
libzstd-dev \
sudo \
&& rm -rf /var/lib/apt/lists/*

RUN pip3 install sphinx
Expand Down
11 changes: 10 additions & 1 deletion dockerfiles/debian-buster-all/scripts/fetch_and_build_deb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down
1 change: 1 addition & 0 deletions dockerfiles/debian-stretch-all/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ RUN echo 'deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main' > /etc
liblz4-1 \
libzstd1 \
libzstd-dev \
sudo \
&& rm -rf /var/lib/apt/lists/*

RUN pip3 install sphinx
Expand Down
11 changes: 10 additions & 1 deletion dockerfiles/debian-stretch-all/scripts/fetch_and_build_deb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
#!/bin/bash

# make bash behave
set -euo pipefail
set -eo pipefail
# In one branch we execute commands inside One Branch steps, since One Branch does not allow executing docker inside
# docker. Additionally, Onebranch needs containers not to close so we make it hang for OneBranch to be able to
# execute commands.
if [[ -z ${CONTAINER_BUILD_RUN_ENABLED} ]]; then
echo "INFO: Image working in waiting mode. Not executing build script"
tail -f /dev/null
exit 0
fi

IFS=$'\n\t'

# constants
Expand Down
Loading