Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace the_od_bods data on pipeline run #3

Merged
merged 5 commits into from
Jan 26, 2023
Merged

Replace the_od_bods data on pipeline run #3

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2281c10
added commit action for /data in the_od_bods and reduced pipeline to …
KarenJewell Jan 7, 2023
5bd6457
attempt 2 with tokens
KarenJewell Jan 7, 2023
31e780a
restored full pipeline
KarenJewell Jan 7, 2023
bdbe4ec
fix: add changes requested in review. Reduced code for testing.
KarenJewell Jan 26, 2023
1846bbb
restored full pipeline
KarenJewell Jan 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 21 additions & 13 deletions .github/workflows/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
required: true
default: true
type: boolean
commitToSite:
commitToRepos:
description: 'Commit to opendata.scot'
required: true
default: true
Expand All @@ -31,6 +31,7 @@ jobs:
with:
repository: 'OpenDataScotland/the_od_bods'
path: 'the_od_bods'
token: ${{ secrets.REPO_COMMIT_TOKEN }}
- name: Check out JKAN repo
uses: actions/checkout@v3
with:
Expand Down Expand Up @@ -72,12 +73,12 @@ jobs:
# USMART portals
- name: Run Script usmart.py
run: python ./usmart.py
working-directory: ${{env.odbodsPath}}
working-directory: ${{env.odbodsPath}}

- name: Run Script ckan.py
run: python ./ckan.py
working-directory: ${{env.odbodsPath}}

# ScotGov Statistics Sparkql
- name: Run Script sparkql_statistics.py
run: python ./sparkql_statistics.py
Expand All @@ -102,12 +103,12 @@ jobs:
- name: Run Script moray_council_scraper.py
run: python ./moray_council_scraper.py
working-directory: ${{env.odbodsPath}}/web-scrapers

# National Library of Scotland feeds
- name: Run Script nls_scraper.py
run: python ./nls_scraper.py
working-directory: ${{env.odbodsPath}}/web-scrapers

# Merge data
- name: Run Script merge_data.py
run: python ./merge_data.py
Expand All @@ -122,15 +123,22 @@ jobs:
- name: Git config
working-directory: ${{env.jkanPath}}
run: |
git config user.name "GitHub Actions"
git config user.email "<>"
- name: Git status
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
- name: Git status JKAN
run: git status
working-directory: ${{env.jkanPath}}
- name: Commit and push
if: ${{ github.event.inputs.commitToSite == 'true' || github.event.schedule == '0 12 * * FRI' }}
run: git add -A && git commit -m "Dataset Sync" && git push origin gh-pages
- name: Commit and push JKAN
if: ${{ github.event.inputs.commitToRepos == 'true' || github.event.schedule == '0 12 * * FRI' }}
run: git add -A && git commit -m 'Dataset Sync' && git push origin gh-pages
working-directory: ${{env.jkanPath}}
- name: Git status the_od_bods
run: git status
working-directory: ${{env.odbodsPath}}
- name: Commit and push the_od_bods
if: ${{ github.event.inputs.commitToRepos == 'true' || github.event.schedule == '0 12 * * FRI' }}
run: git add data && git commit -m "Dataset Sync" && git push origin main
working-directory: ${{env.odbodsPath}}

# Tell the Slack channel!
- name: Slack Notification
Expand All @@ -149,7 +157,7 @@ jobs:

# Debugging: artifact upload if the workflow fails at all or if we're not committing to the site (likely to debug)
- uses: actions/upload-artifact@v3
if: ${{ failure() || github.event.inputs.commitToSite == 'false' }}
if: ${{ failure() || github.event.inputs.commitToRepos == 'false' }}
with:
name: workspace
path: ${{ github.workspace }}
Expand Down