Skip to content

Releases: alan-turing-institute/data-safe-haven

Release 5.3.1 (2025-01-28)

28 Jan 12:19
@jemrobinson jemrobinson
v5.3.1
4a237a9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.3.1 (2025-01-28) Latest
Latest

Release Highlights

  • Fixes issue with expiring SSL certificate
  • Updates Nexus image to fix an initialisation problem

Upgrading from 5.3.0

Run the following command to upgrade an existing SRE

dsh deploy sre YOURSRENAME

What's Changed

Full Changelog: v5.3.0...v5.3.1

Contributors

jemrobinson
Assets 2
Loading

Release 5.3.0 (2025-01-20)

20 Jan 14:20
@JimMadge JimMadge
v5.3.0
de563f8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.3.0 (2025-01-20)

Release Highlights

  • Adds/fixes support for Tier 0 and Tier 1 SREs
  • Adds a reference section for the command line interface to the documentation

Upgrading from 5.2.1

Run the following command to upgrade an existing SRE

dsh deploy sre YOURSRENAME

What's Changed

Full Changelog: v5.2.1...v5.3.0

Contributors

cptanalatriste, jemrobinson, and 3 other contributors
Loading

Release 5.2.1 (2025-01-13)

13 Jan 11:57
@jemrobinson jemrobinson
v5.2.1
bf91ec0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.2.1 (2025-01-13)

Release Highlights

  • Fixes guacamole-user-sync crash which was limiting SREs to a maximum of 10 users
  • Fixes problem with listing users when SRE and SHM are deployed to different subscriptions

Upgrading from 5.2.0

Run the following command to upgrade an existing SRE

dsh deploy sre YOURSRENAME

What's Changed

  • Guacamole user synchronisation problems by @jemrobinson in #2352
  • Retrieve SRE sub name and use that when connecting to guac database by @craddm in #2354

Full Changelog: v5.2.0...v5.2.1

Contributors

jemrobinson and craddm
Loading

Release 5.2.0 (2024-12-05)

05 Dec 14:04
@JimMadge JimMadge
v5.2.0
This tag was signed with the committer’s verified signature. The key has expired.
JimMadge Jim Madge
GPG key ID: 5B629C6D46746CBA
Expired
Verified
Learn about vigilant mode.
3dfa5ce
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.2.0 (2024-12-05)

Release Highlights

  • More logs collected in the log analytics workspace
    • Storage
      • Ingress and egress stores
      • Desired state files
      • Users' home directories
      • Container configuration and persistent state
    • Container services
    • Firewall
  • Better CLI feedback and error messages
  • Documentation improvements

Known issues

Backup is not functional. Following the notice in the documentation will not enable backup.

Upgrading from 5.1.0

In order to upgrade, you will need to carry out the following steps.

⚠️ Some manual interventions are needed. Please ensure that your data is appropriately backed-up before starting. ⚠️

Step-by-step upgrade instructions

N.B. throughout the instructions below, replace YOURSRENAME with the lower-case name of your SRE

Create an upgrade JSON file with the following contents

{
  "nameTable": {
    "sre_data_component": "urn:pulumi:shm-blue-sre-YOURSRENAME::data-safe-haven::dsh:sre:DataComponent::sre_data",
    "sre_desired_state_component": "urn:pulumi:shm-blue-sre-YOURSRENAME::data-safe-haven::dsh:sre:DesiredStateComponent::sre_desired_state"
  },
  "resources": [
    {
      "type": "dsh:sre:NFSV3StorageAccountComponent",
      "name": "sre_data_storage_account_data_private_sensitive",
      "component": true,
      "parent": "sre_data_component"
    },
    {
      "type": "dsh:sre:NFSV3StorageAccountComponent",
      "name": "sre_desired_state_storage_account",
      "component": true,
      "parent": "sre_desired_state_component"
    }
  ]
}

Apply the upgrade JSON as follows

dsh pulumi run YOURSRENAME 'import --file /full/path/to/your/upgrade.json --yes'
dsh pulumi run YOURSRENAME 'state unprotect --all'

Note that the first command might fail - the import should still have succeeded though.

Download the Pulumi state file

dsh pulumi run YOURSRENAME 'stack export --file /full/path/to/a/local/file.json'

Open the Pulumi state file in an editor and find-and-replace the following strings

From To
dsh:sre:DataComponent$azure-native:storage:StorageAccount::sre_data_storage_account_data_private_sensitive dsh:sre:DataComponent$dsh:sre:NFSV3StorageAccountComponent$azure-native:storage:StorageAccount::sre_data_storage_account_data_private_sensitive
dsh:sre:DataComponent$azure-native:storage:StorageAccount$azure-native:network:PrivateEndpoint::sre_data_storage_account_data_private_sensitive dsh:sre:DataComponent$dsh:sre:NFSV3StorageAccountComponent$azure-native:storage:StorageAccount$azure-native:network:PrivateEndpoint::sre_data_storage_account_data_private_sensitive
dsh:sre:DataComponent$azure-native:storage:StorageAccount$azure-native:storage:BlobContainer dsh:sre:DataComponent$dsh:sre:NFSV3StorageAccountComponent$azure-native:storage:StorageAccount$azure-native:storage:BlobContainer
dsh:sre:DataComponent$azure-native:storage:StorageAccount$pulumi-python:dynamic:Resource dsh:sre:DataComponent$dsh:sre:NFSV3StorageAccountComponent$azure-native:storage:StorageAccount$pulumi-python:dynamic:Resource
dsh:sre:DataComponent$azure-native:storage:StorageAccount$azure-native:network:PrivateDnsZoneGroup::sre_data_storage_account_data_private_sensitive dsh:sre:DataComponent$dsh:sre:NFSV3StorageAccountComponent$azure-native:storage:StorageAccount$azure-native:network:PrivateDnsZoneGroup::sre_data_storage_account_data_private_sensitive
dsh:sre:DesiredStateComponent$azure-native:storage:StorageAccount dsh:sre:DesiredStateComponent$dsh:sre:NFSV3StorageAccountComponent$azure-native:storage:StorageAccount

Upload the edited Pulumi state file

dsh pulumi run YOURSRENAME 'stack import --file /full/path/to/a/local/file.json'

Deploy using v5.2.0 which will complete the rest of the upgrade

dsh sre deploy ultramarine

What's Changed

Full Changelog: v5.1.0...v5.2.0

Contributors

craddm, JimMadge, and dependabot
Loading

Release 5.1.0 (2024-11-21)

21 Nov 15:17
@craddm craddm
v5.1.0
72711c5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.1.0 (2024-11-21)

Release Highlights

  • Logs from workspaces are now collected in a centralised log analytics workspace
  • Research user IP address fields in the SRE configuration can now be set to Internet, rather than a specific IP address
  • Bug fixes and documentation improvements

⚠️ Update requires manual intervention ⚠️

In order to update to 5.1.0, you will need to do the following:

N.B. throughout the instructions below, replace YOURSRENAME with the lower-case name of your SRE, and YOURSHMNAME with the lower-case name of your SHM. fqdn refers to the fully-qualified domain name from the SHM config, which can be checked using dsh config show-shm.

Delete the Hedgedoc, Identity, Gitea, and remote desktop container groups

The groups can be deleted via the portal or using Azure CLI.

In the portal, you will find the container groups in the SRE resource group, shm-YOURSHMNAME-sre-YOURSRENAME-rg. The name of the container groups follow the format shm-YOURSHMNAME-sre-YOURSRENAME-container-group-X, where X is the software within the group.

az container delete --name shm-YOURSHMNAME-sre-YOURSRENAME-container-group-hedgedoc --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg
az container delete --name shm-YOURSHMNAME-sre-YOURSRENAME-container-group-identity --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg
az container delete --name shm-YOURSHMNAME-sre-YOURSRENAME-container-group-gitea --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg
az container delete --name shm-YOURSHMNAME-sre-YOURSRENAME-container-group-remote-desktop --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg

Remove the DNS records for the deleted container groups

The CNAME and A records for the Hedgedoc, Identity, and Gitea resources need to be deleted from the public and private DNS zones.

This can be done in the portal, looking in the public DNS Zone for your SRE - YOURSRENAME.fqdn - for CNAME records, and the private DNS Zone - privatelink.YOURSRENAME.fqdn- for the A records.

Alternatively, use the Azure CLI, as below.

az network dns record-set cname delete --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg --zone `YOURSRENAME.fqdn` --name identity
az network dns record-set cname delete --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg --zone `YOURSRENAME.fqdn` --name gitea
az network dns record-set cname delete --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg --zone `YOURSRENAME.fqdn` --name hedgedoc

az network private-dns record-set a delete --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg --zone `privatelink.YOURSRENAME.fqdn` --name identity
az network private-dns record-set a delete --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg --zone `privatelink.YOURSRENAME.fqdn` --name gitea
az network private-dns record-set a delete --resource-group shm-YOURSHMNAME-sre-YOURSRENAME-rg --zone `privatelink.YOURSRENAME.fqdn` --name hedgedoc

Delete the manually deleted resources from the Pulumi state

Run the following DSH CLI commands, ensuring that you have replaced the placeholders with the appropriate SHM and SRE names.

dsh pulumi run YOURSRENAME 'state delete urn:pulumi:shm-YOURSHMNAME-sre-YOURSRENAME::data-safe-haven::dsh:sre:IdentityComponent$pulumi-python:dynamic:Resource::sre_identity_entra_application --target-dependents'
dsh pulumi run YOURSRENAME 'state delete urn:pulumi:shm-YOURSHMNAME-sre-YOURSRENAME::data-safe-haven::dsh:sre:RemoteDesktopComponent$pulumi-python:dynamic:Resource::sre_remote_desktop_entra_application --target-dependents'

N.B. The $ character in the URN above may need to be escaped appropriately for your operating system. As written above, the command will work appropriately on Unix-based systems.

Delete pulumi_vars.yaml from blob storage

The pulumi_vars.yaml file needs to be deleted from blob storage. In the portal, navigate to the storage account for the desired state in the SRE's resource group. Open the desiredstate blob container through the storage browser. In the vars folder, delete the file pulumi_vars.yaml.

Delete the Entra groups and applications

Delete the Microsoft Entra groups and applications previously created by dsh.
These are now managed by Pulumi, which will not be able to run correctly if resources with identical names already exist.

The groups to be deleted are:

Data Safe Haven SRE YOURSRENAME Administrators
Data Safe Haven SRE YOURSRENAME Privileged Users
Data Safe Haven SRE YOURSRENAME Users

The applications to be deleted are:

Data Safe Haven (YOURSHMNAME) Service Principal
sre-YOURSRENAME-guacamole
sre-YOURSRENAME-apricot

Redeploy the SHM and SRE

Finally, redeploy the SHM and SRE from release 5.1.0

dsh shm deploy
dsh sre deploy YOURSRENAME

SRE config files

The method of sanitising SRE names when creating remote configuration files has changed. Previously, hyphens or underscores in the SRE name were removed from the name used for the remote configuration file. If you have an SRE with a hyphen or underscore, you should download the configuration file before upgrading to v5.1.0. Upload the configuration again once you have upgraded to v5.1.0.

What's Changed

New Contributors

Full Changelog: v5.0.1...v5.1.0

Contributors

cptanalatriste, jemrobinson, and 3 other contributors
Loading

Release 5.0.1 (2024-10-24)

24 Oct 12:30
@JimMadge JimMadge
v5.0.1
This tag was signed with the committer’s verified signature. The key has expired.
JimMadge Jim Madge
GPG key ID: 5B629C6D46746CBA
Expired
Verified
Learn about vigilant mode.
2e915ef
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.0.1 (2024-10-24)

Release Highlights

  • Bug fixes
  • Support for deployment of SREs to different subscriptions from their SHM
  • Enhanced user experience and documentation

⚠️ Update Requires Manual Intervention ⚠️

In order to upgrade from v5.0.0 you will need to delete the Microsoft Entra groups and applications previously created by dsh.
These are now managed by Pulumi, which will not be able to run correctly if resources with identical names already exist

You will also need to rerun the dsh shm deploy command, as some resources have been added to the SHM.

What's Changed

Full Changelog: v5.0.0...v5.0.1

Contributors

jemrobinson, craddm, and 3 other contributors
Loading

Release 5.0.0 (2024-08-20)

20 Aug 15:20
@JimMadge JimMadge
v5.0.0
This tag was signed with the committer’s verified signature. The key has expired.
JimMadge Jim Madge
GPG key ID: 5B629C6D46746CBA
Expired
Verified
Learn about vigilant mode.
3003ca4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.0.0 (2024-08-20)

Upgrading

This is a major release and it not compatible with any previous versions.
To use this version you must start a new TRE deployment.

Changes

  • Complete rewrite of code in Python using IAC and configuration management tools Pulumi and Ansible

What's Changed

Read more

Contributors

jemrobinson, edwardchalstrey1, and 6 other contributors
Loading

Release 5.0.0rc2 (2024-06-26)

26 Jul 13:54
@JimMadge JimMadge
v5.0.0-rc2
This tag was signed with the committer’s verified signature. The key has expired.
JimMadge Jim Madge
GPG key ID: 5B629C6D46746CBA
Expired
Verified
Learn about vigilant mode.
996f54f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 5.0.0rc2 (2024-06-26) Pre-release
Pre-release

Known Issues

⚠️ This release is not ready for production usage. ⚠️

  • ClamAV not configured
  • Unstable container service IP addresses
  • Lacking Nvidia utils

What's Changed

  • Use pip-compile for package resolution by @jemrobinson in #1514
  • Add pip-tools to NON_IMPORTABLE_PACKAGES by @edwardchalstrey1 in #1537
  • Add May 2023 DSG to versioning by @jemrobinson in #1545
  • Release v4.1.0 cloud init changes by @edwardchalstrey1 in #1548
  • Update SRD package versions by @github-actions in #1578
  • Update PyPI and CRAN allow lists by @github-actions in #1579
  • Fix deployment issues with MSSQL and PyPi mirrors by @craddm in #1582
  • Update PyPI and CRAN allow lists by @github-actions in #1588
  • Update SRD package versions by @github-actions in #1587
  • Updates for Release v4.1.0 by @craddm in #1590
  • Release v4.1.0 by @craddm in #1586
  • Remove CoCalc by @craddm in #1554
  • Merge 'latest' into 'develop' by @craddm in #1593
  • Add script to automate account deletion by @edwardchalstrey1 in #1508
  • Add @craddm to CODEOWNERS by @jemrobinson in #1594
  • Update PyPI and CRAN allow lists by @github-actions in #1595
  • Remove pulumi testing files from develop branch by @craddm in #1597
  • Update PyPI and CRAN allow lists by @github-actions in #1601
  • Update SRD package versions by @github-actions in #1616
  • Update SRD package versions by @github-actions in #1622
  • Bump urllib3 from 2.0.2 to 2.0.6 in /docs by @dependabot in #1625
  • Improve Pulumi error messages by @craddm in #1624
  • Update PyPI and CRAN allow lists by @github-actions in #1627
  • Update PyPI and CRAN allow lists by @github-actions in #1631
  • Update SRD package versions by @github-actions in #1630
  • Improve Python documentation by @jemrobinson in #1635
  • Use Pulumi random provider by @jemrobinson in #1629
  • Pulumi: Fix selectors not updating by @JimMadge in #1621
  • Bump urllib3 from 2.0.6 to 2.0.7 in /docs by @dependabot in #1647
  • Remove hyphens from SHM and SRE names by @craddm in #1650
  • Update PyPI and CRAN allow lists by @github-actions in #1646
  • Update SRD package versions by @github-actions in #1652
  • Pulumi: Improve login flow by @JimMadge in #1617
  • Update PyPI and CRAN allow lists by @github-actions in #1654
  • Add all contributors table and instructions for how to update by @edwardchalstrey1 in #1649
  • Update PyPI and CRAN allow lists by @github-actions in #1656
  • Update PyPI and CRAN allow lists by @github-actions in #1668
  • Update SRD package versions by @github-actions in #1669
  • Update devcontainer configuration by @craddm in #1662
  • Update outdated parameters that cause breaking change warnings by @craddm in #1663
  • Change default lun from lun1 to lun0 by @craddm in #1667
  • Add context command by @JimMadge in #1655
  • Pulumi: Update dependencies, enable pinning by @JimMadge in #1660
  • Remove unneeded opening bracket in SRE network configuration script by @craddm in #1670
  • Update PyPI and CRAN allow lists by @github-actions in #1671
  • Use memory for the /tmp directory by @craddm in #1672
  • Factor out storage creation from SHM scripts by @craddm in #1673
  • Add missing import for logging module by @JimMadge in #1681
  • Update PyPI and CRAN allow lists by @github-actions in #1682
  • Update help text for Powershell command shmId andsreId arguments by @craddm in #1683
  • Update contributors by @JimMadge in #1684
  • Document removal of persistent SRE storage accounts by @craddm in #1685
  • docs: update @helendduncan as a contributor by @JimMadge in #1686
  • Update PyPI and CRAN allow lists by @github-actions in #1688
  • Update SRD package versions by @github-actions in #1692
  • Update PyPI and CRAN allow lists by @github-actions in #1693
  • Update PyPI and CRAN allow lists by @github-actions in #1694
  • Update DBeaver drivers using Github workflow by @craddm in #1696
  • Update SRD package versions by @github-actions in #1698
  • Bump jinja2 from 3.1.2 to 3.1.3 in /docs by @dependabot in #1700
  • Update SRD package versions by @github-actions in #1701
  • Update PyPI and CRAN allow lists by @github-actions in #1702
  • Update PyPI and CRAN allow lists by @github-actions in #1703
  • Handle no selected context by @JimMadge in #1691
  • Add basic config commands by @JimMadge in #1674
  • Fixing DBeaver driver issues on T2+ SREs by @craddm in #1704
  • Use Pydantic for validation and serialisation by @JimMadge in #1661
  • Improve handling of spaces in file paths by @craddm in #1705
  • Update PyPI and CRAN allow lists by @github-actions in #1706
  • Create pulumi container by @jemrobinson in #1711
  • Fix private link scope by @jemrobinson in #1713
  • Improve handling of SRE names by @JimMadge in #1699
  • Apply changes from updated black version by @jemrobinson in #1718
  • Bump black version by @JimMadge in #1719
  • Fix some issues with context handling at deployment time by @jemrobinson in #1716
  • Update SRD package versions by @github-actions in #1723
  • Correct file path for clamonacc service by @craddm in #1725
  • Add additional multiple data provider guidance to docs by @craddm in #1707
  • Update SRD package versions by @github-actions in #1727
  • Fix PostgreSQL permiss...
Read more

Contributors

jemrobinson, edwardchalstrey1, and 6 other contributors
Loading

Release 4.2.2 (2024-07-15)

15 Jul 14:53
@JimMadge JimMadge
v4.2.2
008d346
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 4.2.2 (2024-07-15)

⚠️ Update Requires Manual Intervention ⚠️

If you are using a 4.2.x SHM and want to upgrade to 4.2.2, please follow the steps below:

For the SHM:

  1. Add a docker section to your SHM config with a username and personal access token (following the SHM deployment instructions)
  2. Re-run Setup_SHM_Networking.ps1 -shmId {shm} from deployment/safe_haven_management/setup

For any SRE that you deployed using an earlier 4.2.x version:

  1. Delete the GUACAMOLE-SRE-{sreId} VM and associated resources from the
    RG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP resource group
  2. Re-run the deployment script Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before} from deployment/secure_research_environment/setup

Known issues

  • As for 4.2.0, 4.2.1

Bug Fixes

  • Workaround for an issue where Let's Encrypt refused to provide certificates for uppercase FQDNs #1938
  • Fix for change in Azure supported public IP address SKU for VPNs, which prevented deployment of the virtual network gateway for accessing domain controllers #1947
  • Require supply of Docker Hub credentials to work round change in Docker download rate limits #1994
  • Update approved IP address list for Ubuntu apt repositories
  • Update to backup policy rules for Blob storage #1988

Full Changelog: v4.2.1...v4.2.2

Loading

Release 4.2.1 (2024-05-31)

31 May 14:42
@JimMadge JimMadge
v4.2.1
bee9fc4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Release 4.2.1 (2024-05-31)

⚠️ Update Requires Manual Intervention ⚠️

If you are using a 4.2.0 SHM and want to upgrade to 4.2.1, please follow the steps below:

  1. Delete the GUACAMOLE-SRE-{sreId} VM and associated resources from the RG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP resource group
  2. Re-run the deployment script Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before} from deployment/secure_research_environment/setup

Known issues

  • As for 4.2.0

Bug Fixes

Full Changelog: v4.2.0...v4.2.1

Loading