Merge branch 'test-manager-fixes-misc'

Author: MarkusPettersson98

.github/workflows/desktop-e2e.yml

@@ -151,7 +151,7 @@ jobs:
         shell: bash -ieo pipefail {0}
         run: |
           git fetch --tags --force
-          ./test/ci-runtests.sh ${{ matrix.os }}
+          ./test/scripts/ci-runtests.sh ${{ matrix.os }}
       - uses: actions/upload-artifact@v3
         if: '!cancelled()'
         with:
@@ -226,7 +226,7 @@ jobs:
         shell: bash -ieo pipefail {0}
         run: |
           git fetch --tags --force
-          ./test/ci-runtests.sh ${{ matrix.os }}
+          ./test/scripts/ci-runtests.sh ${{ matrix.os }}
       - uses: actions/upload-artifact@v3
         if: '!cancelled()'
         with:
@@ -297,7 +297,7 @@ jobs:
         shell: bash -ieo pipefail {0}
         run: |
           git fetch --tags --force
-          ./test/ci-runtests.sh ${{ matrix.os }}
+          ./test/scripts/ci-runtests.sh ${{ matrix.os }}
       - uses: actions/upload-artifact@v3
         if: '!cancelled()'
         with:

test/README.md

@@ -1,31 +1,34 @@
-# Project structure
+# Mullvad VPN end to end test framework
 
-## test-manager
+## Project structure
+
+### test-manager
 
 The client part of the testing environment. This program runs on the host and connects over a
 virtual serial port to the `test-runner`.
 
 The tests themselves are defined in this package, using the interface provided by `test-runner`.
 
-## test-runner
+### test-runner
 
 The server part of the testing environment. This program runs in guest VMs and provides the
 `test-manager` with the building blocks (RPCs) needed to create tests.
 
-## test-rpc
+### test-rpc
 
 A support library for the other two packages. Defines an RPC interface, transports, shared types,
 etc.
 
-# Prerequisities
+
+## Prerequisites
 
 For macOS, the host machine must be macOS. All other platforms assume that the host is Linux.
 
-## All platforms
+### All platforms
 
 * Get the latest stable Rust from https://rustup.rs/.
 
-## macOS
+### macOS
 
 Normally, you would use Tart here. It can be installed with Homebrew. You'll also need
 `wireguard-tools`, a protobuf compiler, and OpenSSL:
@@ -34,7 +37,7 @@ Normally, you would use Tart here. It can be installed with Homebrew. You'll als
 brew install cirruslabs/cli/tart wireguard-tools pkg-config openssl protobuf
 ```
 
-### Wireshark
+#### Wireshark
 
 Wireshark is also required. More specifically, you'll need `wireshark-chmodbpf`, which can be found
 in the Wireshark installer here: https://www.wireshark.org/download.html
@@ -47,7 +50,7 @@ dseditgroup -o edit -a THISUSER -t user access_bpf
 
 This lets us monitor traffic on network interfaces without root access.
 
-## Linux
+### Linux
 
 For running tests on Linux and Windows guests, you will need these tools and libraries:
 
@@ -58,94 +61,109 @@ dnf install git gcc protobuf-devel libpcap-devel qemu \
     wireguard-tools
 ```
 
-# Building the test runner
+## Setting up testing environment
+
+First you need to build the images for running tests on, see [`BUILD_OS_IMAGE.md`](./docs/BUILD_OS_IMAGE.md). The `test-manager` then needs to be configured to use the image.
+
+Here is an example of how to create a new OS configuration for Linux and macOS:
+
+### Linux
+
+```bash
+# Create or edit configuration
+# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md
+cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \
+    --package-type deb --architecture x64 \
+    --provisioner ssh --ssh-user test --ssh-password test
+
+# Try it out to see if it works - you should reach the VM's graphical desktop environment
+cargo run --bin test-manager run-vm debian11
+```
+
+### macOS
+
+
+```bash
+# Download some VM image
+tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base
+
+# Create or edit configuration
+# Use SSH to deploy the test runner since the image doesn't contain a runner
+cargo run --bin test-manager set macos-ventura tart ventura-base macos \
+    --architecture aarch64 \
+    --provisioner ssh --ssh-user admin --ssh-password admin
+
+# Try it out to see if it works
+cargo run -p test-manager run-vm macos-ventura
+```
+
+## Testing the app
+
+To automatically download and test a pre-built version of the app, use the `test-by-version.sh` script, see `test-by-version.sh --help` for instructions.
+
+To manually invoke `test-manager`, start by checking out the desired git version of this repo. Next, [build the app](../BuildInstructions.md) for the target platform then build the GUI test binary using `$(cd ../gui && npm run build-test-executable)`. The newly built packages will be located in the `../dist` folder by default.
+
+Next: build the `test-runner`
+
+### Building the test runner
 
 Building the `test-runner` binary is done with the `build-runner.sh` script.
 Currently, only `x86_64` platforms are supported for Windows/Linux and `ARM64` (Apple Silicon) for macOS.
 
 For example, building `test-runner` for Windows would look like this:
 
 ``` bash
-./container-run.sh ./build-runner.sh windows
+./scripts/container-run.sh ./scripts/build-runner.sh windows
 ```
 
-## Linux
-Using `podman` is the recommended way to build the `test-runner`. See the [Linux section under Prerequisities](#Prerequisities) for more details.
+#### Linux
+Using `podman` is the recommended way to build the `test-runner`. See the [Linux section under Prerequisities](#prerequisites) for more details.
 
 ``` bash
-./container-run.sh ./build-runner.sh linux
+./scripts/container-run.sh ./scripts/build-runner.sh linux
 ```
 
-## macOS
+#### macOS
 
 ``` bash
-./build-runner.sh macos
+./scripts/build-runner.sh macos
 ```
 
-## Windows
+#### Windows
 The `test-runner` binary for Windows may be cross-compiled from a Linux host.
 
 ``` bash
-./container-run.sh ./build-runner.sh windows
+./scripts/container-run.sh ./scripts/build-runner.sh windows
 ```
 
-# Building base images
-
-See [`BUILD_OS_IMAGE.md`](./docs/BUILD_OS_IMAGE.md) for how to build images for running tests on.
-
-# Running tests
+### Running the tests
 
-See `cargo run --bin test-manager` for details.
+After configuring the VM image using `test-manager set` and building the required packages (see [previous step](#setting-up-testing-environment)), `test-manager run-tests` is used to launch the tests. See `cargo run --bin test-manager -- run-tests --help` for details.
 
-## Linux
+Here is an example of how to run all tests using the Linux/macOS VM we set up earlier:
 
-Here is an example of how to create a new OS configuration and then run all tests:
+#### Linux
 
 ```bash
-# Create or edit configuration
-# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md
-cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \
-    --package-type deb --architecture x64 \
-    --provisioner ssh --ssh-user test --ssh-password test
-
-# Try it out to see if it works - you should reach the VM's graphical desktop environment
-cargo run --bin test-manager run-vm debian11
-
 # Run all tests
-cargo run --bin test-manager run-tests debian11 \
+cargo run --bin test-manager run-tests --vm debian11 \
     --display \
     --account 0123456789 \
     --app-package <git hash or tag> \
     --app-package-to-upgrade-from 2023.2
 ```
 
-## macOS
-
-Here is an example of how to create a new OS configuration (on Apple Silicon) and then run all
-tests:
+#### macOS
 
 ```bash
-# Download some VM image
-tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base
-
-# Create or edit configuration
-# Use SSH to deploy the test runner since the image doesn't contain a runner
-cargo run --bin test-manager set macos-ventura tart ventura-base macos \
-    --architecture aarch64 \
-    --provisioner ssh --ssh-user admin --ssh-password admin
-
-# Try it out to see if it works
-#cargo run -p test-manager run-vm macos-ventura
-
 # Run all tests
-cargo run --bin test-manager run-tests macos-ventura \
+cargo run --bin test-manager run-tests --vm macos-ventura \
     --display \
     --account 0123456789 \
     --app-package <git hash or tag> \
     --app-package-to-upgrade-from 2023.2
 ```
 
-## Note on `ci-runtests.sh`
+## Note on `scripts/ci-runtests.sh`
 
-Account tokens are read (newline-delimited) from the path specified by the environment variable
-`ACCOUNT_TOKENS`. Round robin is used to select an account for each VM.
+`scripts/ci-runtests.sh` is the script that GitHub actions uses to invokes the `test-manager`, with similar functionality as `test-by-version.sh`. Note that account tokens are read (newline-delimited) from the path specified by the environment variable `ACCOUNT_TOKENS`. Round robin is used to select an account for each VM.