Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an integration test for VNC + websockify #93

Closed
wants to merge 23 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
7aee660
Add an integration test for VNC + websockify
yuvipanda Feb 15, 2024
44137e0
Add missing packages to dev-requirements.txt
yuvipanda Feb 15, 2024
557135e
Show output as test is being run
yuvipanda Feb 15, 2024
fbf2fd1
Dynamically allocate forwarded port
yuvipanda Feb 15, 2024
00808aa
Use full names for all options
yuvipanda Feb 15, 2024
af61281
Explicitly wait for container to be ready
yuvipanda Feb 15, 2024
cbe3e3d
Dynamically allocate websocat port
yuvipanda Feb 15, 2024
66809be
Quieten wget
yuvipanda Feb 15, 2024
e8b864a
Cleanup github workflow file
yuvipanda Feb 15, 2024
d13d5bd
Don't buffer py.test output
yuvipanda Feb 15, 2024
6e7df3e
Put screenshot in a temporary directory
yuvipanda Feb 15, 2024
76b801e
Add a couple more debug log items
yuvipanda Feb 15, 2024
dea42a5
Provide container logs in the integration test
yuvipanda Feb 15, 2024
3d44feb
Don't pass "-it" to docker run
yuvipanda Feb 15, 2024
e8de6d4
Wait longer for capturing screenshot
yuvipanda Feb 15, 2024
b8710d2
Document why we are waiting longer here
yuvipanda Feb 15, 2024
25bee53
Debug log about attempting to capture screenshot correctly
yuvipanda Feb 15, 2024
5054205
Show full traceback when pytest ends
yuvipanda Feb 15, 2024
adec7b2
Add tests for JS & HTML
yuvipanda Feb 15, 2024
86c6cdf
Try increasing shm size on docker run
yuvipanda Mar 25, 2024
8ea28eb
Revert "Try increasing shm size on docker run"
yuvipanda Mar 26, 2024
c285c0b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 13, 2024
7a59e95
Provide tty for docker run command
yuvipanda Jun 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 11 additions & 121 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# This is a GitHub workflow defining a set of jobs with a set of steps.
# ref: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions
#
name: Test
name: Integration Tests

on:
pull_request:
Expand Down Expand Up @@ -36,129 +36,19 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Build image
run: |
docker build --progress=plain --build-arg vncserver=${{ matrix.vncserver }} -t test .

- name: (inside container) websockify --help
run: |
docker run test websockify --help
- uses: actions/setup-python@v5
with:
python-version: "3.10"

- name: (inside container) vncserver -help
run: |
# -help flag is not available for TurboVNC, but it emits the -help
# equivalent information anyhow if passed -help, but also errors. Due
# to this, we fallback to use the errorcode of vncsrever -list.
docker run test bash -c "vncserver -help || vncserver -list > /dev/null"

- name: Install websocat, a test dependency"
- name: Install testing requirements
run: |
pip install -r dev-requirements.txt
# Install websocat, needed for integration tests
wget -q https://github.com/vi/websocat/releases/download/v1.12.0/websocat.x86_64-unknown-linux-musl \
-O /usr/local/bin/websocat
-O /usr/local/bin/websocat
chmod +x /usr/local/bin/websocat

- name: Test vncserver
if: always()
- name: Run Integration tests
run: |
container_id=$(docker run -d -it -p 5901:5901 test vncserver -xstartup /opt/install/jupyter_remote_desktop_proxy/share/xstartup -verbose -fg -geometry 1680x1050 -SecurityTypes None -rfbport 5901)
sleep 1

echo "::group::Install netcat, a test dependency"
docker exec --user root $container_id bash -c '
apt update
apt install -y netcat
'
echo "::endgroup::"

docker exec -it $container_id timeout --preserve-status 1 nc -v localhost 5901 2>&1 | tee -a /dev/stderr | \
grep --quiet RFB && echo "Passed test" || { echo "Failed test" && TEST_OK=false; }

echo "::group::vncserver logs"
docker exec $container_id bash -c 'cat ~/.vnc/*.log'
echo "::endgroup::"

docker stop $container_id > /dev/null
if [ "$TEST_OK" == "false" ]; then
echo "One or more tests failed!"
exit 1
fi

- name: Test websockify'ed vncserver
if: always()
run: |
container_id=$(docker run -d -it -p 5901:5901 test websockify --verbose --log-file=/tmp/websockify.log --heartbeat=30 5901 -- vncserver -xstartup /opt/install/jupyter_remote_desktop_proxy/share/xstartup -verbose -fg -geometry 1680x1050 -SecurityTypes None -rfbport 5901)
sleep 1

echo "::group::Install websocat, a test dependency"
docker exec --user root $container_id bash -c '
wget -q https://github.com/vi/websocat/releases/download/v1.12.0/websocat.x86_64-unknown-linux-musl \
-O /usr/local/bin/websocat
chmod +x /usr/local/bin/websocat
'
echo "::endgroup::"

docker exec -it $container_id websocat --binary --one-message --exit-on-eof "ws://localhost:5901/" 2>&1 | tee -a /dev/stderr | \
grep --quiet RFB && echo "Passed test" || { echo "Failed test" && TEST_OK=false; }

echo "::group::websockify logs"
docker exec $container_id bash -c "cat /tmp/websockify.log"
echo "::endgroup::"

echo "::group::vncserver logs"
docker exec $container_id bash -c 'cat ~/.vnc/*.log'
echo "::endgroup::"

docker stop $container_id > /dev/null
if [ "$TEST_OK" == "false" ]; then
echo "One or more tests failed!"
exit 1
fi

- name: Test project's proxy to websockify'ed vncserver
if: always()
run: |
container_id=$(docker run -d -it -p 8888:8888 -e JUPYTER_TOKEN=secret test)
sleep 3

curl --silent --fail 'http://localhost:8888/desktop/?token=secret' | grep --quiet 'Jupyter Remote Desktop Proxy' && echo "Passed get index.html test" || { echo "Failed get index.html test" && TEST_OK=false; }
curl --silent --fail 'http://localhost:8888/desktop/static/dist/viewer.js?token=secret' > /dev/null && echo "Passed get viewer.js test" || { echo "Failed get viewer.js test" && TEST_OK=false; }

# The first attempt often fails, but the second always(?) succeeds.
#
# This could be related to jupyter-server-proxy's issue
# https://github.com/jupyterhub/jupyter-server-proxy/issues/459
# because the client/proxy websocket handshake completes before the
# proxy/server handshake. This issue is tracked for this project by
# https://github.com/jupyterhub/jupyter-remote-desktop-proxy/issues/105.
#
websocat --binary --one-message --exit-on-eof 'ws://localhost:8888/desktop-websockify/?token=secret' 2>&1 \
| tee -a /dev/stderr \
| grep --quiet RFB \
&& echo "Passed initial websocket test" \
|| { \
echo "Failed initial websocket test" \
&& sleep 1 \
&& websocat --binary --one-message --exit-on-eof 'ws://localhost:8888/desktop-websockify/?token=secret' 2>&1 \
| tee -a /dev/stderr \
| grep --quiet RFB \
&& echo "Passed second websocket test" \
|| { echo "Failed second websocket test" && TEST_OK=false; } \
}

echo "::group::jupyter_server logs"
docker logs $container_id
echo "::endgroup::"

echo "::group::vncserver logs"
docker exec $container_id bash -c 'cat ~/.vnc/*.log'
echo "::endgroup::"

timeout 5 docker stop $container_id > /dev/null && echo "Passed SIGTERM test" || { echo "Failed SIGTERM test" && TEST_OK=false; }

if [ "$TEST_OK" == "false" ]; then
echo "One or more tests failed!"
exit 1
fi

# TODO: Check VNC desktop works, e.g. by comparing Playwright screenshots
# https://playwright.dev/docs/test-snapshots
export PYTHONUNBUFFERED=1
py.test integration-tests/ -s --full-trace
4 changes: 4 additions & 0 deletions dev-requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
pytest
pytest-asyncio # Used for unused_tcp_port fixture
pytest-image-diff
vncdotool
Binary file added integration-tests/expected.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
150 changes: 150 additions & 0 deletions integration-tests/test_vnc.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
"""
Test the VNC server is serving images
"""

import json
import secrets
import subprocess
import tempfile
import time
from pathlib import Path
from urllib.request import urlopen

import pytest
from vncdotool import api

REPO_ROOT = Path(__file__).parent.parent


# Rebuild once every test session if needed, relying on docker cache
# to keep that fast
@pytest.fixture(scope="session")
def container_image() -> str:
"""
Provide a built container image name
"""
# Use a different tag name here each time, relying on docker cache to
# keep things fast and never having to deal with stale image problems
image_name = f"jupyter-remote-desktop-proxy-integration-test:{secrets.token_hex(8)}"
cmd = ["docker", "build", "-t", image_name, str(REPO_ROOT)]
subprocess.check_call(cmd)
return image_name


@pytest.fixture
def container(container_image) -> tuple[str, str]:
"""
Provide a running container with jupyter server running

Returns a tuple of (port, token), where port is the *local* port
that is forwarded to the jupyter server port inside the docker container,
and token is the authentication token to be used when talking to the
remote container.
"""
token = secrets.token_hex(16)
container_name = f"remote-desktop-proxy-integration-test-{secrets.token_hex(4)}"
cmd = [
"docker",
"run",
"--publish-all",
"--rm",
"-it",
"--name",
container_name,
"--security-opt",
"seccomp=unconfined",
"--security-opt",
"apparmor=unconfined",
container_image,
"jupyter",
"server",
f"--IdentityProvider.token={token}",
]
proc = subprocess.Popen(cmd)

print("Waiting for container to come online...")
# Try 5 times, with a 2s wait in between
for current_try in range(5):
time.sleep(5)
try:
container_info = json.loads(
subprocess.check_output(
['docker', 'container', 'inspect', container_name]
).decode()
)
except subprocess.CalledProcessError as e:
print(f"Container not ready yet, inspect returned {e.returncode}")
time.sleep(2)
continue

container_health = container_info[0]["State"]["Health"]["Status"]
if container_health == "healthy":
break

print(f"Current container health status: {container_health}")
time.sleep(2)
else:
raise TimeoutError("Could not start docker container in time")

exposed_port = container_info[0]["NetworkSettings"]["Ports"]["8888/tcp"][0]
origin = f"{exposed_port['HostIp']}:{exposed_port['HostPort']}"

print(f"Container started at {origin}")
try:
yield (origin, token)
finally:
proc.kill()
proc.wait()
subprocess.check_call(['docker', 'container', 'stop', container_name])


def test_vnc_screenshot(container, image_diff, unused_tcp_port):
origin, token = container
websocat_proc = subprocess.Popen(
[
'websocat',
'--binary',
'--exit-on-eof',
f'tcp-l:127.0.0.1:{unused_tcp_port}',
f'ws://{origin}/desktop-websockify/?token={token}',
]
)
print(f"websocat proxying 127.0.0.1:{unused_tcp_port} to VNC server")
try:
# :: is used to indicate port, as that is what VNC expects.
# A single : is used to indicate display number. In our case, we
# do not use multiple displays so no need to specify that.
with api.connect(
f'127.0.0.1::{unused_tcp_port}'
) as client, tempfile.TemporaryDirectory() as d:
# Wait a bit for the desktop to fully render, as it is only started
# up when our connect call completes.
# FIXME: Repeatedly take a few screenshots here in a retry loop until
# a timeout or the images match
time.sleep(15)
screenshot_target = Path(d) / "screenshot.jpeg"
print("Connected to VNC server. Attempting to capture screenshot...")
client.captureScreen(str(screenshot_target))

# This asserts if the images are different, so test will fail
image_diff(REPO_ROOT / "integration-tests/expected.jpeg", screenshot_target)
finally:
# Explicitly shutdown vncdo, as otherwise a stray thread keeps
# running forever
api.shutdown()

websocat_proc.kill()
websocat_proc.wait()


def test_desktop_page(container):
origin, token = container
# Check if the rendered HTML file is returned
desktop_url = f'http://{origin}/desktop/?token={token}'
with urlopen(desktop_url) as f:
assert '<title>Jupyter Remote Desktop Proxy</title>' in f.read().decode()

# Check if built JS file is served
js_url = f'http://{origin}/desktop/static/dist/viewer.js?token={token}'
resp = urlopen(js_url)
assert resp.status == 200
Loading