Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Link device QR code fails with 412 M_CONCURRENT_WRITE #18155

Open
waclaw66 opened this issue Feb 12, 2025 · 20 comments
Open

Link device QR code fails with 412 M_CONCURRENT_WRITE #18155

waclaw66 opened this issue Feb 12, 2025 · 20 comments

Comments

@waclaw66
Copy link

waclaw66 commented Feb 12, 2025

Description

I have set up MAS (with Authentik OIDC) on my Matrix instance, enabled MSC3861 and MSC4108 on Synapse according docs, everything looks good, login via Authentik works fine, however...
I want to add another Android device via QR code, when I scan QR code with Element X from Element Web following Element Web GET requests to
https://matrix.domain/_matrix/client/v3/devices/ZD6NzBC8RQ38jWLUIXBmQFPXB81etmMpbdD423%2F00BA fails with 404 {"errcode":"M_NOT_FOUND","error":"Not found"}
Not sure if this is expected.
After the 2-digit code from Element X is entered on Element Web folowing PUT request to https://matrix.domain/_synapse/client/rendezvous/01JKWWGPR4N9YXYG6RCCY38H0W fails with {"errcode":"M_UNKNOWN","error":"ETag does not match","org.matrix.msc4108.errcode":"M_CONCURRENT_WRITE"}
MAS request is confirmed and page closed, Element X stucks on that 2-digit code screen and fails after timeout same as Element Web.

What could be wrong there?
Are docs regarding worker settings up to date?
https://element-hq.github.io/synapse/develop/workers.html?highlight=workers#worker-configuration
Currently the /_synapse/client/rendezvous/{id} and /_matrix/client/v3/devices/{id} endpoints are handled by master worker instead of client worker.

Thanks.

EDIT: rolling back MAS, still unstable and broken...
Error: [{"message":"Database inconsistency on table compat_sessions column device_id row 01HQ3FB21QVAPX9E710P93212F","locations":[{"line":7,"column":7}],"path":["viewer","appSessions"]}]
while listing user account sessions

Steps to reproduce

above

Homeserver

Synapse

Synapse Version

1.124.0

Installation Method

Docker (matrixdotorg/synapse)

Database

Postgres 16

Workers

Multiple workers

Platform

Fedora 41

Configuration

No response

Relevant log output

13:44:06.512 => POST https://matrix.domain/_matrix/client/unstable/org.matrix.msc4108/rendezvous with  if-match: undefined init.js:1:1116008
13:44:06.524 Received etag: "47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU" init.js:1:1116008
13:44:06.525 negotiateProtocols(isNewDevice=false didScanCode=false) init.js:1:1116008
13:44:06.525 Waiting for LoginInitiateMessage init.js:1:1116008
13:44:06.525 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU" init.js:1:1116008
13:44:07.551 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU" init.js:1:1116008
13:44:08.636 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU" init.js:1:1116008
13:44:08.704 Received: pT0LeCYhLr0+QpBIVFI2Smj/FT5f5r6iLj4Or6mlBscqt44TNCWk9bOoAnKq|uZl/0zy3eR038JBJFOUdhRmZy+BPMyRR2HYAVM6nhXY with etag "I1u8Oo9y8tCOK8O8j1z6sr-wL98k3bPHUGifrWGitBc" init.js:1:1116008
13:44:08.705 LoginInitiateMessage received init.js:1:1116008
13:44:08.705 Sending LoginOkMessage init.js:1:1116008
13:44:08.705 => PUT https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 with RlbgviIvFPW5Iy+7zmi2s5GLslf2fsQIcPgYxIFI77+a10svpQI2 if-match: "I1u8Oo9y8tCOK8O8j1z6sr-wL98k3bPHUGifrWGitBc" init.js:1:1116008
13:44:08.735 Received etag: "hWCpMmGjUmMKPavmx9RV-CjhVvTFkyPDmT-ipM0v5X0" init.js:1:1116008
13:44:08.735 Waiting for protocol message init.js:1:1116008
13:44:08.735 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "hWCpMmGjUmMKPavmx9RV-CjhVvTFkyPDmT-ipM0v5X0" init.js:1:1116008
13:44:09.784 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "hWCpMmGjUmMKPavmx9RV-CjhVvTFkyPDmT-ipM0v5X0" init.js:1:1116008
13:44:09.874 Received: 4zLD45EBhAlFQXzR3qY/Kq67za99MzGlSNXB2TrQj1cVXJSY5s5/UKcpF/kn91Bmbo2mU+ejI+3qSCYD/CdPkpjE4yx8qH7GLrKIZ0ceAoXtHpTfe4a/RTO/n6PgmO4lFRbIlpUJNbH1In2TeMfkIm6Get4p0wi+t5g8x3bvG+pDATNUPFFvc3yiimYGsb1GbuctgOOPhYQXwEr9XkIxdozwGW6CH6v0TIY1JadMYr2eUUpkSGFtFyNtDrRZLETSEGhQIsIfNHRV7nitFdvxiFZ89Y2TTeVeFHA3silvsB/GvahO03dEbOLwOWWGmmgeUbRZ3iuxyDxeK5LdpvY1DhF7oNFyT95DRa0/hsBALLxngVaX/g4d81ysJbEytfPl6FUZOkbvdA with etag W/"Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:09.874 <= {"type": "m.login.protocol", ...} init.js:1:1116008
13:44:09.875 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/devices/qJfXY%2F7CFbXPd%2B38QoUoyCYFrPBLedlgNgRn0uwXCmU init.js:1:1116008
13:44:09.883 XHRGET
https://matrix.domain/_matrix/client/v3/devices/qJfXY/7CFbXPd+38QoUoyCYFrPBLedlgNgRn0uwXCmU
[HTTP/1.1 404 Not Found 50ms]

13:44:09.944 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/devices/qJfXY%2F7CFbXPd%2B38QoUoyCYFrPBLedlgNgRn0uwXCmU [70ms 404] init.js:1:1116008
13:44:10.242 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [6727ms 200] init.js:1:1116008
13:44:10.254 [Perf]: receiveSyncChanges took 10ms init.js:1:1116008
13:44:10.256 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:10.774 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [519ms 200] init.js:1:1116008
13:44:10.785 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:10.786 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:10.860 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [74ms 200] init.js:1:1116008
13:44:10.869 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:10.870 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:11.990 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1119ms 200] init.js:1:1116008
13:44:11.999 [Perf]: receiveSyncChanges took 8ms init.js:1:1116008
13:44:12.001 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:12.320 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [320ms 200] init.js:1:1116008
13:44:12.330 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:12.331 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:14.002 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1672ms 200] init.js:1:1116008
13:44:14.012 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:14.013 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:14.404 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [391ms 200] init.js:1:1116008
13:44:14.421 [Perf]: receiveSyncChanges took 12ms init.js:1:1116008
13:44:14.423 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:14.509 => {"type": "m.login.protocol_accepted", ...} init.js:1:1116008
13:44:14.510 => PUT https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 with 4wpFgyN+c4u1LuO27i4dDGOa2vXabfh2+nHHJqCwLbReTwQ6XQ0Bk9zeondtTQNXl1XdHw if-match: W/"Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:14.630 XHRPUT
https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4
[HTTP/1.1 412 Precondition Failed 4ms]

13:44:14.786 Received etag: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:14.787 Waiting for outcome message init.js:1:1116008
13:44:14.787 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:14.839 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [417ms 200] init.js:1:1116008
13:44:14.857 [Perf]: receiveSyncChanges took 13ms init.js:1:1116008
13:44:14.858 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:15.202 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [345ms 200] init.js:1:1116008
13:44:15.212 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:15.213 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:15.387 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [174ms 200] init.js:1:1116008
13:44:15.397 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:15.398 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:16.087 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:17.127 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:18.168 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:18.805 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [3407ms 200] init.js:1:1116008
13:44:18.818 [Perf]: receiveSyncChanges took 13ms init.js:1:1116008
13:44:18.820 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:19.220 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:20.185 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1366ms 200] init.js:1:1116008
13:44:20.196 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:20.197 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:20.334 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:20.787 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [589ms 200] init.js:1:1116008
13:44:20.796 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:20.797 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:20.936 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [140ms 200] init.js:1:1116008
13:44:20.945 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:20.946 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:21.206 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [259ms 200] init.js:1:1116008
13:44:21.217 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:44:21.218 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:21.360 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:22.072 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [853ms 200] init.js:1:1116008
13:44:22.085 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:44:22.089 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:22.398 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:23.340 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1251ms 200] init.js:1:1116008
13:44:23.349 [Perf]: receiveSyncChanges took 8ms init.js:1:1116008
13:44:23.352 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:23.450 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:23.795 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [442ms 200] init.js:1:1116008
13:44:23.807 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:44:23.808 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:24.461 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [652ms 200] init.js:1:1116008
13:44:24.474 [Perf]: receiveSyncChanges took 12ms init.js:1:1116008
13:44:24.475 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:24.475 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:25.512 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:26.551 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:27.405 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [2930ms 200] init.js:1:1116008
13:44:27.420 [Perf]: receiveSyncChanges took 13ms init.js:1:1116008
13:44:27.424 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:27.587 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:28.159 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [735ms 200] init.js:1:1116008
13:44:28.170 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:44:28.172 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:28.702 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:29.802 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:29.935 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1763ms 200] init.js:1:1116008
13:44:29.945 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:29.946 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:30.012 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [66ms 200] init.js:1:1116008
13:44:30.025 [Perf]: receiveSyncChanges took 12ms init.js:1:1116008
13:44:30.026 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:30.823 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:31.863 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:32.901 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:33.236 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [3209ms 200] init.js:1:1116008
13:44:33.253 [Perf]: receiveSyncChanges took 14ms init.js:1:1116008
13:44:33.256 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:34.019 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:34.359 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1102ms 200] init.js:1:1116008
13:44:34.374 [Perf]: receiveSyncChanges took 14ms init.js:1:1116008
13:44:34.376 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:35.045 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:36.157 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:37.257 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:38.178 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [3802ms 200] init.js:1:1116008
13:44:38.192 [Perf]: receiveSyncChanges took 13ms init.js:1:1116008
13:44:38.193 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:38.302 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:39.383 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:40.421 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:40.873 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [2680ms 200] init.js:1:1116008
13:44:40.887 [Perf]: receiveSyncChanges took 13ms init.js:1:1116008
13:44:40.889 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:41.352 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [463ms 200] init.js:1:1116008
13:44:41.361 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:41.362 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:41.447 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [84ms 200] init.js:1:1116008
13:44:41.457 [Perf]: receiveSyncChanges took 10ms init.js:1:1116008
13:44:41.459 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:41.463 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:42.006 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [548ms 200] init.js:1:1116008
13:44:42.016 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:42.017 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:42.262 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [245ms 200] init.js:1:1116008
13:44:42.273 [Perf]: receiveSyncChanges took 10ms init.js:1:1116008
13:44:42.275 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:42.544 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:43.355 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1079ms 200] init.js:1:1116008
13:44:43.365 [Perf]: receiveSyncChanges took 8ms init.js:1:1116008
13:44:43.366 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:43.628 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:44.670 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:45.326 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1960ms 200] init.js:1:1116008
13:44:45.341 [Perf]: receiveSyncChanges took 15ms init.js:1:1116008
13:44:45.342 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:45.693 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:46.597 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1255ms 200] init.js:1:1116008
13:44:46.609 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:44:46.611 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:46.689 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [79ms 200] init.js:1:1116008
13:44:46.699 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:46.700 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:46.721 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:47.612 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [912ms 200] init.js:1:1116008
13:44:47.626 [Perf]: receiveSyncChanges took 14ms init.js:1:1116008
13:44:47.627 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:47.786 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [158ms 200] init.js:1:1116008
13:44:47.796 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:47.797 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:47.807 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:48.890 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:49.366 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1570ms 200] init.js:1:1116008
13:44:49.381 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:49.382 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:49.930 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:51.046 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:52.097 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:52.365 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [2983ms 200] init.js:1:1116008
13:44:52.377 [Perf]: receiveSyncChanges took 12ms init.js:1:1116008
13:44:52.378 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:53.121 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:53.733 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1354ms 200] init.js:1:1116008
13:44:53.747 [Perf]: receiveSyncChanges took 13ms init.js:1:1116008
13:44:53.748 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:54.156 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:55.303 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:56.337 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:57.394 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:58.175 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [4428ms 200] init.js:1:1116008
13:44:58.188 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:44:58.189 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:58.289 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [100ms 200] init.js:1:1116008
13:44:58.298 [Perf]: receiveSyncChanges took 9ms init.js:1:1116008
13:44:58.299 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:58.436 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:44:58.526 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [226ms 200] init.js:1:1116008
13:44:58.537 [Perf]: receiveSyncChanges took 10ms init.js:1:1116008
13:44:58.538 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:58.763 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [224ms 200] init.js:1:1116008
13:44:58.775 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:44:58.776 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:44:59.477 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:45:00.241 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1466ms 200] init.js:1:1116008
13:45:00.254 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:45:00.255 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:45:00.515 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:45:00.811 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [556ms 200] init.js:1:1116008
13:45:00.823 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:45:00.824 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:45:01.537 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:45:02.569 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:45:02.812 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1989ms 200] init.js:1:1116008
13:45:02.824 [Perf]: receiveSyncChanges took 11ms init.js:1:1116008
13:45:02.825 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:45:03.606 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:45:04.628 FetchHttpApi: <-- GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx [1802ms 200] init.js:1:1116008
13:45:04.639 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:45:04.640 [Perf]: receiveSyncChanges took 12ms init.js:1:1116008
13:45:04.641 FetchHttpApi: --> GET https://matrix.domain/_matrix/client/v3/sync?filter=xxx&timeout=xxx&org.matrix.msc4222.use_state_after=xxx&set_presence=xxx&since=xxx init.js:1:1116008
13:45:05.680 => GET https://matrix.domain/_synapse/client/rendezvous/01JKX308VTEFKZS3T9AR1BAZB4 if-none-match: "Bqxcj7ycWGQIrksuP_7avb6py9_cPt1uS856yz0mjK8" init.js:1:1116008
13:45:06.005 Rendezvous failed: expired init.js:1:1116008

Anything else that would be useful to know?

No response

@waclaw66
Copy link
Author

waclaw66 commented Feb 18, 2025

@sandhose would you please check te log above, I don't see any problems with etag.
It would be helpful to see expected and actual etag within Synapse log.

@waclaw66
Copy link
Author

I've tested MAS with completely new Synapse instance without workers with the same result.

@waclaw66
Copy link
Author

waclaw66 commented Feb 20, 2025

Seems login via QR doesn't work now even on beta.element.io, it worked fine few days back. It fails after timeout on the 6-letter confirmation code screen on Element X.
However I have observed a small difference between my instance and beta.ems.host. The PUT request to https://matrix.domain/_synapse/client/rendezvous/{id} on my Element Web instance forwards header if-match prefixed with W/ and fails with 412, Element Web on beta.element.io forwards if-match header without that prefix and returns 202. That's the only difference. Etags seems to be correct, etag from if-match matches (except for the W/ prefix) etag returned by the previous GET request on both instances.
Version of Synapse/Element Web are the same, Element X 25.02.0, not sure about MAS - I'm using 0.14.0.

@sandhose
Copy link
Member

This feels like there is a caching proxy in front of Synapse, or something that compresses on the fly. Do you happen to have Cloudflare in front of Synapse? Or nginx with gzip on?

@sandhose
Copy link
Member

@waclaw66 Do you have a way to test this PR in your setup? #18178

This should solve the problem if you indeed have a proxy in between which compresses the responses on the fly

@waclaw66
Copy link
Author

@sandhose you are right, turning off gzip for Synapse on my nginx proxy did the trick, PUT forwards if-match without W/ prefix now and returns 202. Neverheless it fails afterwards on 6-letter confirmation code screen same as on beta.element.io, but that is another issue I suppose.
What endpoint causes this issue and should be forwarded without gzip compression?

@waclaw66
Copy link
Author

@waclaw66 Do you have a way to test this PR in your setup? #18178

I'm using docker image for Synapse, is there a way how test this PR on docker?

@sandhose
Copy link
Member

sandhose commented Feb 20, 2025

What endpoint causes this issue and should be forwarded without gzip compression?

It's /_matrix/client/unstable/org.matrix.msc4108/rendezvous and /_synapse/client/rendezvous.*

I'm using docker image for Synapse, is there a way how test this PR on docker?

You can build it locally with

docker build 'https://github.com/element-hq/synapse.git#quenting/rendezvous-headers' -f docker/Dockerfile -t my-custom-synapse-tag

That patch should make it unnecessary to exclude certain endpoint from compression

@waclaw66
Copy link
Author

waclaw66 commented Feb 20, 2025

I built docker image and deployed it to my test Synapse container (with gzip enabled), however it still returns 412 :(
I doublechecked that the right image is used, it is, version now returns {"server_version":"1.125.0rc1"}.

@sandhose
Copy link
Member

Neverheless it fails afterwards on 6-letter confirmation code screen same as on beta.element.io, but that is another issue I suppose.

I tried that, and it looks like this can happen when the local session doesn't have access to the cross-signing key or isn't connected to the key backup

I built docker image and deployed it to my test Synapse container, however it still returns 412 :(

Can you check in your request logs in EW? Right after you display the QR code, you should see:

  • a POST request to /_matrix/client/unstable/org.matrix.msc4108/rendezvous which returns a 201, with an ETag header with a certain value
  • a GET request right after to /_synapse/client/rendezvous/...., with:
    • '304 Not modified' as HTTP status code
    • If-None-Match: [The ETag value] in the response header
    • ETag with the same value in the response headers

@sandhose
Copy link
Member

Here's a script I have to make sure a rendezvous implementation works correctly:

# /// script
# requires-python = ">=3.12"
# dependencies = ["requests"]
# ///

import logging
from http.client import HTTPConnection

import requests

HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

base = "https://example.com" # CHANGE ME

response = requests.post(
    f"{base}/_matrix/client/unstable/org.matrix.msc4108/rendezvous",
    data="INIITIAL" * 128,
    headers={"Content-Type": "text/plain"},
)
assert response.status_code == 201, response.status_code

session = response.json()["url"]
assert "ETag" in response.headers
etag = response.headers["ETag"]

response = requests.get(session, headers={"If-None-Match": etag})
assert response.status_code == 304, response.status_code

response = requests.get(session)
assert response.status_code == 200, response.status_code
assert response.text == "INIITIAL" * 128
assert response.headers["ETag"] == etag

response = requests.put(
    session, data="UPDATED\n" * 128, headers={"If-Match": etag, "Content-Type": "text/plain"}
)
assert response.status_code == 202, response.status_code
assert "ETag" in response.headers
assert response.headers["ETag"] != etag

If you have uv installed you can just do uv run script.py. Else you'll need Python with requests installed

Change the 'base' with where Synapse is exposed

@waclaw66
Copy link
Author

Here's a script I have to make sure a rendezvous implementation works correctly:

# uv run check.py
Installed 5 packages in 7ms
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): matrix-test.domain:443
send: b'POST /_matrix/client/unstable/org.matrix.msc4108/rendezvous HTTP/1.1\r\nHost: matrix-test.domain\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: text/plain\r\nContent-Length: 1024\r\n\r\n'
send: b'INIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIAL'
reply: 'HTTP/1.1 201 Created\r\n'
header: Server: nginx/1.26.3
header: Date: Thu, 20 Feb 2025 08:56:06 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Access-Control-Allow-Origin: *
header: Access-Control-Expose-Headers: etag
header: Pragma: no-cache
header: Cache-Control: no-store, no-transform
header: ETag: "1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"
header: Expires: Thu, 20 Feb 2025 08:57:06 GMT
header: Last-Modified: Thu, 20 Feb 2025 08:56:06 GMT
DEBUG:urllib3.connectionpool:https://matrix-test.domain:443 "POST /_matrix/client/unstable/org.matrix.msc4108/rendezvous HTTP/1.1" 201 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): matrix-test.domain:443
send: b'GET /_synapse/client/rendezvous/01JMH94HZD117DP207HKPG1HT8 HTTP/1.1\r\nHost: matrix-test.domain\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nIf-None-Match: "1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"\r\n\r\n'
reply: 'HTTP/1.1 304 Not Modified\r\n'
header: Server: nginx/1.26.3
header: Date: Thu, 20 Feb 2025 08:56:06 GMT
header: Connection: keep-alive
header: Access-Control-Allow-Origin: *
header: Access-Control-Expose-Headers: etag
header: Pragma: no-cache
header: Cache-Control: no-store, no-transform
header: ETag: "1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"
header: Expires: Thu, 20 Feb 2025 08:57:06 GMT
header: Last-Modified: Thu, 20 Feb 2025 08:56:06 GMT
DEBUG:urllib3.connectionpool:https://matrix-test.domain:443 "GET /_synapse/client/rendezvous/01JMH94HZD117DP207HKPG1HT8 HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): matrix-test.domain:443
send: b'GET /_synapse/client/rendezvous/01JMH94HZD117DP207HKPG1HT8 HTTP/1.1\r\nHost: matrix-test.domain\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx/1.26.3
header: Date: Thu, 20 Feb 2025 08:56:06 GMT
header: Content-Type: text/plain
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Vary: Accept-Encoding
header: Access-Control-Allow-Origin: *
header: Access-Control-Expose-Headers: etag
header: Pragma: no-cache
header: Cache-Control: no-store, no-transform
header: ETag: W/"1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"
header: Expires: Thu, 20 Feb 2025 08:57:06 GMT
header: Last-Modified: Thu, 20 Feb 2025 08:56:06 GMT
header: Content-Encoding: gzip
DEBUG:urllib3.connectionpool:https://matrix-test.domain:443 "GET /_synapse/client/rendezvous/01JMH94HZD117DP207HKPG1HT8 HTTP/1.1" 200 None
Traceback (most recent call last):
  File "/root/temp/matrix/check.py", line 38, in <module>
    assert response.headers["ETag"] == etag
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

@waclaw66
Copy link
Author

With etag values visible...
W/"1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8" != "1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"

That was exactly my suspicions.

@sandhose
Copy link
Member

This is super useful: I think this is because nginx will switch to Transfer-Encoding: chunked on some responses, as it doesn't know the Content-Length of those responses. I've pushed another commit to the same PR/branch, could you try it out?

@waclaw66
Copy link
Author

Unfortunatelly...

send: b'POST /_matrix/client/unstable/org.matrix.msc4108/rendezvous HTTP/1.1\r\nHost: matrix-test.domain\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: text/plain\r\nContent-Length: 1024\r\n\r\n'
send: b'INIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIALINIITIAL'
reply: 'HTTP/1.1 201 Created\r\n'
header: Server: nginx/1.26.3
header: Date: Thu, 20 Feb 2025 09:34:27 GMT
header: Content-Type: application/json
header: Content-Length: 96
header: Connection: keep-alive
header: Access-Control-Allow-Origin: *
header: Access-Control-Expose-Headers: etag
header: Pragma: no-cache
header: Cache-Control: no-store, no-transform
header: ETag: "1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"
header: Expires: Thu, 20 Feb 2025 09:35:27 GMT
header: Last-Modified: Thu, 20 Feb 2025 09:34:27 GMT
DEBUG:urllib3.connectionpool:https://matrix-test.domain:443 "POST /_matrix/client/unstable/org.matrix.msc4108/rendezvous HTTP/1.1" 201 96
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): matrix-test.domain:443
send: b'GET /_synapse/client/rendezvous/01JMHBAR8V4AW1VTZE4XV5M2RR HTTP/1.1\r\nHost: matrix-test.domain\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nIf-None-Match: "1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"\r\n\r\n'
reply: 'HTTP/1.1 304 Not Modified\r\n'
header: Server: nginx/1.26.3
header: Date: Thu, 20 Feb 2025 09:34:27 GMT
header: Connection: keep-alive
header: Access-Control-Allow-Origin: *
header: Access-Control-Expose-Headers: etag
header: Pragma: no-cache
header: Cache-Control: no-store, no-transform
header: ETag: "1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"
header: Expires: Thu, 20 Feb 2025 09:35:27 GMT
header: Last-Modified: Thu, 20 Feb 2025 09:34:27 GMT
DEBUG:urllib3.connectionpool:https://matrix-test.domain:443 "GET /_synapse/client/rendezvous/01JMHBAR8V4AW1VTZE4XV5M2RR HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): matrix-test.domain:443
send: b'GET /_synapse/client/rendezvous/01JMHBAR8V4AW1VTZE4XV5M2RR HTTP/1.1\r\nHost: matrix-test.domain\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx/1.26.3
header: Date: Thu, 20 Feb 2025 09:34:27 GMT
header: Content-Type: text/plain
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Vary: Accept-Encoding
header: Access-Control-Allow-Origin: *
header: Access-Control-Expose-Headers: etag
header: Pragma: no-cache
header: Cache-Control: no-store, no-transform
header: ETag: W/"1tZH7IWqY7fZ0wc7XSuKjDnYNdt4ykPzzXO9Ad-TBI8"
header: Expires: Thu, 20 Feb 2025 09:35:27 GMT
header: Last-Modified: Thu, 20 Feb 2025 09:34:27 GMT
header: Content-Encoding: gzip
DEBUG:urllib3.connectionpool:https://matrix-test.domain:443 "GET /_synapse/client/rendezvous/01JMHBAR8V4AW1VTZE4XV5M2RR HTTP/1.1" 200 None
Traceback (most recent call last):
  File "/root/temp/matrix/check.py", line 40, in <module>
    assert response.headers["ETag"] == etag
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

@sandhose
Copy link
Member

sandhose commented Feb 20, 2025

I missed that before, looks like nginx still gzips the body. Looks like nginx doesn't respect the Cache-Control: no-transform directive.

Do you have gzip_proxied set? If so, can you set it to gzip_proxied no_etag?

@waclaw66
Copy link
Author

waclaw66 commented Feb 20, 2025

Unfortunately gzip_proxied no_etag does not help, only gzip off.

@waclaw66
Copy link
Author

waclaw66 commented Feb 20, 2025

The workaround to force uncompressed response from Synapse's side would be header like:
Accept-Encoding: identity;q=0
I've checked that within above python script and it works fine.

Anyway, why is uncompressed response required? Etags are the same, the only difference is the weak validator.

@sandhose
Copy link
Member

the only difference is the weak validator.

Whilst this is true for nginx, it isn't the case for other implementations like Caddy, which output a strong ETag with -gzip added at the end. On the other hand, Caddy and Cloudflare respect the Cache-Control: no-transform, but nginx doesn't

The workaround to force uncompressed response from Synapse's side would be header like: Accept-Encoding: identity;q=0

That's on the client side, right? I guess it would require a change on the MSC for that, I don't really like this workaround :(

I don't really know how we could fix this on the Synapse side, other than asking people to check their reverse proxy config and disable compression

@waclaw66
Copy link
Author

waclaw66 commented Feb 20, 2025

Whilst this is true for nginx, it isn't the case for other implementations like Caddy, which output a strong ETag with -gzip added at the end. On the other hand, Caddy and Cloudflare respect the Cache-Control: no-transform, but nginx doesn't

Then nginx should be fixed. Will try to investigate...

That's on the client side, right? I guess it would require a change on the MSC for that, I don't really like this workaround :(

Yes, it's client side, not Synapse as I wrote.

I don't really know how we could fix this on the Synapse side, other than asking people to check their reverse proxy config and disable compression

Disabling gzip for those enpoints works for me, I can live with that. However it would be good to document it, because it will be a problem for others when MAS is used widely.

Thank you for helping me to investigate the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants