🐛 Use postgresql modules for migration

[jmontleon]

When using psql, errors in applying the sql generated from the dump do not reach bash without adding additional options. psql only generates a non-zero exit code when something fatal happens, like being unable to connect to the database host. So, all ansible sees is exit code 0 from psql and continues on, as if everything is great.

I did some reading up on postgres modules for ansible and we do not need to shell out to do this, so I have implemented it with modules.

Since handling the actual migration with the ansible modules I have not encountered a failure. Continuing to run upgrades to test ...

[rayfordj]

/lgtm

[rayfordj]

Suggested change

	path: /tmp/keycloak.sql
	path: /tmp/keycloak.sql

I don't see it used anywhere else besides here ('Dump database', 'Restore database', & 'Clean up database dump file'), so I've no concern leaving it as it is, but did have the thought as I was reviewing, "should this be hard-coded?", and wanted to raise it.

[jmontleon]

I'm not concerned about it being hardcoded, except that it's predictable if someone was looking for it to try and do something malicious. I think it's easy to generate a name, so I'll probably do that. As an aside I just updated the PR with the change that I think will actually resolve the random lost DB, and added a comment with explanation.

[jmontleon]

At least I hope it fixes. One success does not make a pattern. Especially since it took all day testing to get a failure. But it at least looks promising.

[jmontleon]

I left a comment on https://issues.redhat.com/browse/MTA-3335?filter=-1

It looks like the old service was flapping between the new and old DB pod because the selector matched both. You can see that in the verbose output from the source ping task I added. Note between retries the version changed from 15 to 12. This should not be happening.

I added a version label for the new deployment and service to prevent it matching the old db pod so the new service was stable, but since nothing was done for the old service it's matching and flipping between both old and new pod.

I have updated the name label of the new deployment/pod to incorporate the version in this PR which should prevent the old service from matching and stop the unpredictable results.

Old pod deployment labels/service selector:

    app.kubernetes.io/component: postgresql
    app.kubernetes.io/name: tackle-keycloak-postgresql
    app.kubernetes.io/part-of: tackle

Updated new deployment labels/service selector:

    app.kubernetes.io/component: postgresql
    app.kubernetes.io/name: tackle-keycloak-postgresql-15
    app.kubernetes.io/part-of: tackle
    version: "15"

TASK [Ping source database] *****************************************************************************************
task path: /opt/ansible/test.yml:22
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: 1000900000
<localhost> EXEC /bin/sh -c 'echo ~1000900000 && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /opt/ansible/.ansible/tmp `"&& mkdir "` echo /opt/ansible/.ansible/tmp/ansible-tmp-1721678295.6744003-4493-237665731786268 `" && echo ansible-tmp-1721678295.6744003-4493-237665731786268="` echo /opt/ansible/.ansible/tmp/ansible-tmp-1721678295.6744003-4493-237665731786268 `" ) && sleep 0'
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/database/postgresql/postgresql_ping.py
<localhost> PUT /opt/ansible/.ansible/tmp/ansible-local-446033c7e04m/tmpfczk_9n1 TO /opt/ansible/.ansible/tmp/ansible-tmp-1721678295.6744003-4493-237665731786268/AnsiballZ_postgresql_ping.py
<localhost> EXEC /bin/sh -c 'chmod u+x /opt/ansible/.ansible/tmp/ansible-tmp-1721678295.6744003-4493-237665731786268/ /opt/ansible/.ansible/tmp/ansible-tmp-1721678295.6744003-4493-237665731786268/AnsiballZ_postgresql_ping.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.8 /opt/ansible/.ansible/tmp/ansible-tmp-1721678295.6744003-4493-237665731786268/AnsiballZ_postgresql_ping.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /opt/ansible/.ansible/tmp/ansible-tmp-1721678295.6744003-4493-237665731786268/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: Ping source database (12 retries left).Result was: {
    "attempts": 1,
    "changed": false,
    "invocation": {
        "module_args": {
            "ca_cert": null,
            "db": "keycloak_db",
            "login_host": "tackle-kcpgsql",
            "login_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "login_unix_socket": "",
            "login_user": "user-q4wZ",
            "port": 5432,
            "ssl_mode": "prefer"
        }
    },
    "is_available": true,
    "retries": 13,
    "server_version": {
        "major": 15,
        "minor": 2
    }
}
<localhost> EXEC /bin/sh -c 'echo ~1000900000 && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /opt/ansible/.ansible/tmp `"&& mkdir "` echo /opt/ansible/.ansible/tmp/ansible-tmp-1721678311.1710331-4493-71238003131228 `" && echo ansible-tmp-1721678311.1710331-4493-71238003131228="` echo /opt/ansible/.ansible/tmp/ansible-tmp-1721678311.1710331-4493-71238003131228 `" ) && sleep 0'
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/database/postgresql/postgresql_ping.py
<localhost> PUT /opt/ansible/.ansible/tmp/ansible-local-446033c7e04m/tmpb_g0z_lm TO /opt/ansible/.ansible/tmp/ansible-tmp-1721678311.1710331-4493-71238003131228/AnsiballZ_postgresql_ping.py
<localhost> EXEC /bin/sh -c 'chmod u+x /opt/ansible/.ansible/tmp/ansible-tmp-1721678311.1710331-4493-71238003131228/ /opt/ansible/.ansible/tmp/ansible-tmp-1721678311.1710331-4493-71238003131228/AnsiballZ_postgresql_ping.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.8 /opt/ansible/.ansible/tmp/ansible-tmp-1721678311.1710331-4493-71238003131228/AnsiballZ_postgresql_ping.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /opt/ansible/.ansible/tmp/ansible-tmp-1721678311.1710331-4493-71238003131228/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "attempts": 2,
    "changed": false,
    "invocation": {
        "module_args": {
            "ca_cert": null,
            "db": "keycloak_db",
            "login_host": "tackle-kcpgsql",
            "login_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "login_unix_socket": "",
            "login_user": "user-q4wZ",
            "port": 5432,
            "ssl_mode": "prefer"
        }
    },
    "is_available": true,
    "server_version": {
        "major": 12,
        "minor": 11
    }
}