From 48d0c45f5b0b63860a7c7eeb2f823e5f11ee93b4 Mon Sep 17 00:00:00 2001 From: Christoph Ladurner Date: Tue, 16 Jul 2024 17:25:36 +0200 Subject: [PATCH] upgrade: add steps to migrate to v12 --- docs/releases/upgrading/upgrade-v12.0.md | 207 +++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 208 insertions(+) create mode 100644 docs/releases/upgrading/upgrade-v12.0.md diff --git a/docs/releases/upgrading/upgrade-v12.0.md b/docs/releases/upgrading/upgrade-v12.0.md new file mode 100644 index 00000000..10dfd01a --- /dev/null +++ b/docs/releases/upgrading/upgrade-v12.0.md @@ -0,0 +1,207 @@ +# Upgrading from v11 to v12 + +## Prerequisites + +The steps listed in this article require an existing local installation of InvenioRDM v11. + +!!! warning "Backup" + + Always backup your database and files before you try to perform an upgrade. + +!!! info "Older Versions" + + In case you have an InvenioRDM installation older than v11, you can gradually upgrade + using the existing infrastructure to v11 and afterwards continue from here. + +## Upgrade Steps + +!!! warning "Upgrade your invenio-cli" + + Make sure you have the latest `invenio-cli` installed. For InvenioRDM v12 it + is at least v1.3.1 + + ```bash + $ invenio-cli --version + invenio-cli, version 1.3.1 + ``` + +!!! info "Virtual environments" + + In case you are not inside a virtual environment, make sure that you prefix each `invenio` + command with `pipenv run`. + + +**Local development** + +Changing the Python version in your development environment highly +depends on your setup, and there is no golden rule. +One example would be to use [PyEnv](https://github.com/pyenv/pyenv). + +!!! warning "Risk of losing data" + + Your virtual env folder may contain uploaded files. If you kept the default + location it is in in `var/instance/data`. If you need to keep those files, + make sure you copy them over to the new virtual env in the same location. + The command `invenio files location list` shows the file upload location. + +If you upgraded your python version, you should recreate your virtualenv before +running `invenio-cli` or `pipenv` commands below. + + +### Upgrade InvenioRDM + +Python 3.9 or Python 3.12 are required to run InvenioRDM v12. + +There are two options to upgrade your system: + +#### Upgrade option 1: In-place + +This approach upgrades the dependencies in place. Your virtual environment for the +v11 version will be gone afterwards. + +```bash +cd + +# Upgrade to InvenioRDM v12 +invenio-cli packages update 12.0.0 +pipenv uninstall flask-babelex + +# Re-build assets +invenio-cli assets build +``` + +#### Upgrade option 2: New virtual environment + +This approach will create a new virtual environment and leaves the v11 as it is. +If you are using a docker image on your production instance this will be the +option you chose. + +##### Step 1 +- create a new virtual environment +- activate your new virtual environment +- install `invenio-cli` by `pip install invenio-cli` + +##### Step 2 +Update the file `/Pipfile`. + +```diff +[packages] +---invenio-app-rdm = {extras = [...], version = "~=11.0.0"} ++++invenio-app-rdm = {extras = [...], version = "~=12.0.0"} +``` + +Due to a dependency upgrade, update the following line. +```diff +---my-site = {editable="True", path="./site"} ++++my-site = {editable=true, path="./site"} +``` + +##### Step 3 +Update the `Pipfile.lock` file: + +```bash +invenio-cli packages lock +``` + +##### Step 4 +Install InvenioRDM v12: + +```bash +invenio-cli install +``` + +### Database migration + +Execute the database migration: + +```bash +invenio alembic upgrade +``` + +### Declare usage statistics processing queues + +```shell +invenio queues declare +``` + +### Data migration + +Execute the data migration: + +```bash +pipenv run invenio shell $(find $(pipenv --venv)/lib/*/site-packages/invenio_app_rdm -name migrate_11_0_to_12_0.py) +``` + +### Rebuild search indices + +```bash +invenio index destroy --yes-i-know +invenio index init +invenio rdm rebuild-all-indices +``` + +In v12, record statistics will be stored in search indices rather than the +database. These indices are created through some *index templates* machinery +rather than having indices registered directly in `Invenio-Search`. As such, the +search indices for statistics are not affected by `invenio index destroy +--yes-i-know` and are totally functional after the rebuild step. + +### New roles + +```bash +invenio roles create administration-moderation +invenio roles create administration + +invenio access allow administration-moderation role administration-moderation +invenio access allow administration-access role administration +invenio access allow superuser-access role administration +``` + +### New configuration variables + +```bash +COMMUNITIES_IDENTITIES_CACHE_REDIS_URL = "URI_TO_REDIS" +USERS_RESOURCES_ADMINISTRATION_ENABLED = True +THEME_SITENAME = "Project name for header and UI" +``` + +## Big Changes + +- remove: dependency of flask-babelex +- add: concept doi (aka parent doi) +- add: statistics. (NOTE: statistic is stored in opensearch/libresearch) +- add: administration panel +- add: set quota +- add: branded communities +- add: share options for drafts and records +- add: built-in optional metadata fields +- change: the default delimiter for the licenses vocabulary file has been + changed from ";" to ",". The default licenses are available + [here](https://github.com/inveniosoftware/invenio-rdm-records/blob/master/invenio_rdm_records/fixtures/data/vocabularies/licenses.csv). + You can now add and update licenses with `pipenv run invenio rdm-records + add-to-fixture licenses` + + + +## OPEN PROBLEMS + +- users-user-v2.0.0 vs users-user-v1.0.0 indices + the problem is that user-v1.0.0 does not have the `confirmed_at` attribute + which is needed in `/api/users`. + SOLUTION 1: + ```bash + invenio index destroy --yes-i-know + invenio index init + invenio rdm rebuild-all-indices + ``` + SOLUTION 2: + ```bash + invenio index delete users-user-v1.0.0-NUMBER --force --yes-i-know + invenio index create users-user-v2.0.0-NUMBER -b path/to/invenio_users_resources/records/mappings/os-v2/users/user-v2.0.0.json + invenio shell + from invenio_search.proxies import current_search_client + current_search_client.indices.put_alias("users-user-v2.0.0-NUMBER", "users-user-v2.0.0") + current_search_client.indices.put_alias("users-user-v2.0.0-NUMBER", "users") + exit + invenio rdm rebuild-all-indices -o users + ``` diff --git a/mkdocs.yml b/mkdocs.yml index d1ed85b7..562178ba 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -210,6 +210,7 @@ nav: - Version v2.0.0: releases/versions/version-v2.0.0.md - Version v1.0.0: releases/versions/version-v1.0.0.md - Upgrade guide: + - Upgrade from v11 to v12: releases/upgrading/upgrade-v12.0.md - Upgrade policy: releases/upgrading/index.md - Upgrade from v10 to v11: releases/upgrading/upgrade-v11.0.md - Upgrade from v9 to v10: releases/upgrading/upgrade-v10.0.md