Skip to content

Migration Tool

Jump to bottom
Stuart Marshall edited this page Mar 5, 2025 · 2 revisions

A tool has been created to help with the data transfer when updating models. This will identify changes in the model and will transform data to fit the current model.

This is a work in progress so please use with caution. Please raise issues for any bugs you encounter and be sure to check any modified data.

Commands

Migrate Operation

To run the migration tool use the following command:

python manage.py migrate_safely -o migrate -m MODEL

-m is the flag for the model -o specifies the operation that is being used

Running this export the current graph and data and will save a backup of this in case of errors It will then run several comparisons between the new model and the existing model to check for changes

  • New Nodes - Any new nodes that have been added
  • New Functions - New functions that have not been found on the previous model
  • New Concepts - Any concepts that have not been registered in the db
  • Updated Node Names - Any labels that have been changed on nodes
  • Updated Concepts - Any nodes that were previously concepts and have been changed to a new concept
    • This will display the node id and name, the old concept and name, the new concept and name (will display 'needs registering' if it doesn't exist)
  • Deleted Nodes - The id and name of any nodes that have been removed
  • Datatype changes - Any nodes that have had the datatype changed

For the concept changes you will need to provide a mapping file in order to convert the data - see Mapping

The tool will stop and wait for input before continuing from the check. Hit Enter to convert the data

Once the conversion is complete a file called transformed_MODELNAME.json will be created. You can check this for changes

Hit enter again and the model change and data transfer will begin

Mapping

A mapping file can be provided with the -M flag:

python manage.py migrate_safely -o migrate -m MODEL -M MAPPINGFILE.json

The file is a json containing the node id keys with a value containing another object with the original concept id as keys and the new concept ids as values

{
   "6d09da12-e53e-11ef-93e2-0242ac120004":{ //node id
        "default": null, // default value
        "ea5bb474-e217-11ef-803e-0242ac120003": "8780196a-dfda-11ef-a67e-0242ac120006" // old concept id: new concept id
    } 
}

This allows you to provide mapping for multiple nodes in one file

You can also provide a default value. This will set the value on conversion if no match is found. If you do not need to convert, you can simply set the default to null and all values will be converted to null

Reverse

If any issues occur or you need to upload the old model this can be done using the -r flag

python manage.py migrate_safely -o migrate -m MODEL -r

This will use the exported backup model and the exported backup data to reverse the changes made. The data is stored in root as: backup_MODELNAME stale_data_MODELNAME

Data Conversions

Some data is automatically converted, not all datatype conversions are currently handled. The current datatypes handled are:

  • domain-value -> concept
  • concept -> domain
  • concept -> concept-list

These are handled automatically and the string values for domains and concepts will be compared to find a match. If no match is found null value will be given. Note This could be updated to allow the mapping file to be used

Groups

Functionality has been included for stripping the groups of members and re-hydrating an empty group model

Remove Members

You can export the groups and remove the members but retaining the links between groups by running:

python manage.py migrate_safely -o remove_members

This will export the groups with members as a standard export and a json named Empty_Groups_DATE with the members removed. This will be saved in coral/pkg/business_data/files

Rehydrate Members

This allows you to import a set of empty groups and hydrate it with the members set in the current environment. You will need to upload the empty group set into the environment manually. The idea is that you would create your empty group from the dev environment that has been updated with the changes. Then move this into staging and run the rehydrate members to update staging with the new groups, retaining the current members

You can run

python manage.py migrate_safely -o rehydrate_members

This will also export a copy of the original groups with members into coral/pkg/business_data/files

Clone this wiki locally