How to synchronise full donor or cord JSON files?

[mmelchers]

We are trying to get donors into the system, but it seems like when we send a "full" file, the donors do not get synced to the matching database(s). Is there a reason for that? I don't know how to get the donors into the matching database now except for sending them all again in a Diff file and as updates. Is there a better way?

[benbelow]

Hi @mmelchers

This behaviour is by design - of the two types of input file, "full" and "diff":

• "diff" will automatically publish messages on an internal service bus for each updated donor, for consumption by the matching database, to ensure donors are up to date
  • this process isn't the most efficient - it can handle the load of daily donor updates / additions / deletions no problem, but when asking it to import all donors this way, it would take a lot longer than ideal (we're probably looking at an order of extra days)
• "full" will import all donors to the master Atlas donor store, but not to the matching databases, as you've identified. There is a second process that will import all donors from the master store, known as the "Hla Refresh" - this also needs to be run every 3 months when the new nomenclature is published.
  • as this reads directly from the donor store database, rather than individual service bus messages, it is orders of magnitude faster for a large quantity of donors. This should take hours, rather than days, for the WMDA sized dataset.

So, once you've finished importing all donors to the store, you'll need to run the "Data Refresh", which can be manually triggered.

There is some documentation on these processes in the repo, I'll share some links:

• Details of the two import types

• Details on the data refresh

[mmelchers]

there does not seem to be a "RunDataRefreshManual" function. There is a "RunDataRefresh" function and there is a "SubmitDataRefreshRequestManual". Which one is it supposed to be? It seems to be the "SubmitDataRefreshRequestManual" function.

[benbelow]

Ah, apologies - it looks like this was renamed and didn't make it to the docs, I'll update them now.

• RunDataRefresh is an internal function, triggered by a service bus
• SubmitDataRefreshRequestManual is what you want - it's an HTTP request that will allow you to request a data refresh to occur with specific parameters.
  • (the only parameter available to configure is ForceDataRefresh - if set it will force the full refresh, even if there's no change in latest nomenclature. If false, it will not perform the refresh if it detects that the current donor store is up to date)