[Bug]: Datastore/Firestore Bulk Delete template performance degration

[Abacn]

Related Template(s)

Datastore/Firestore Bulk Delete template

Template Version

2021-12-06-00_rc01

What happened?

Performance degration of DatastoreToDatastoreDelete template is found since version "2021-12-06-00_rc01":

- *SDK: 2.32.0*
- template_version: 2021-11-01-00_RC00
- time: *3 min 15 sec*

- *SDK: 2.34.0* (*data loss bug?)
- template_version: 2021-12-06-00_rc01
- time: *27 min 25 sec*

because this template has long been deprecated. The current recommendation is to stick at version "2021-11-01-00_RC00". To do this, select template path "gs://dataflow-templates/2021-11-01-00_RC00/Datastore_to_Datastore_Delete" when initiating the Template.

Relevant log output

[Abacn]

I believe root cause is apache/beam#14713 , and checked that the change of RampupThrottling isn't effective in 2021-11-01-00_RC00, while starting for 2021-12-06-00_rc01 the job graph of the template has changed