google_pubsub_subscription - add 'filename_datetime_format' to cloud_storage_config

[husseyd]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Description

Hi

Pub/Sub Subscriptions with Cloud Storage config now allow definition of the filenameDatetimeFormat. Links below to Gcloud CLI and Pub/Sub API contain further information.

I suggest adding a field filename_datetime_format to the cloud_storage_config block for google_pubsub_subscription to handle this.

References to Pub/Sub API in reference section and sample configuration provided. Please note that the Pub/Sub service/API enforces some constraints on this field - namely than all of YYYYMMDDhhmmss must be used, but can be broken up with characters like /, -, _,:

New or Affected Resource(s)

• google_pubsub_subscription

Potential Terraform Configuration

resource "google_pubsub_subscription" "example" {
  name  = "example-subscription"
  topic = google_pubsub_topic.example.id

  cloud_storage_config {
    bucket = google_storage_bucket.example.name

    filename_datetime_format = "YYYYMM/DD/hh/mm_ss"
    filename_prefix = "pre-"
    filename_suffix = "-%{random_suffix}"

    max_bytes = 1000
    max_duration = "300s"
  }
  depends_on = [ 
    google_storage_bucket.example,
    google_storage_bucket_iam_member.admin,
  ]
}

References

See:

• https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/create#--cloud-storage-file-datetime-format
• https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions#CloudStorageConfig (field = filenameDatetimeFormat)

b/336319061

[husseyd]

Added note:

The google_pubsub_subscription resource will actually silently overwrite any filename_datetime_format config on the subscription that has been set on the subscription.

I discovered this as follows:

1. Create pubsub subscription in Terraform.
2. Update filename datetime format attribute of my pubsub subscription via Pubsub API (since the TF resource does not cater for this).
3. Make another change to the pubscription in Terraform (changed cloud_storage_config.max_bytes) (No diff was shown in the plan for the datetime format change)
4. Filename datetime format of the subsciption was reverted to the default after TF apply

This could be a bit of an issue as if any of my subscriptions are updated in TF, they will then revert to the default datetime format and files will appear in a different directory in GCS than they should

[lahuang4]

Apologies for the trouble. I'm working on adding filename_datetime_format support and hope to have it in your hands soon!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.