feat(preprod): Add initial preprod artifact models

[trevor-e]

See https://www.notion.so/sentry/Emerge-upload-models-1f18b10e4b5d803e9bc4ffd27a25ac9f for more details.

I've omitted the Git fields for now and will tackle that in follow-up.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #92056      +/-   ##
==========================================
+ Coverage   81.76%   85.56%   +3.80%     
==========================================
  Files       10212    10214       +2     
  Lines      585180   585338     +158     
  Branches    22701    22669      -32     
==========================================
+ Hits       478468   500843   +22375     
+ Misses     106290    83779   -22511     
- Partials      422      716     +294     

[github-actions]

This PR has a migration; here is the generated SQL for src/sentry/migrations/0909_emerge_upload_models.py

for 0909_emerge_upload_models in sentry

--
-- Create model PreprodBuildConfiguration
--
CREATE TABLE "sentry_preprodbuildconfiguration" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NOT NULL, "organization_id" bigint NOT NULL, "name" varchar(255) NOT NULL, "project_id" bigint NOT NULL);
--
-- Create model PreprodArtifact
--
CREATE TABLE "sentry_preprodartifact" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NOT NULL, "organization_id" bigint NOT NULL, "date_built" timestamp with time zone NULL, "state" integer NOT NULL CHECK ("state" >= 0), "artifact_type" integer NULL CHECK ("artifact_type" >= 0), "error_code" integer NULL CHECK ("error_code" >= 0), "error_message" text NULL, "build_version" varchar(255) NULL, "build_number" bigint NULL, "build_uuid" uuid NULL, "misc" text NULL, "file_id" bigint NULL, "project_id" bigint NOT NULL, "build_configuration_id" bigint NULL);
--
-- Create model PreprodArtifactSizeMetrics
--
CREATE TABLE "sentry_preprodartifactsizemetrics" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NOT NULL, "metrics_artifact_type" integer NULL CHECK ("metrics_artifact_type" >= 0), "state" integer NOT NULL CHECK ("state" >= 0), "error_code" integer NULL CHECK ("error_code" >= 0), "error_message" text NULL, "processing_version" varchar(255) NULL, "min_install_size" bigint NULL CHECK ("min_install_size" >= 0), "max_install_size" bigint NULL CHECK ("max_install_size" >= 0), "min_download_size" bigint NULL CHECK ("min_download_size" >= 0), "max_download_size" bigint NULL CHECK ("max_download_size" >= 0), "preprod_artifact_id" bigint NOT NULL);
CREATE UNIQUE INDEX CONCURRENTLY "sentry_preprodbuildconfiguration_project_id_name_aa024645_uniq" ON "sentry_preprodbuildconfiguration" ("project_id", "name");
ALTER TABLE "sentry_preprodbuildconfiguration" ADD CONSTRAINT "sentry_preprodbuildconfiguration_project_id_name_aa024645_uniq" UNIQUE USING INDEX "sentry_preprodbuildconfiguration_project_id_name_aa024645_uniq";
ALTER TABLE "sentry_preprodbuildconfiguration" ADD CONSTRAINT "sentry_preprodbuildc_project_id_bc1f93ff_fk_sentry_pr" FOREIGN KEY ("project_id") REFERENCES "sentry_project" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodbuildconfiguration" VALIDATE CONSTRAINT "sentry_preprodbuildc_project_id_bc1f93ff_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_preprodbuildconfiguration_organization_id_b9d7fd5b" ON "sentry_preprodbuildconfiguration" ("organization_id");
CREATE INDEX CONCURRENTLY "sentry_preprodbuildconfiguration_project_id_bc1f93ff" ON "sentry_preprodbuildconfiguration" ("project_id");
ALTER TABLE "sentry_preprodartifact" ADD CONSTRAINT "sentry_preprodartifact_file_id_d5007350_fk_sentry_file_id" FOREIGN KEY ("file_id") REFERENCES "sentry_file" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodartifact" VALIDATE CONSTRAINT "sentry_preprodartifact_file_id_d5007350_fk_sentry_file_id";
ALTER TABLE "sentry_preprodartifact" ADD CONSTRAINT "sentry_preprodartifact_project_id_ea7579b7_fk_sentry_project_id" FOREIGN KEY ("project_id") REFERENCES "sentry_project" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodartifact" VALIDATE CONSTRAINT "sentry_preprodartifact_project_id_ea7579b7_fk_sentry_project_id";
ALTER TABLE "sentry_preprodartifact" ADD CONSTRAINT "sentry_preprodartifa_build_configuration__ed071cfe_fk_sentry_pr" FOREIGN KEY ("build_configuration_id") REFERENCES "sentry_preprodbuildconfiguration" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodartifact" VALIDATE CONSTRAINT "sentry_preprodartifa_build_configuration__ed071cfe_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_preprodartifact_organization_id_f2debf0c" ON "sentry_preprodartifact" ("organization_id");
CREATE INDEX CONCURRENTLY "sentry_preprodartifact_file_id_d5007350" ON "sentry_preprodartifact" ("file_id");
CREATE INDEX CONCURRENTLY "sentry_preprodartifact_project_id_ea7579b7" ON "sentry_preprodartifact" ("project_id");
CREATE INDEX CONCURRENTLY "sentry_preprodartifact_build_configuration_id_ed071cfe" ON "sentry_preprodartifact" ("build_configuration_id");
CREATE UNIQUE INDEX CONCURRENTLY "sentry_preprodartifactsi_preprod_artifact_id_metr_bbdc0162_uniq" ON "sentry_preprodartifactsizemetrics" ("preprod_artifact_id", "metrics_artifact_type");
ALTER TABLE "sentry_preprodartifactsizemetrics" ADD CONSTRAINT "sentry_preprodartifactsi_preprod_artifact_id_metr_bbdc0162_uniq" UNIQUE USING INDEX "sentry_preprodartifactsi_preprod_artifact_id_metr_bbdc0162_uniq";
ALTER TABLE "sentry_preprodartifactsizemetrics" ADD CONSTRAINT "sentry_preprodartifa_preprod_artifact_id_5ddef58c_fk_sentry_pr" FOREIGN KEY ("preprod_artifact_id") REFERENCES "sentry_preprodartifact" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodartifactsizemetrics" VALIDATE CONSTRAINT "sentry_preprodartifa_preprod_artifact_id_5ddef58c_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_preprodartifactsizemetrics_preprod_artifact_id_5ddef58c" ON "sentry_preprodartifactsizemetrics" ("preprod_artifact_id");

[cmanallen]

Looks good! A couple comments for Dan to verify internal idioms.

[wedamija]

I'm not a huge fan of dumping ground json blobs, is there a way we can have a schema here? Or do we have a general idea of what will go in here?

[cmanallen]

The misc naming can be improved as well. If the JSON field is retained we'd want a dictionary word here. Some possible names: "extra", "context", "tags". If this is arbitrary, user-specified build data then tags would fit the Sentry naming convention.

[github-actions]

This PR has a migration; here is the generated SQL for src/sentry/preprod/migrations/0001_emerge_upload_models.py

for 0001_emerge_upload_models in preprod

--
-- Create model PreprodBuildConfiguration
--
CREATE TABLE "sentry_preprodbuildconfiguration" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NOT NULL, "name" varchar(255) NOT NULL, "project_id" bigint NOT NULL);
--
-- Create model PreprodArtifact
--
CREATE TABLE "sentry_preprodartifact" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NOT NULL, "file_id" bigint NULL, "date_built" timestamp with time zone NULL, "state" integer NOT NULL CHECK ("state" >= 0), "artifact_type" integer NULL CHECK ("artifact_type" >= 0), "error_code" integer NULL CHECK ("error_code" >= 0), "error_message" text NULL, "build_version" varchar(255) NULL, "build_number" bigint NULL, "extras" text NULL, "project_id" bigint NOT NULL, "build_configuration_id" bigint NULL);
--
-- Create model PreprodArtifactSizeMetrics
--
CREATE TABLE "sentry_preprodartifactsizemetrics" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NOT NULL, "metrics_artifact_type" integer NULL CHECK ("metrics_artifact_type" >= 0), "state" integer NOT NULL CHECK ("state" >= 0), "error_code" integer NULL CHECK ("error_code" >= 0), "error_message" text NULL, "processing_version" varchar(255) NULL, "min_install_size" bigint NULL CHECK ("min_install_size" >= 0), "max_install_size" bigint NULL CHECK ("max_install_size" >= 0), "min_download_size" bigint NULL CHECK ("min_download_size" >= 0), "max_download_size" bigint NULL CHECK ("max_download_size" >= 0), "preprod_artifact_id" bigint NOT NULL);
CREATE UNIQUE INDEX CONCURRENTLY "sentry_preprodbuildconfiguration_project_id_name_aa024645_uniq" ON "sentry_preprodbuildconfiguration" ("project_id", "name");
ALTER TABLE "sentry_preprodbuildconfiguration" ADD CONSTRAINT "sentry_preprodbuildconfiguration_project_id_name_aa024645_uniq" UNIQUE USING INDEX "sentry_preprodbuildconfiguration_project_id_name_aa024645_uniq";
ALTER TABLE "sentry_preprodbuildconfiguration" ADD CONSTRAINT "sentry_preprodbuildc_project_id_bc1f93ff_fk_sentry_pr" FOREIGN KEY ("project_id") REFERENCES "sentry_project" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodbuildconfiguration" VALIDATE CONSTRAINT "sentry_preprodbuildc_project_id_bc1f93ff_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_preprodbuildconfiguration_project_id_bc1f93ff" ON "sentry_preprodbuildconfiguration" ("project_id");
ALTER TABLE "sentry_preprodartifact" ADD CONSTRAINT "sentry_preprodartifact_project_id_ea7579b7_fk_sentry_project_id" FOREIGN KEY ("project_id") REFERENCES "sentry_project" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodartifact" VALIDATE CONSTRAINT "sentry_preprodartifact_project_id_ea7579b7_fk_sentry_project_id";
ALTER TABLE "sentry_preprodartifact" ADD CONSTRAINT "sentry_preprodartifa_build_configuration__ed071cfe_fk_sentry_pr" FOREIGN KEY ("build_configuration_id") REFERENCES "sentry_preprodbuildconfiguration" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodartifact" VALIDATE CONSTRAINT "sentry_preprodartifa_build_configuration__ed071cfe_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_preprodartifact_file_id_d5007350" ON "sentry_preprodartifact" ("file_id");
CREATE INDEX CONCURRENTLY "sentry_preprodartifact_project_id_ea7579b7" ON "sentry_preprodartifact" ("project_id");
CREATE INDEX CONCURRENTLY "sentry_preprodartifact_build_configuration_id_ed071cfe" ON "sentry_preprodartifact" ("build_configuration_id");
CREATE UNIQUE INDEX CONCURRENTLY "sentry_preprodartifactsi_preprod_artifact_id_metr_bbdc0162_uniq" ON "sentry_preprodartifactsizemetrics" ("preprod_artifact_id", "metrics_artifact_type");
ALTER TABLE "sentry_preprodartifactsizemetrics" ADD CONSTRAINT "sentry_preprodartifactsi_preprod_artifact_id_metr_bbdc0162_uniq" UNIQUE USING INDEX "sentry_preprodartifactsi_preprod_artifact_id_metr_bbdc0162_uniq";
ALTER TABLE "sentry_preprodartifactsizemetrics" ADD CONSTRAINT "sentry_preprodartifa_preprod_artifact_id_5ddef58c_fk_sentry_pr" FOREIGN KEY ("preprod_artifact_id") REFERENCES "sentry_preprodartifact" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_preprodartifactsizemetrics" VALIDATE CONSTRAINT "sentry_preprodartifa_preprod_artifact_id_5ddef58c_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_preprodartifactsizemetrics_preprod_artifact_id_5ddef58c" ON "sentry_preprodartifactsizemetrics" ("preprod_artifact_id");

[trevor-e]

Renamed misc -> extras. To give you an example of stuff we threw in here with our old schema:

• time processing was enqueued
• time processing was dequeued
• user agent of uploader
• hostname that processed the upload
• the user's build directory
• some override fields for whether a feature is enabled
• etc

Nothing that really needs a column of its own or a strict schema.

[markstory]

You don't need to include migration tests for schema modifications. We only need migration tests for backfills or data mutations.

[trevor-e]

Yea this one felt a bit silly to write, but wanted to follow "For this reason, every migration should have a corresponding integration test." from the docs lol