Skip to content

Commit d1c785c

Browse files
averakaverak
authored
Merge pull request #8 from averak/feature/7-schema-apply
schema applyのI/Fを実装した
2 parents 5858990 + 79682dc commit d1c785c

File tree

10 files changed

+465
-89
lines changed

10 files changed

+465
-89
lines changed

‎README.md

+42-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,45 @@
22

33
![CI](https://github.com/averak/protobq/workflows/CI/badge.svg)
44

5-
A tool for idempotent schema management in BigQuery using Protocol Buffers.
5+
`protobq` is a tool designed to simplify and streamline schema management for materialized views in BigQuery.
6+
Instead of managing both the base tables and materialized views separately, developers only need to define the schema of the materialized view.
7+
Based on this schema, `protobq` automatically constructs and maintains the corresponding base table.
8+
9+
This approach ensures consistency between the materialized view and its source data, allowing developers to focus on high-level data modeling without worrying about the complexities of table creation or maintenance.
10+
11+
Key features:
12+
13+
- **Idempotent Schema Management**: Define your materialized view schema declaratively, and let `protobq` handle updates and changes seamlessly.
14+
- **Base Table Automation**: Automatically create and manage the base table from your materialized view schema.
15+
- **BigQuery-Native Optimization**: Leverage BigQuery’s best practices, such as partitioning, clustering, and incremental refreshes, directly through schema definitions.
16+
- **Protocol Buffers Integration**: Use Protocol Buffers to define your schemas, enabling compatibility, extensibility, and multi-language support.
17+
18+
## Philosophy
19+
20+
### Why Protocol Buffers?
21+
22+
- **Schema-First Approach**
23+
BigQuery’s schema-driven nature aligns with protobuf, enabling structured and type-safe schema definitions.
24+
25+
- **Versioning and Evolution**
26+
Protobuf supports backward and forward compatibility, simplifying schema updates and ensuring long-term maintainability.
27+
28+
- **Seamless BigQuery Integration**
29+
BigQuery types map directly to protobuf types (`STRING` → `string`, etc.), ensuring consistency and reducing conversion complexity.
30+
31+
- **Readable and Extensible**
32+
Protobuf schemas are both human-readable and machine-readable, aiding collaboration, automation, and extensibility.
33+
34+
### Why Materialized View First?
35+
36+
- **Simplified Architecture**
37+
Consolidating data into a unified base table simplifies data pipelines and downstream processes.
38+
39+
- **Query Optimization**
40+
Materialized views allow flexible clustering and partitioning, improving query performance for diverse use cases.
41+
42+
- **Cost and Performance Benefits**
43+
Precomputed results lower query costs and significantly improve performance for repetitive workloads.
44+
45+
- **Consistency and Reusability**
46+
A single base table ensures data integrity and facilitates schema reuse across multiple materialized views.

‎cmd/protobq/main.go

+12-1
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
package main
22

33
import (
4+
"context"
45
"log"
56
"os"
67

8+
"github.com/averak/protobq/internal"
79
"github.com/urfave/cli"
810
)
911

@@ -21,8 +23,17 @@ func newCliApp() *cli.App {
2123
{
2224
Name: "apply",
2325
Usage: "apply schema",
26+
Flags: []cli.Flag{
27+
&cli.StringFlag{
28+
Name: "project-id",
29+
Required: true,
30+
},
31+
},
2432
Action: func(c *cli.Context) error {
25-
// TODO: implements me
33+
err := internal.Apply(context.Background(), c.String("project-id"))
34+
if err != nil {
35+
return err
36+
}
2637
return nil
2738
},
2839
},

‎go.mod

+43
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,49 @@ require (
88
)
99

1010
require (
11+
cloud.google.com/go v0.116.0 // indirect
12+
cloud.google.com/go/auth v0.11.0 // indirect
13+
cloud.google.com/go/auth/oauth2adapt v0.2.6 // indirect
14+
cloud.google.com/go/bigquery v1.65.0 // indirect
15+
cloud.google.com/go/compute/metadata v0.5.2 // indirect
16+
cloud.google.com/go/iam v1.2.2 // indirect
17+
github.com/apache/arrow/go/v15 v15.0.2 // indirect
1118
github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
19+
github.com/felixge/httpsnoop v1.0.4 // indirect
20+
github.com/go-logr/logr v1.4.2 // indirect
21+
github.com/go-logr/stdr v1.2.2 // indirect
22+
github.com/goccy/go-json v0.10.2 // indirect
23+
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
24+
github.com/google/flatbuffers v23.5.26+incompatible // indirect
25+
github.com/google/s2a-go v0.1.8 // indirect
26+
github.com/google/uuid v1.6.0 // indirect
27+
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
28+
github.com/googleapis/gax-go/v2 v2.14.0 // indirect
29+
github.com/klauspost/compress v1.16.7 // indirect
30+
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
31+
github.com/pierrec/lz4/v4 v4.1.18 // indirect
1232
github.com/russross/blackfriday/v2 v2.1.0 // indirect
33+
github.com/zeebo/xxh3 v1.0.2 // indirect
34+
go.opencensus.io v0.24.0 // indirect
35+
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
36+
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
37+
go.opentelemetry.io/otel v1.29.0 // indirect
38+
go.opentelemetry.io/otel/metric v1.29.0 // indirect
39+
go.opentelemetry.io/otel/trace v1.29.0 // indirect
40+
golang.org/x/crypto v0.29.0 // indirect
41+
golang.org/x/exp v0.0.0-20231006140011-7918f672742d // indirect
42+
golang.org/x/mod v0.20.0 // indirect
43+
golang.org/x/net v0.31.0 // indirect
44+
golang.org/x/oauth2 v0.24.0 // indirect
45+
golang.org/x/sync v0.9.0 // indirect
46+
golang.org/x/sys v0.27.0 // indirect
47+
golang.org/x/text v0.20.0 // indirect
48+
golang.org/x/time v0.8.0 // indirect
49+
golang.org/x/tools v0.24.0 // indirect
50+
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
51+
google.golang.org/api v0.210.0 // indirect
52+
google.golang.org/genproto v0.0.0-20241118233622-e639e219e697 // indirect
53+
google.golang.org/genproto/googleapis/api v0.0.0-20241113202542-65e8d215514f // indirect
54+
google.golang.org/genproto/googleapis/rpc v0.0.0-20241118233622-e639e219e697 // indirect
55+
google.golang.org/grpc v1.67.1 // indirect
1356
)

0 commit comments

Comments
 (0)