Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schema applyのI/Fを実装した #8

Merged
merged 2 commits into from
Dec 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,45 @@

![CI](https://github.com/averak/protobq/workflows/CI/badge.svg)

A tool for idempotent schema management in BigQuery using Protocol Buffers.
`protobq` is a tool designed to simplify and streamline schema management for materialized views in BigQuery.
Instead of managing both the base tables and materialized views separately, developers only need to define the schema of the materialized view.
Based on this schema, `protobq` automatically constructs and maintains the corresponding base table.

This approach ensures consistency between the materialized view and its source data, allowing developers to focus on high-level data modeling without worrying about the complexities of table creation or maintenance.

Key features:

- **Idempotent Schema Management**: Define your materialized view schema declaratively, and let `protobq` handle updates and changes seamlessly.
- **Base Table Automation**: Automatically create and manage the base table from your materialized view schema.
- **BigQuery-Native Optimization**: Leverage BigQuery’s best practices, such as partitioning, clustering, and incremental refreshes, directly through schema definitions.
- **Protocol Buffers Integration**: Use Protocol Buffers to define your schemas, enabling compatibility, extensibility, and multi-language support.

## Philosophy

### Why Protocol Buffers?

- **Schema-First Approach**
BigQuery’s schema-driven nature aligns with protobuf, enabling structured and type-safe schema definitions.

- **Versioning and Evolution**
Protobuf supports backward and forward compatibility, simplifying schema updates and ensuring long-term maintainability.

- **Seamless BigQuery Integration**
BigQuery types map directly to protobuf types (`STRING` → `string`, etc.), ensuring consistency and reducing conversion complexity.

- **Readable and Extensible**
Protobuf schemas are both human-readable and machine-readable, aiding collaboration, automation, and extensibility.

### Why Materialized View First?

- **Simplified Architecture**
Consolidating data into a unified base table simplifies data pipelines and downstream processes.

- **Query Optimization**
Materialized views allow flexible clustering and partitioning, improving query performance for diverse use cases.

- **Cost and Performance Benefits**
Precomputed results lower query costs and significantly improve performance for repetitive workloads.

- **Consistency and Reusability**
A single base table ensures data integrity and facilitates schema reuse across multiple materialized views.
13 changes: 12 additions & 1 deletion cmd/protobq/main.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
package main

import (
"context"
"log"
"os"

"github.com/averak/protobq/internal"
"github.com/urfave/cli"
)

Expand All @@ -21,8 +23,17 @@ func newCliApp() *cli.App {
{
Name: "apply",
Usage: "apply schema",
Flags: []cli.Flag{
&cli.StringFlag{
Name: "project-id",
Required: true,
},
},
Action: func(c *cli.Context) error {
// TODO: implements me
err := internal.Apply(context.Background(), c.String("project-id"))
if err != nil {
return err
}
return nil
},
},
Expand Down
43 changes: 43 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,49 @@ require (
)

require (
cloud.google.com/go v0.116.0 // indirect
cloud.google.com/go/auth v0.11.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.6 // indirect
cloud.google.com/go/bigquery v1.65.0 // indirect
cloud.google.com/go/compute/metadata v0.5.2 // indirect
cloud.google.com/go/iam v1.2.2 // indirect
github.com/apache/arrow/go/v15 v15.0.2 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/flatbuffers v23.5.26+incompatible // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.14.0 // indirect
github.com/klauspost/compress v1.16.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
go.opentelemetry.io/otel v1.29.0 // indirect
go.opentelemetry.io/otel/metric v1.29.0 // indirect
go.opentelemetry.io/otel/trace v1.29.0 // indirect
golang.org/x/crypto v0.29.0 // indirect
golang.org/x/exp v0.0.0-20231006140011-7918f672742d // indirect
golang.org/x/mod v0.20.0 // indirect
golang.org/x/net v0.31.0 // indirect
golang.org/x/oauth2 v0.24.0 // indirect
golang.org/x/sync v0.9.0 // indirect
golang.org/x/sys v0.27.0 // indirect
golang.org/x/text v0.20.0 // indirect
golang.org/x/time v0.8.0 // indirect
golang.org/x/tools v0.24.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/api v0.210.0 // indirect
google.golang.org/genproto v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241113202542-65e8d215514f // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/grpc v1.67.1 // indirect
)
Loading
Loading