Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AWS billing datasource partition updates and HiveTable partition validation #1003

Closed
wants to merge 3 commits into from
Closed

Fix AWS billing datasource partition updates and HiveTable partition validation #1003

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a38007f
charts,manifests: Fix HiveTable partitions partitionSpec validation
Oct 23, 2019
093af85
pkg/operator: Fix AWS partition values column names
Oct 23, 2019
cdb26b5
pkg/operator/reporting: Fix quoting of partition values and partition…
Oct 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion charts/metering-ansible-operator/templates/crds/hive.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,13 @@ spec:
- location
properties:
partitionSpec:
type: array
type: object
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
additionalProperties:
type: string
location:
type: string
description: |
Expand Down
4 changes: 3 additions & 1 deletion manifests/deploy/ocp-testing/metering-ansible-operator/hive.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,13 @@ spec:
- location
properties:
partitionSpec:
type: array
type: object
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
additionalProperties:
type: string
location:
type: string
description: |
Expand Down
4 changes: 3 additions & 1 deletion manifests/deploy/ocp-testing/olm/bundle/4.3/hive.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,13 @@ spec:
- location
properties:
partitionSpec:
type: array
type: object
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
additionalProperties:
type: string
location:
type: string
description: |
Expand Down
4 changes: 3 additions & 1 deletion manifests/deploy/openshift/metering-ansible-operator/hive.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,13 @@ spec:
- location
properties:
partitionSpec:
type: array
type: object
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
additionalProperties:
type: string
location:
type: string
description: |
Expand Down
4 changes: 3 additions & 1 deletion manifests/deploy/openshift/olm/bundle/4.3/hive.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,13 @@ spec:
- location
properties:
partitionSpec:
type: array
type: object
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
additionalProperties:
type: string
location:
type: string
description: |
Expand Down
4 changes: 3 additions & 1 deletion manifests/deploy/openshift/telemeter/list.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,11 +210,13 @@ objects:
minLength: 1
type: string
partitionSpec:
additionalProperties:
type: string
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
type: array
type: object
required:
- partitionSpec
- location
Expand Down
4 changes: 3 additions & 1 deletion manifests/deploy/upstream/metering-ansible-operator/hive.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,13 @@ spec:
- location
properties:
partitionSpec:
type: array
type: object
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
additionalProperties:
type: string
location:
type: string
description: |
Expand Down
4 changes: 3 additions & 1 deletion manifests/deploy/upstream/olm/bundle/4.3/hive.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,13 @@ spec:
- location
properties:
partitionSpec:
type: array
type: object
description: |
PartitionSpec is a map containing string keys and values, where each key
is expected to be the name of a partition column, and the value is the
value of the partition column.
additionalProperties:
type: string
location:
type: string
description: |
Expand Down
11 changes: 7 additions & 4 deletions pkg/operator/aws_usage_hive.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,15 @@ WITH SERDEPROPERTIES (
"timestamp.formats" = "yyyy-MM-dd'T'HH:mm:ssZ"
)
`

billingPeriodStartPartitionColumnName = "billing_period_start"
billingPeriodEndPartitionColumnName = "billing_period_end"
)

var (
AWSUsageHivePartitions = []hive.Column{
{Name: "billing_period_start", Type: "string"},
{Name: "billing_period_end", Type: "string"},
{Name: billingPeriodStartPartitionColumnName, Type: "string"},
{Name: billingPeriodEndPartitionColumnName, Type: "string"},
}
)

Expand Down Expand Up @@ -151,8 +154,8 @@ func getDesiredPartitions(bucket string, manifests []*aws.Manifest) ([]metering.
p := metering.HiveTablePartition{
Location: location,
PartitionSpec: hive.PartitionSpec{
"start": start,
"end": end,
billingPeriodStartPartitionColumnName: start,
billingPeriodEndPartitionColumnName: end,
},
}
desiredPartitions = append(desiredPartitions, p)
Expand Down
4 changes: 2 additions & 2 deletions pkg/operator/reporting/db.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ func (m *HiveManager) AddPartition(tableName string, partitionColumns []hive.Col
partitionSpecStr := FmtPartitionSpec(partitionColumns, partition.PartitionSpec)
locationStr := ""
if partition.Location != "" {
locationStr = "LOCATION " + partition.Location
locationStr = fmt.Sprintf("LOCATION '%s'", partition.Location)
}
_, err := m.execer.Exec(fmt.Sprintf("ALTER TABLE %s ADD IF NOT EXISTS PARTITION (%s) %s", tableName, partitionSpecStr, locationStr))
return err
Expand All @@ -70,7 +70,7 @@ func FmtPartitionSpec(partitionColumns []hive.Column, partSpec hive.PartitionSpe
val := partSpec[col.Name]
// Quote strings
if strings.ToLower(col.Type) == "string" {
val = "`" + val + "`"
val = fmt.Sprintf("'%s'", val)
}
partitionVals = append(partitionVals, fmt.Sprintf("`%s`=%s", col.Name, val))
}
Expand Down