Skip to content

Commit

Permalink
update dev docs
Browse files Browse the repository at this point in the history
Signed-off-by: Ivan Milchev <ivan@mondoo.com>
  • Loading branch information
imilchev committed Feb 1, 2024
1 parent 5a687fb commit 5d129e2
Show file tree
Hide file tree
Showing 2 changed files with 114 additions and 1 deletion.
113 changes: 113 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ go 1.21
use (
./cnquery
./cnquery/providers/atlassian
./cnquery/providers/arista
./cnquery/providers/aws
./cnquery/providers/azure
Expand All @@ -200,6 +201,118 @@ use (
)
```

## Providers development best practices

The more time we spent building providers, the more learn about how to do that better in the future. Here we described learnings that would help you get started with providers development.

### Referencing MQL resources
Often we have a top-level MQL resource, which we want to reference in another top-level resource.

For example, GCP networks can be retrieved for a project. That is a top-level resource:
```
// GCP Compute Engine
private gcp.project.computeService {
// Google Compute Engine VPC network in a project
networks() []gcp.project.computeService.network
}
```

However, we have a reference to a GCP network in a GCP Compute Address. This allows us to quickly navigate to the network in which an address is created:
```
private gcp.project.computeService.address {
// Static IP address
address string
// Network in which to reserve the address
network() gcp.project.computeService.network
}
```

The simple wait to implement the reference would be to call the GCP API every time `gcp.project.computeService.address.network` is executed. However, this means we are going to be doing an excessive amount of API calls when scanning large GCP projects. If we have 10 addresses, this would mean we will do 10 separete API calls to get the network for each of them.

Check failure on line 231 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`separete` is not a recognized word. (unrecognized-spelling)

MQL has powerful caching capabilities that would allow to achieve the same end result with a single (or fewer) API calls.

First, create an init function for `gcp.project.computeService.network`, which is the resource we are cross-referencing:
```go
func initGcpProjectComputeServiceNetwork(runtime *plugin.Runtime, args map[string]*llx.RawData) (map[string]*llx.RawData, plugin.Resource, error) {

Check failure on line 237 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`llx` is not a recognized word. (unrecognized-spelling)

Check failure on line 237 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`llx` is not a recognized word. (unrecognized-spelling)
// Here we check that the resource isn't fully initialized yet
if len(args) > 2 {

Check failure on line 239 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`len` is not a recognized word. (unrecognized-spelling)
return args, nil, nil
}

// If no args are set, try reading them from the platform ID
if len(args) == 0 {

Check failure on line 244 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`len` is not a recognized word. (unrecognized-spelling)
if ids := getAssetIdentifier(runtime); ids != nil {
args["name"] = llx.StringData(ids.name)

Check failure on line 246 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`llx` is not a recognized word. (unrecognized-spelling)
args["projectId"] = llx.StringData(ids.project)

Check failure on line 247 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`llx` is not a recognized word. (unrecognized-spelling)
} else {
return nil, nil, errors.New("no asset identifier found")
}
}

// We create a gcp.project.computeService resource which would allow us to retrieve networks via MQL
obj, err := CreateResource(runtime, "gcp.project.computeService", map[string]*llx.RawData{

Check failure on line 254 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`llx` is not a recognized word. (unrecognized-spelling)
"projectId": args["projectId"],
})
if err != nil {
return nil, nil, err
}

// Cast the resource to the appropriate type
computeSvc := obj.(*mqlGcpProjectComputeService)
// List the networks: equivalent to gcp.project.computeService.networks MQL query. This will retrieve all networks in the project and cache them in the MQL cache. Consecutive calls to this will retrieve the data from cache and will not execute any API calls.
networks := computeSvc.GetNetworks()
if networks.Error != nil {
return nil, nil, networks.Error
}

// Filter the networks in memory by comparing them with the input arguments
for _, n := range networks.Data {
network := n.(*mqlGcpProjectComputeServiceNetwork)
name := network.GetName()
if name.Error != nil {
return nil, nil, name.Error
}
projectId := network.GetProjectId()
if projectId.Error != nil {
return nil, nil, projectId.Error
}

// return the resource if found
if name.Data == args["name"].Value && projectId.Data == args["projectId"].Value {
return args, network, nil
}
}
return nil, nil, fmt.Errorf("not found")

Check failure on line 286 in docs/development.md

View workflow job for this annotation

GitHub Actions / Run spell check

`Errorf` is not a recognized word. (unrecognized-spelling)
}
```

Then, we implement the function for retrieving the network for a GCP compute address as follows:
```go
func (g *mqlGcpProjectComputeServiceAddress) network() (*mqlGcpProjectComputeServiceNetwork, error) {
if g.NetworkUrl.Error != nil {
return nil, g.NetworkUrl.Error
}
networkUrl := g.NetworkUrl.Data

// Format is https://www.googleapis.com/compute/v1/projects/project1/global/networks/net-1
params := strings.TrimPrefix(networkUrl, "https://www.googleapis.com/compute/v1/")
parts := strings.Split(params, "/")
resId := resourceId{Project: parts[1], Region: parts[2], Name: parts[4]}

// Use the init function for the resource to find the one that we need
res, err := CreateResource(g.MqlRuntime, "gcp.project.computeService.network", map[string]*llx.RawData{
"name": llx.StringData(resId.Name),
"projectId": llx.StringData(resId.Project),
})
if err != nil {
return nil, err
}
return res.(*mqlGcpProjectComputeServiceNetwork), nil
}

```

## Contribute changes

### Mark PRs with emojis
Expand Down
2 changes: 1 addition & 1 deletion shared/proto/cnquery.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 5d129e2

Please sign in to comment.