Skip to content

Commit 67b2ca0

Browse files
authored
[CLOUD-700] Add GCP terraform example (#345)
1 parent 1377bad commit 67b2ca0

File tree

1 file changed

+171
-1
lines changed

1 file changed

+171
-1
lines changed

docs/setup_installation/common/terraform.md

+171-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Hopsworks.ai Terraform Provider
22

3-
[Managed.hopsworks.ai](https://managed.hopsworks.ai) allows users to create and manage their clusters using the [Hopsworks.ai terraform provider](https://registry.terraform.io/providers/logicalclocks/hopsworksai/latest). In this guide, we first provide brief description on how to get started on AWS and AZURE, then we show how to import an existing cluster to be managed by terraform.
3+
[Managed.hopsworks.ai](https://managed.hopsworks.ai) allows users to create and manage their clusters using the [Hopsworks.ai terraform provider](https://registry.terraform.io/providers/logicalclocks/hopsworksai/latest). In this guide, we first provide brief description on how to get started on AWS, AZURE, and GCP, then we show how to import an existing cluster to be managed by terraform.
44

55
## Getting Started with AWS
66

@@ -242,6 +242,175 @@ terraform apply -var="resource_group=<YOUR_RESOURCE_GROUP>"
242242
terraform destroy -var="resource_group=<YOUR_RESOURCE_GROUP>"
243243
```
244244

245+
## Getting Started with GCP
246+
247+
Complete the following steps to start using Hopsworks.ai Terraform Provider on GCP.
248+
249+
1. Create a [managed.hopsworks.ai](https://managed.hopsworks.ai) API KEY as described in details [here](api_key.md), and export the API KEY as follows
250+
```bash
251+
export HOPSWORKSAI_API_KEY=<YOUR_API_KEY>
252+
```
253+
2. Download the proper Terraform CLI for your os from [here](https://www.terraform.io/downloads.html).
254+
3. Install the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install-sdk) and run `gcloud init` to configure your GCP credentials.
255+
256+
### Example
257+
In this section, we provide a simple example to create a Hopsworks cluster on GCP along with all its required resources (Google storage bucket and service account with the required permissions).
258+
259+
1. In your terminal, run the following to create a demo directory and cd to it
260+
```bash
261+
mkdir demo
262+
cd demo
263+
```
264+
2. In this empty directory, create an empty file `main.tf`. Open the file and paste the following configurations to it then save it.
265+
```hcl
266+
terraform {
267+
required_version = ">= 0.14.0"
268+
269+
required_providers {
270+
google = {
271+
source = "hashicorp/google"
272+
version = "5.13.0"
273+
}
274+
hopsworksai = {
275+
source = "logicalclocks/hopsworksai"
276+
}
277+
time = {
278+
source = "hashicorp/time"
279+
version = "0.10.0"
280+
}
281+
}
282+
}
283+
284+
variable "region" {
285+
type = string
286+
default = "europe-north1"
287+
}
288+
289+
variable "project" {
290+
type = string
291+
}
292+
293+
provider "google" {
294+
region = var.region
295+
project = var.project
296+
}
297+
298+
provider "hopsworksai" {
299+
}
300+
301+
provider "time" {
302+
303+
}
304+
305+
# Create required google resources, a storage bucket and an service account with the required hopsworks permissions
306+
data "hopsworksai_gcp_service_account_custom_role_permissions" "service_account" {
307+
308+
}
309+
310+
resource "google_project_iam_custom_role" "service_account_role" {
311+
role_id = "tf.HopsworksAIInstances"
312+
title = "Hopsworks AI Instances"
313+
description = "Role that allows Hopsworks AI Instances to access resources"
314+
permissions = data.hopsworksai_gcp_service_account_custom_role_permissions.service_account.permissions
315+
}
316+
317+
resource "google_service_account" "service_account" {
318+
account_id = "tf-hopsworks-ai-instances"
319+
display_name = "Hopsworks AI instances"
320+
description = "Service account for Hopsworks AI instances"
321+
}
322+
323+
resource "google_project_iam_binding" "service_account_role_binding" {
324+
project = var.project
325+
role = google_project_iam_custom_role.service_account_role.id
326+
327+
members = [
328+
google_service_account.service_account.member
329+
]
330+
}
331+
332+
resource "google_storage_bucket" "bucket" {
333+
name = "tf-hopsworks-bucket"
334+
location = var.region
335+
force_destroy = true
336+
}
337+
338+
resource "time_sleep" "wait_60_seconds" {
339+
depends_on = [google_project_iam_binding.service_account_role_binding]
340+
create_duration = "60s"
341+
}
342+
343+
# Create a simple cluster with two workers with two different configuration
344+
data "google_compute_zones" "available" {
345+
region = var.region
346+
}
347+
348+
locals {
349+
zone = data.google_compute_zones.available.names.0
350+
}
351+
352+
resource "hopsworksai_cluster" "cluster" {
353+
name = "tf-cluster"
354+
355+
head {
356+
instance_type = "e2-standard-8"
357+
}
358+
359+
gcp_attributes {
360+
project_id = var.project
361+
region = var.region
362+
zone = local.zone
363+
service_account_email = google_service_account.service_account.email
364+
bucket {
365+
name = google_storage_bucket.bucket.name
366+
}
367+
}
368+
369+
rondb {
370+
single_node {
371+
instance_type = "e2-highmem-4"
372+
}
373+
}
374+
375+
autoscale {
376+
non_gpu_workers {
377+
instance_type = "e2-standard-8"
378+
disk_size = 256
379+
min_workers = 1
380+
max_workers = 5
381+
standby_workers = 0.5
382+
downscale_wait_time = 300
383+
}
384+
}
385+
386+
open_ports {
387+
ssh = true
388+
}
389+
390+
# waiting for 60 seconds after service account permissions has been granted
391+
# to avoid permissions validation failure on hopsworks when creating the cluster
392+
depends_on = [time_sleep.wait_60_seconds]
393+
}
394+
395+
output "hopsworks_cluster_url" {
396+
value = hopsworksai_cluster.cluster.url
397+
}
398+
```
399+
3. Initialize the terraform directory by running the following command
400+
```bash
401+
terraform init
402+
```
403+
4. Now you can apply the changes to create all required resources. Replace the placeholders with your GCP project id
404+
```bash
405+
terraform apply -var="project=<YOUR_PROJECT_ID>"
406+
```
407+
5. Once terraform finishes creating the resources, it will output the url to the newly created cluster. Notice that for now, you have to navigate to your [managed.hopsworks.ai dashboard](https://managed.hopsworks.ai/dashboard) to get your login credentials.
408+
409+
6. After you finish working with the cluster, you can terminate it along with the other GCP resources using the following command
410+
```bash
411+
terraform destroy -var="project=<YOUR_PROJECT_ID>"
412+
```
413+
245414
## Importing an existing cluster to terraform
246415

247416
In this section, we show how to use `terraform import` to manage your existing Hopsworks cluster.
@@ -393,3 +562,4 @@ actions need to be performed.
393562
* Check the [Hopsworks.ai terraform provider documentation](https://registry.terraform.io/providers/logicalclocks/hopsworksai/latest/docs) for more details about the different resources and data sources supported by the provider and a description of their attributes.
394563
* Check the [Hopsworks.ai terraform AWS examples](https://github.com/logicalclocks/terraform-provider-hopsworksai/tree/main/examples/complete/aws), each example contains a README file describing how to run it and more details about configuring it.
395564
* Check the [Hopsworks.ai terraform AZURE examples](https://github.com/logicalclocks/terraform-provider-hopsworksai/tree/main/examples/complete/azure), each example contains a README file describing how to run it and more details about configuring it.
565+
* Check the [Hopsworks.ai terraform GCP examples](https://github.com/logicalclocks/terraform-provider-hopsworksai/tree/main/examples/complete/gcp), each example contains a README file describing how to run it and more details about configuring it.

0 commit comments

Comments
 (0)