|
1 | 1 | # Hopsworks.ai Terraform Provider
|
2 | 2 |
|
3 |
| -[Managed.hopsworks.ai](https://managed.hopsworks.ai) allows users to create and manage their clusters using the [Hopsworks.ai terraform provider](https://registry.terraform.io/providers/logicalclocks/hopsworksai/latest). In this guide, we first provide brief description on how to get started on AWS and AZURE, then we show how to import an existing cluster to be managed by terraform. |
| 3 | +[Managed.hopsworks.ai](https://managed.hopsworks.ai) allows users to create and manage their clusters using the [Hopsworks.ai terraform provider](https://registry.terraform.io/providers/logicalclocks/hopsworksai/latest). In this guide, we first provide brief description on how to get started on AWS, AZURE, and GCP, then we show how to import an existing cluster to be managed by terraform. |
4 | 4 |
|
5 | 5 | ## Getting Started with AWS
|
6 | 6 |
|
@@ -242,6 +242,175 @@ terraform apply -var="resource_group=<YOUR_RESOURCE_GROUP>"
|
242 | 242 | terraform destroy -var="resource_group=<YOUR_RESOURCE_GROUP>"
|
243 | 243 | ```
|
244 | 244 |
|
| 245 | +## Getting Started with GCP |
| 246 | + |
| 247 | +Complete the following steps to start using Hopsworks.ai Terraform Provider on GCP. |
| 248 | + |
| 249 | +1. Create a [managed.hopsworks.ai](https://managed.hopsworks.ai) API KEY as described in details [here](api_key.md), and export the API KEY as follows |
| 250 | +```bash |
| 251 | +export HOPSWORKSAI_API_KEY=<YOUR_API_KEY> |
| 252 | +``` |
| 253 | +2. Download the proper Terraform CLI for your os from [here](https://www.terraform.io/downloads.html). |
| 254 | +3. Install the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install-sdk) and run `gcloud init` to configure your GCP credentials. |
| 255 | + |
| 256 | +### Example |
| 257 | +In this section, we provide a simple example to create a Hopsworks cluster on GCP along with all its required resources (Google storage bucket and service account with the required permissions). |
| 258 | + |
| 259 | +1. In your terminal, run the following to create a demo directory and cd to it |
| 260 | +```bash |
| 261 | +mkdir demo |
| 262 | +cd demo |
| 263 | +``` |
| 264 | +2. In this empty directory, create an empty file `main.tf`. Open the file and paste the following configurations to it then save it. |
| 265 | +```hcl |
| 266 | +terraform { |
| 267 | + required_version = ">= 0.14.0" |
| 268 | +
|
| 269 | + required_providers { |
| 270 | + google = { |
| 271 | + source = "hashicorp/google" |
| 272 | + version = "5.13.0" |
| 273 | + } |
| 274 | + hopsworksai = { |
| 275 | + source = "logicalclocks/hopsworksai" |
| 276 | + } |
| 277 | + time = { |
| 278 | + source = "hashicorp/time" |
| 279 | + version = "0.10.0" |
| 280 | + } |
| 281 | + } |
| 282 | +} |
| 283 | +
|
| 284 | +variable "region" { |
| 285 | + type = string |
| 286 | + default = "europe-north1" |
| 287 | +} |
| 288 | +
|
| 289 | +variable "project" { |
| 290 | + type = string |
| 291 | +} |
| 292 | +
|
| 293 | +provider "google" { |
| 294 | + region = var.region |
| 295 | + project = var.project |
| 296 | +} |
| 297 | +
|
| 298 | +provider "hopsworksai" { |
| 299 | +} |
| 300 | +
|
| 301 | +provider "time" { |
| 302 | +
|
| 303 | +} |
| 304 | +
|
| 305 | +# Create required google resources, a storage bucket and an service account with the required hopsworks permissions |
| 306 | +data "hopsworksai_gcp_service_account_custom_role_permissions" "service_account" { |
| 307 | +
|
| 308 | +} |
| 309 | +
|
| 310 | +resource "google_project_iam_custom_role" "service_account_role" { |
| 311 | + role_id = "tf.HopsworksAIInstances" |
| 312 | + title = "Hopsworks AI Instances" |
| 313 | + description = "Role that allows Hopsworks AI Instances to access resources" |
| 314 | + permissions = data.hopsworksai_gcp_service_account_custom_role_permissions.service_account.permissions |
| 315 | +} |
| 316 | +
|
| 317 | +resource "google_service_account" "service_account" { |
| 318 | + account_id = "tf-hopsworks-ai-instances" |
| 319 | + display_name = "Hopsworks AI instances" |
| 320 | + description = "Service account for Hopsworks AI instances" |
| 321 | +} |
| 322 | +
|
| 323 | +resource "google_project_iam_binding" "service_account_role_binding" { |
| 324 | + project = var.project |
| 325 | + role = google_project_iam_custom_role.service_account_role.id |
| 326 | +
|
| 327 | + members = [ |
| 328 | + google_service_account.service_account.member |
| 329 | + ] |
| 330 | +} |
| 331 | +
|
| 332 | +resource "google_storage_bucket" "bucket" { |
| 333 | + name = "tf-hopsworks-bucket" |
| 334 | + location = var.region |
| 335 | + force_destroy = true |
| 336 | +} |
| 337 | +
|
| 338 | +resource "time_sleep" "wait_60_seconds" { |
| 339 | + depends_on = [google_project_iam_binding.service_account_role_binding] |
| 340 | + create_duration = "60s" |
| 341 | +} |
| 342 | +
|
| 343 | +# Create a simple cluster with two workers with two different configuration |
| 344 | +data "google_compute_zones" "available" { |
| 345 | + region = var.region |
| 346 | +} |
| 347 | +
|
| 348 | +locals { |
| 349 | + zone = data.google_compute_zones.available.names.0 |
| 350 | +} |
| 351 | +
|
| 352 | +resource "hopsworksai_cluster" "cluster" { |
| 353 | + name = "tf-cluster" |
| 354 | +
|
| 355 | + head { |
| 356 | + instance_type = "e2-standard-8" |
| 357 | + } |
| 358 | +
|
| 359 | + gcp_attributes { |
| 360 | + project_id = var.project |
| 361 | + region = var.region |
| 362 | + zone = local.zone |
| 363 | + service_account_email = google_service_account.service_account.email |
| 364 | + bucket { |
| 365 | + name = google_storage_bucket.bucket.name |
| 366 | + } |
| 367 | + } |
| 368 | +
|
| 369 | + rondb { |
| 370 | + single_node { |
| 371 | + instance_type = "e2-highmem-4" |
| 372 | + } |
| 373 | + } |
| 374 | +
|
| 375 | + autoscale { |
| 376 | + non_gpu_workers { |
| 377 | + instance_type = "e2-standard-8" |
| 378 | + disk_size = 256 |
| 379 | + min_workers = 1 |
| 380 | + max_workers = 5 |
| 381 | + standby_workers = 0.5 |
| 382 | + downscale_wait_time = 300 |
| 383 | + } |
| 384 | + } |
| 385 | +
|
| 386 | + open_ports { |
| 387 | + ssh = true |
| 388 | + } |
| 389 | +
|
| 390 | + # waiting for 60 seconds after service account permissions has been granted |
| 391 | + # to avoid permissions validation failure on hopsworks when creating the cluster |
| 392 | + depends_on = [time_sleep.wait_60_seconds] |
| 393 | +} |
| 394 | +
|
| 395 | +output "hopsworks_cluster_url" { |
| 396 | + value = hopsworksai_cluster.cluster.url |
| 397 | +} |
| 398 | +``` |
| 399 | +3. Initialize the terraform directory by running the following command |
| 400 | +```bash |
| 401 | +terraform init |
| 402 | +``` |
| 403 | +4. Now you can apply the changes to create all required resources. Replace the placeholders with your GCP project id |
| 404 | +```bash |
| 405 | +terraform apply -var="project=<YOUR_PROJECT_ID>" |
| 406 | +``` |
| 407 | +5. Once terraform finishes creating the resources, it will output the url to the newly created cluster. Notice that for now, you have to navigate to your [managed.hopsworks.ai dashboard](https://managed.hopsworks.ai/dashboard) to get your login credentials. |
| 408 | + |
| 409 | +6. After you finish working with the cluster, you can terminate it along with the other GCP resources using the following command |
| 410 | +```bash |
| 411 | +terraform destroy -var="project=<YOUR_PROJECT_ID>" |
| 412 | +``` |
| 413 | + |
245 | 414 | ## Importing an existing cluster to terraform
|
246 | 415 |
|
247 | 416 | In this section, we show how to use `terraform import` to manage your existing Hopsworks cluster.
|
@@ -393,3 +562,4 @@ actions need to be performed.
|
393 | 562 | * Check the [Hopsworks.ai terraform provider documentation](https://registry.terraform.io/providers/logicalclocks/hopsworksai/latest/docs) for more details about the different resources and data sources supported by the provider and a description of their attributes.
|
394 | 563 | * Check the [Hopsworks.ai terraform AWS examples](https://github.com/logicalclocks/terraform-provider-hopsworksai/tree/main/examples/complete/aws), each example contains a README file describing how to run it and more details about configuring it.
|
395 | 564 | * Check the [Hopsworks.ai terraform AZURE examples](https://github.com/logicalclocks/terraform-provider-hopsworksai/tree/main/examples/complete/azure), each example contains a README file describing how to run it and more details about configuring it.
|
| 565 | +* Check the [Hopsworks.ai terraform GCP examples](https://github.com/logicalclocks/terraform-provider-hopsworksai/tree/main/examples/complete/gcp), each example contains a README file describing how to run it and more details about configuring it. |
0 commit comments