README.md

Advanced Platform Development with Kubernetes:

Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning.

Source code from the book Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning by Craig Johnston (@cjimti) ISBN 978-1-4842-5610-7 Apress; 1st ed. edition (September, 2020)

Listings

Chapter 2: DevOps Infrastructure

• Listing 2-1: Cert Manager
• Listing 2-2: Cluster Issuer
• Listing 2-3: Gitlab Namespace
• Listing 2-4: Gitlab TLS Certificate
• Listing 2-5: Gitlab Services Configuration
• Listing 2-6: Gitlab Configuration
• Listing 2-7: Gitlab Deployment
• Listing 2-8: Gitlab Ingress

Chapter 3: Development Environment

• Listing 3-1: Ingress Nginx Namespace
• Listing 3-2: Ingress Nginx RBAC Service Account
• Listing 3-3: RBAC Ingress Nginx Cluster Role
• Listing 3-4: RBAC Ingress Nginx Role
• Listing 3-5: RBAC Ingress Nginx Role Binding
• Listing 3-6: RBAC Ingress Nginx Cluster Role Binding
• Listing 3-7: Ingress Nginx Services
• Listing 3-8: Ingress Nginx Services ConfigMaps
• Listing 3-9: Ingress Nginx Deployment Default Backends
• Listing 3-10: Ingress Nginx DaemonSet
• Listing 3-11: Cert Manager Namespace
• Listing 3-12: Cert Manager Cluster Issuer
• Listing 3-13: CephBlockPool and StorageClass
• Listing 3-14: CephFilesystem
• Listing 3-15: Monitoring README

Chapter 4: In-Platform CI/CD

• Listing 4-1: ServiceAccount and CluserRoleBinding for GitLab
• Listing 4-2: Custom JupyterLab
• Listing 4-3: Python communicating with Prometheus
• Listing 4-4: GitLab CI pipeline configuration
• Listing 4-5: Editing the notebook-testing default Service Account
• Listing 4-6: Source fragment from Dockerfile

Chapter 5: Pipeline

• Listing 5-1: Development Cluster configuration layout
• Listing 5-2: data Namespace
• Listing 5-3: Certificates for the data Namespace
• Listing 5-4: Zookeeper Service
• Listing 5-5: Zookeeper Headless Service
• Listing 5-6: Zookeeper StatefulSet
• Listing 5-7: Kafka Service
• Listing 5-8: Kafka Headless Service
• Listing 5-9: Kafka StatefulSet
• Listing 5-10: Kafka Pod Disruption Budget
• Listing 5-11: Kafka test client Pod
• Listing 5-12: Mosquitto MQTT Service
• Listing 5-13: Mosquitto configuration ConfigMap
• Listing 5-14: Mosquitto Deployment
• Listing 5-15: Data Pipeline Development Cluster configuration layout

Chapter 6: Indexing and Analytics

• Listing 6-1: Installing kernel headers and the rdb kernel module
• Listing 6-2: Development environment prerequisites
• Listing 6-3: cluster-apk8s-dev4 TLS Certificate
• Listing 6-4: Elasticsearch Service
• Listing 6-5: Elasticsearch Statefulset
• Listing 6-6: Logstash Service
• Listing 6-7: Logstash configuration ConfigMap
• Listing 6-8: Logstash pipeline configuration ConfigMap
• Listing 6-9: Logstash Deployment
• Listing 6-10: Kibana Service
• Listing 6-11: Kibana configuration ConfigMap
• Listing 6-12: Kibana Deployment
• Listing 6-13: Kibana Ingress
• Listing 6-14: Keycloak Web Service
• Listing 6-15: Keycloak administrator and keystore credentials
• Listing 6-16: Keycloak deployment
• Listing 6-17: Keycloak Ingress
• Listing 6-18: data-lab Namespace
• Listing 6-19: datalab ServiceAccount
• Listing 6-20: datauser and hub Roles for the data-user Namespace
• Listing 6-21: data-user and hub RoleBindings
• Listing 6-22: JupyterHub Helm values
• Listing 6-23: JupyterHub Ingress
• Listing 6-24: Indexing and analytics development cluster configuration layout

Chapter 7: Data Lake

• Listing 7-1: MinIO operator
• Listing 7-2: MinIO cluster configuration
• Listing 7-3: MinIO cluster Ingress
• Listing 7-4: MinIO Server external connection configuration
• Listing 7-5: Example bucket event notification
• Listing 7-6: Go application: Compressor
• Listing 7-7: Dockerfile for the Compressor application
• Listing 7-8: Update 005-data-lab/000-namespace/07-role.yml
• Listing 7-9 Monitor Job state and cleanup

Chapter 8: Data Warehouses

• Listing 8-1: MySQL operator Namespace
• Listing 8-2: MySQL cluster configuration
• Listing 8-3: Rook Cassandra operator
• Listing 8-4: Rook Cassandra RBAC configuration
• Listing 8-5: Rook Cassandra cluster configuration
• Listing 8-6: Apache Hive configuration template hive-site-template.xml
• Listing 8-7: Apache Hive container entrypoint.sh script
• Listing 8-8: Apache Hive Dockerfile
• Listing 8-10: Apache Hive docker-compose.yml
• Listing 8-11: Apache Hive Service
• Listing 8-12: Apache Hive Deployment
• Listing 8-13: Presto Helm configuration
• Listing 8-14: Presto Ingress

Chapter 9: Routing and Transformation

• Listing 9-1: OpenFaas Ingress
• Listing 9-2: NiFi Headless Service
• Listing 9-3: NiFi StatefulSet
• Listing 9-4: NiFi Ingress
• Listing 9-5: HTTP post an Elasticsearch index template
• Listing 9-6: HTTP post an Elasticsearch Sentiment Analysis query
• Listing 9-7: Example aggregation output from Elasticsearch Sentiment Analysis query

Chapter 10: Platforming Blockchain

• Listing 10-1: Bootnode Service
• Listing 10-2: Bootnode Deployment
• Listing 10-5: Bootnode Registrar Service
• Listing 10-4: Bootnode Deployment
• Listing 10-5: Ethstats Service
• Listing 10-6: Ethstats Secret
• Listing 10-7: Ethstats Deployment
• Listing 10-8: Ethstats Ingress
• Listing 10-9: Geth Secret
• Listing 10-10: Geth ConfigMap
• Listing 10-11: Geth Deployment
• Listing 10-12: Geth transaction node Service
• Listing 10-13: Geth transaction node Deployment
• Listing 10-14: OpenFaaS function for returning details on the last block in the Blockchain

Chapter 11: Platforming AIML

• Listing 11-1: Signal Monitor script ConfigMap
• Listing 11-2: Signal Monitor DaemonSet deployment
• Listing 11-3: MQTT Client Dockerfile
• Listing 11-4: Jolt transformation
• Listing 11-5: ConfigMap with Python data dump script
• Listing 11-6: Python data dump script requirements.txt
• Listing 11-7: Python Dockerfile with requirements
• Listing 11-8: Python data dump hourly CronJob
• Listing 11-9: Updated singleuser configuration for JupyterHub
• Listing 11-10: MLflow container requirements.txt
• Listing 11-11: MLflow Dockerfile
• Listing 11-12: MLflow Service
• Listing 11-13: MLflow StatefulSet
• Listing 11-14: MLflow Ingress
• Listing 11-15: Secret containing s3 (MinIO) config and credentials
• Listing 11-16: SeldonDeployment with Ingress