Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning.
Source code from the book Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning by Craig Johnston (@cjimti) ISBN 978-1-4842-5610-7 Apress; 1st ed. edition (September, 2020)
Chapter 2: DevOps Infrastructure
- Listing 2-1: Cert Manager
- Listing 2-2: Cluster Issuer
- Listing 2-3: Gitlab Namespace
- Listing 2-4: Gitlab TLS Certificate
- Listing 2-5: Gitlab Services Configuration
- Listing 2-6: Gitlab Configuration
- Listing 2-7: Gitlab Deployment
- Listing 2-8: Gitlab Ingress
Chapter 3: Development Environment
- Listing 3-1: Ingress Nginx Namespace
- Listing 3-2: Ingress Nginx RBAC Service Account
- Listing 3-3: RBAC Ingress Nginx Cluster Role
- Listing 3-4: RBAC Ingress Nginx Role
- Listing 3-5: RBAC Ingress Nginx Role Binding
- Listing 3-6: RBAC Ingress Nginx Cluster Role Binding
- Listing 3-7: Ingress Nginx Services
- Listing 3-8: Ingress Nginx Services ConfigMaps
- Listing 3-9: Ingress Nginx Deployment Default Backends
- Listing 3-10: Ingress Nginx DaemonSet
- Listing 3-11: Cert Manager Namespace
- Listing 3-12: Cert Manager Cluster Issuer
- Listing 3-13: CephBlockPool and StorageClass
- Listing 3-14: CephFilesystem
- Listing 3-15: Monitoring README
Chapter 4: In-Platform CI/CD
- Listing 4-1: ServiceAccount and CluserRoleBinding for GitLab
- Listing 4-2: Custom JupyterLab
- Listing 4-3: Python communicating with Prometheus
- Listing 4-4: GitLab CI pipeline configuration
- Listing 4-5: Editing the notebook-testing default Service Account
- Listing 4-6: Source fragment from Dockerfile
Chapter 5: Pipeline
- Listing 5-1: Development Cluster configuration layout
- Listing 5-2: data Namespace
- Listing 5-3: Certificates for the data Namespace
- Listing 5-4: Zookeeper Service
- Listing 5-5: Zookeeper Headless Service
- Listing 5-6: Zookeeper StatefulSet
- Listing 5-7: Kafka Service
- Listing 5-8: Kafka Headless Service
- Listing 5-9: Kafka StatefulSet
- Listing 5-10: Kafka Pod Disruption Budget
- Listing 5-11: Kafka test client Pod
- Listing 5-12: Mosquitto MQTT Service
- Listing 5-13: Mosquitto configuration ConfigMap
- Listing 5-14: Mosquitto Deployment
- Listing 5-15: Data Pipeline Development Cluster configuration layout
Chapter 6: Indexing and Analytics
- Listing 6-1: Installing kernel headers and the rdb kernel module
- Listing 6-2: Development environment prerequisites
- Listing 6-3: cluster-apk8s-dev4 TLS Certificate
- Listing 6-4: Elasticsearch Service
- Listing 6-5: Elasticsearch Statefulset
- Listing 6-6: Logstash Service
- Listing 6-7: Logstash configuration ConfigMap
- Listing 6-8: Logstash pipeline configuration ConfigMap
- Listing 6-9: Logstash Deployment
- Listing 6-10: Kibana Service
- Listing 6-11: Kibana configuration ConfigMap
- Listing 6-12: Kibana Deployment
- Listing 6-13: Kibana Ingress
- Listing 6-14: Keycloak Web Service
- Listing 6-15: Keycloak administrator and keystore credentials
- Listing 6-16: Keycloak deployment
- Listing 6-17: Keycloak Ingress
- Listing 6-18: data-lab Namespace
- Listing 6-19: datalab ServiceAccount
- Listing 6-20: datauser and hub Roles for the data-user Namespace
- Listing 6-21: data-user and hub RoleBindings
- Listing 6-22: JupyterHub Helm values
- Listing 6-23: JupyterHub Ingress
- Listing 6-24: Indexing and analytics development cluster configuration layout
Chapter 7: Data Lake
- Listing 7-1: MinIO operator
- Listing 7-2: MinIO cluster configuration
- Listing 7-3: MinIO cluster Ingress
- Listing 7-4: MinIO Server external connection configuration
- Listing 7-5: Example bucket event notification
- Listing 7-6: Go application: Compressor
- Listing 7-7: Dockerfile for the Compressor application
- Listing 7-8: Update
005-data-lab/000-namespace/07-role.yml - Listing 7-9 Monitor Job state and cleanup
Chapter 8: Data Warehouses
- Listing 8-1: MySQL operator Namespace
- Listing 8-2: MySQL cluster configuration
- Listing 8-3: Rook Cassandra operator
- Listing 8-4: Rook Cassandra RBAC configuration
- Listing 8-5: Rook Cassandra cluster configuration
- Listing 8-6: Apache Hive configuration template
hive-site-template.xml - Listing 8-7: Apache Hive container
entrypoint.shscript - Listing 8-8: Apache Hive
Dockerfile - Listing 8-10: Apache Hive
docker-compose.yml - Listing 8-11: Apache Hive Service
- Listing 8-12: Apache Hive Deployment
- Listing 8-13: Presto Helm configuration
- Listing 8-14: Presto Ingress
Chapter 9: Routing and Transformation
- Listing 9-1: OpenFaas Ingress
- Listing 9-2: NiFi Headless Service
- Listing 9-3: NiFi StatefulSet
- Listing 9-4: NiFi Ingress
- Listing 9-5: HTTP post an Elasticsearch index template
- Listing 9-6: HTTP post an Elasticsearch Sentiment Analysis query
- Listing 9-7: Example aggregation output from Elasticsearch Sentiment Analysis query
Chapter 10: Platforming Blockchain
- Listing 10-1: Bootnode Service
- Listing 10-2: Bootnode Deployment
- Listing 10-5: Bootnode Registrar Service
- Listing 10-4: Bootnode Deployment
- Listing 10-5: Ethstats Service
- Listing 10-6: Ethstats Secret
- Listing 10-7: Ethstats Deployment
- Listing 10-8: Ethstats Ingress
- Listing 10-9: Geth Secret
- Listing 10-10: Geth ConfigMap
- Listing 10-11: Geth Deployment
- Listing 10-12: Geth transaction node Service
- Listing 10-13: Geth transaction node Deployment
- Listing 10-14: OpenFaaS function for returning details on the last block in the Blockchain
Chapter 11: Platforming AIML
- Listing 11-1: Signal Monitor script ConfigMap
- Listing 11-2: Signal Monitor DaemonSet deployment
- Listing 11-3: MQTT Client Dockerfile
- Listing 11-4: Jolt transformation
- Listing 11-5: ConfigMap with Python data dump script
- Listing 11-6: Python data dump script
requirements.txt - Listing 11-7: Python
Dockerfilewith requirements - Listing 11-8: Python data dump hourly CronJob
- Listing 11-9: Updated
singleuserconfiguration for JupyterHub - Listing 11-10: MLflow container
requirements.txt - Listing 11-11: MLflow
Dockerfile - Listing 11-12: MLflow Service
- Listing 11-13: MLflow StatefulSet
- Listing 11-14: MLflow Ingress
- Listing 11-15: Secret containing s3 (MinIO) config and credentials
- Listing 11-16: SeldonDeployment with Ingress