Add Raft Node Support for Health Checks and Metrics Collection #643
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add functionality to monitor Raft node health status and expose key metrics: - Add new Prometheus metrics for Raft health status, leader status, and last contact latency - Implement GetHealthStatus() method to track node health state - Monitor leadership status and communication with leader - Track last contact time with leader - Add helper functions for metric value conversion and time parsing The metrics will help monitor cluster health and leadership changes in production.
- Added Raft node dependency to `Options` and `HealthChecker` structs. - Updated `liveness` function to include Raft node health status. - Modified health check logic in `healthcheck.go` and `http_server.go` to consider Raft node status. - Enhanced `healthcheck_test.go` to include tests for Raft node integration. - Ensured Raft node is properly initialized and cleaned up in tests.
- Introduced a new test `TestGetHealthStatus` to check the health status of Raft nodes. - The test covers three scenarios: 1. When the node is the leader, it should be healthy and recognize itself as the leader. 2. When the node is a follower, it should be healthy and recognize the leader. 3. When no leader is available, the node should not be healthy and should report an error. - Utilized temporary directories and a test logger for setup. - Ensured proper shutdown of nodes after tests to prevent resource leaks.
- Corrected variable naming from `leaderId` to `leaderID` for consistency. - Added missing periods to comments for proper punctuation. - Used `%w` in `fmt.Errorf` for error wrapping.
- Updated the `runCmd` in `cmd/run.go` to include `RaftNode` in the API configuration. - This change ensures that the RaftNode is properly initialized and passed to the API object, which may be necessary for distributed consensus or state management.
- Added port mappings for 9090 to each gatewayd service. - Introduced health checks for gatewayd-2 and gatewayd-3 services to ensure they are running correctly. - Added service dependencies for write-postgres, read-postgres, redis, and install_plugins to ensure proper startup order. - Linked gatewayd-2 and gatewayd-3 services to write-postgres, read-postgres, and redis for improved connectivity.
- Added a comment to document the GetHealthStatus method, explaining its purpose. - Removed an unnecessary comment about caching commonly used values, as it was redundant with the code's functionality.
Overview
Packages and Vulnerabilities (6 package changes and 1 vulnerability changes)
Changes for packages of type
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Package | Versionghcr.io/gatewayd-io/gatewayd:ed25ae1 |
Versiongatewaydio/gatewayd:latest |
|
|---|---|---|---|
| ➖ | ca-certificates | 20240705-r0 |
|
| ➖ | openssl | 3.3.2-r0 |
|
| ➖ | pax-utils | 1.3.7-r2 |
Changes for packages of type golang (3 changes)
| Package | Versionghcr.io/gatewayd-io/gatewayd:ed25ae1 |
Versiongatewaydio/gatewayd:latest |
|
|---|---|---|---|
| ♾️ | github.com/gatewayd-io/gatewayd | (devel) |
0.0.0-20241214123014-b6df86a6fe94 |
| ♾️ | golang.org/x/net | 0.33.0 |
0.32.0 |
    |
|||
| Added vulnerabilities (1): |
|||
| ♾️ | stdlib | go1.23.4 |
1.23.4 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket(s)
Description
This pull request addresses the Raft Health Check Integration as part of the enhancements needed for the Raft cluster management. The following changes have been made:
livenessfunction to incorporate Raft node health checks.docker-compose-raft.yamlto include health checks for services.Development Checklist
make gen-docscommand.Legal Checklist