[TASK] Set Up Grafana and Prometheus for Monitoring

[walidoow]

Task: Set Up Grafana and Prometheus for Monitoring

Description

Set up Grafana and Prometheus for monitoring the Kubernetes cluster and microservices to track performance, resource usage, and health metrics.

Priority: P2
Risk: High

Acceptance Criteria:

• Prometheus is installed and configured to scrape metrics from the Kubernetes cluster and microservices.
• Grafana is installed and configured to visualize metrics from Prometheus.
• Default dashboards are created for cluster monitoring (e.g., CPU, memory, pod statuses) and microservices metrics.
• Alerts are configured in Prometheus for critical thresholds.
• Documentation is provided for maintaining and extending monitoring.

Task Breakdown:

1. Prometheus Setup:

   • Install Prometheus in the Kubernetes cluster using Helm
   • Configure Prometheus to scrape metrics from:
     • Kubernetes components (API server, kubelet, etc.).
     • Microservices exposing metrics (e.g., using Spring Boot Actuator or custom metrics).
   • Expose Prometheus service for internal or external access.

2. Grafana Setup:

   • Install Grafana in the Kubernetes cluster using Helm
   • Configure Grafana to use Prometheus as a data source.
   • Secure Grafana with authentication (e.g., admin credentials or SSO).

3. Dashboard Creation:

   • Import default Kubernetes monitoring dashboards into Grafana (available from Grafana Marketplace).
   • Create or import dashboards for microservices metrics, such as:
     • Request counts, latency, and error rates.
     • Resource usage (CPU, memory) per service.

4. Alerting Setup:

   • Define alerting rules in Prometheus for critical thresholds (e.g., high CPU usage, pod restarts).
   • Integrate alert notifications with email, Slack, or other channels via Alertmanager.

5. Testing:

   • Verify Prometheus is scraping metrics from the cluster and microservices.
   • Validate that Grafana dashboards display correct metrics.
   • Test alerting functionality by simulating threshold breaches.

6. Documentation:

   • Document the setup process for Prometheus and Grafana.
   • Include instructions for adding new microservices or custom metrics.
   • Provide guidance on extending dashboards and managing alerts.