Skip to content

Using Datadog and Prometheus with Gluu Cloud Native edition#

Prerequisites#

  • Datadog Agent 6.5.0+

Instructions#

  1. Install Gluu server using one of the following
  2. Kustomize
  3. Helm

  4. Create a directory called monitoring for the manifests needed.

    mkdir monitoring && cd monitoring
    
  5. Get zip file of manifests and unzip

    wget https://github.com/GluuFederation/cloud-native-edition/raw/4.1/monitoring.zip
    unzip monitoring.zip
    
  6. Create a namespace for Datadog and Prometheus

    kubectl create ns datadog
    kubectl create ns prometheus
    

Using Kubectl commands#

  1. Configure RBAC permissions for the datadog agent in datadog namespace.

    kubectl create -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/clusterrole.yaml" -ns datadog
    
    kubectl create -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/serviceaccount.yaml" -ns datadog
    
    kubectl create -f "https://raw.githubusercontent.com/DataDog/datadog-agent/master/Dockerfiles/manifests/rbac/serviceaccount.yaml" -ns datadog
    
  2. Create a Secret containing datadog API key. It will be used in datadog agent Daemonset.

    Warning

    If the name of the secret below is to be changed , change it in the respective DaemonSet as well.

    kubectl create secret generic datadog-secret --from-literal api-key="<API-KEY>"
    
  3. Create a Datadog agent with custom metrics and APM logs collection enabled.

    kubectl apply -f datadog/
    
  4. Install Prometheus

    kubectl apply -f prometheus/
    

    Note

    For datadog and prometheus integration, prometheus deployment has the annotations that allow datadog autodiscovery feature and gets all the metrics that prometheus collects from the cluster.

          annotations:
              ad.datadoghq.com/prometheus.check_names: |
                ["openmetrics"]
              ad.datadoghq.com/prometheus.init_configs: |
                [{}]
              ad.datadoghq.com/prometheus.instances: |
                [
                  {
                    "prometheus_url": "http://%%host%%:%%port%%/metrics",
                    "namespace": "monitoring",
                    "metrics": [ {"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]
                  }
                ]
    
  5. [Optional] Install kube state metrics which exposes all the metrics on /metrics URI. Prometheus can scrape all the metrics exposed by kube state metrics. This will be created in the kube-system namespace.

    kubectl apply -f kube-state-metrics/
    
    1. Add the following configuration as part of prometheus job configuration.
    - job_name: 'kube-state-metrics'
      static_configs:
        - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
    

    Note

    This part has been included in by default in proth-cm. If it is not being used, it should be removed. If not it will cause health check error in prometheus targets.

Using helm#

Prerequisites#

  • Helm 2.10+
  • Tiller

Datadog#

  1. Install Datadog

    helm install --name datadog-agent-v1 \
      --set datadog.apiKey=<DataDog API Key> \
      --set datadog.apmEnabled=true \
      --set datadog.logsEnabled=true \
      stable/datadog
      --namespace datadog
    
  2. Install prometheus

    Note

    The custom values.yaml file above has the annotations needed to integrate datadog with prometheus.

    helm install --name prometheus --namespace prometheus -f helm/prometheus-values.yaml stable/prometheus
    
    1. Access prometheus UI via port-forwarding or switch prometheus service upon installation to LoadBalancer

      export PROMETHEUS_POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
      kubectl --namespace prometheus port-forward PROMETHEUS_POD_NAME 9090
      
  3. [Optional] Install kube state metrics which exposes all the metrics on /metrics URI. Prometheus can scrape all the metrics exposed by kube state metrics. This will be created in the kube-system namespace.

    helm2 install --name kube-state stable/kube-state-metrics
    
    1. Add the kube-state metrics job in prometheus serverFiles.
          - job_name: 'kube-state-metrics'
            static_configs:
              - targets: ['kube-state-kube-state-metrics.default.svc.cluster.local:8080']
    

    Note

    All the metrics will be exported to datadog metrics -> summary