grafana Posts

Exposing Openshift Prometheus API and Display it on External Monitoring Tools

Theres one question comes up during discussion with my colleague regarding on how we can monitor our application which are being deployed on top of Openshift. Actually Openshift has its own monitoring tools, but sometimes we need an external monitoring tools for monitor our distributed application especially when deployed in a multiple different clusters of Openshift.

In the end, the high level concept is pretty much like this.

But first in order to achieve it we need to make sure that our thanos-querier are both accesible by External Grafana, and also secure.

Before we go there, lets start by creating an “mw” namespace first and deploying a simple java apps there.

oc new-project mw

oc new-app registry.access.redhat.com/ubi8/openjdk-8~https://github.com/edwin/hello-world-fuse-on-ocp -n mw

Create a new serviceaccount,

oc create sa ext-monitor -n mw

And gives a “cluster-monitoring-view” role to it,

oc adm policy add-cluster-role-to-user cluster-monitoring-view -z ext-monitor -n mw

Next step is getting the ServiceAccount JWT token by using below command,

oc sa get-token ext-monitor -n mw

It will generate something like this, and save it somewhere.

Next is setuping our own External Monitoring tools by using grafana, and login with admin/admin credential.

docker pull grafana/grafana

docker run -d -p 3000:3000 grafana/grafana

Create a new Data sources, and select Prometheus as our new Datasource.

Fill in some data, and put our thanos-querier as our HTTP URL.

Create new HTTP Header, and put Authorization as the Key. And put “Bearer (your ServiceAccount JWT token)” as the value. We can add some custom query parameters for defining which namespace to be monitored.

Press Save and Test button after.

Next step is creating a dashboard,

And an empty Panel,

Change our Data source into our newly created Data source, and run below query in Metric Browser field

sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{namespace='mw'}) by (pod)

The result should be seen on below image,

Fyi, on this tutorial Im using Openshift 4.8.

Installing Prometheus and Grafana on Top of Openshift 4

Creating and deploying application on top of Openshift is one thing, but doing health monitoring for those applications is a whole new different thing. Thats why on this session im trying to share on how to deploy Prometheus, for monitoring appliction statistics, and Grafana, for visualizing it, to support the day-to-day activity and operations.

So before started too far, lets start with a brief introduction on what is Prometheus and Grafana.

According to its site (https://prometheus.io/), Prometheus is an open-source systems monitoring and alerting toolkit. It has multiple visualization tools and Grafana as one of it. On Grafana’s website, https://grafana.com/, we can see that it is an open source visualization and analytics software, which allows us to query, visualize, alerting and explore our metrics freely.

Okay, there are multiple ways of installing those two items. One way is by using Openshift Operator, and another way is installed it manually either by using template, or a simple container image installation. For this time, we are using the last approach, and that is container image deployment.

Lets start by installing Prometheus first,

for this example, im using openshift/prometheus image from docker hub.

Once image has been created, next is creating a configuration file for storing list of prometheus api needed to be scrape from, for this example im scrapping statistics from application App01 which located on project Project01 on top of the same OCP4 instance.

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets: []
    scheme: http
    timeout: 10s
scrape_configs:
- job_name: project01
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - app01.project01.svc.cluster.local:8080
  basic_auth:
      username: dev01
      password: password

Save it with name “prometheus.yml”, and push it to OCP4 secret by using below command

oc create cm prometheus-config --from-file=prometheus.yaml

And mount it to Prometheus’s DeploymentConfig

oc volume dc/prometheus --add --name=prometheus-config --type=configmap --configmap-name=prometheus-config --mount-path=/etc/prometheus/

Expose your installed Prometheus service, and access it directly thru browser. A successfully installation will shows a target end point like this,

Next step is to install Grafana. The same installation methodology is being use, and that is using container image deployment. Im using grafana/grafana:6.0.1 image from DockerHub for this example.

Login by using admin/admin credentials,

Set the datasource as prometheus,

Add prometheus url and scrape interval, and press Save button

We can start by creating a Grafana dashboard based on Prometheus statistics,

There is other approach of installating Prometheus and Grafana, and that is by using yml template. I cover those approach on my Github page,

https://github.com/edwin/prometheus-and-grafana-openshift4-template-yml

Have fun.