Exposing Openshift Prometheus API and Display it on External Monitoring Tools

Theres one question comes up during discussion with my colleague regarding on how we can monitor our application which are being deployed on top of Openshift. Actually Openshift has its own monitoring tools, but sometimes we need an external monitoring tools for monitor our distributed application especially when deployed in a multiple different clusters of Openshift.

In the end, the high level concept is pretty much like this.

But first in order to achieve it we need to make sure that our thanos-querier are both accesible by External Grafana, and also secure.

Before we go there, lets start by creating an “mw” namespace first and deploying a simple java apps there.

oc new-project mw

oc new-app registry.access.redhat.com/ubi8/openjdk-8~https://github.com/edwin/hello-world-fuse-on-ocp -n mw

Create a new serviceaccount,

oc create sa ext-monitor -n mw

And gives a “cluster-monitoring-view” role to it,

oc adm policy add-cluster-role-to-user cluster-monitoring-view -z ext-monitor -n mw

Next step is getting the ServiceAccount JWT token by using below command,

oc sa get-token ext-monitor -n mw

It will generate something like this, and save it somewhere.

Next is setuping our own External Monitoring tools by using grafana, and login with admin/admin credential.

docker pull grafana/grafana

docker run -d -p 3000:3000 grafana/grafana

Create a new Data sources, and select Prometheus as our new Datasource.

Fill in some data, and put our thanos-querier as our HTTP URL.

Create new HTTP Header, and put Authorization as the Key. And put “Bearer (your ServiceAccount JWT token)” as the value. We can add some custom query parameters for defining which namespace to be monitored.

Press Save and Test button after.

Next step is creating a dashboard,

And an empty Panel,

Change our Data source into our newly created Data source, and run below query in Metric Browser field

sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{namespace='mw'}) by (pod)

The result should be seen on below image,

Fyi, on this tutorial Im using Openshift 4.8.

No Comments

Leave a Comment

Please be polite. We appreciate that.
Your email address will not be published and required fields are marked