jaeger Posts

Tracing Red Hat Fuse Transaction with Jaeger

Jaeger is open source software for tracing transactions between distributed services. It’s used for monitoring and troubleshooting complex microservices environments. And in this example, im trying trying to use it for tracing http request on Red Hat Fuse, or its opensource version which is Apache Camel. The purpose is to see how much time is needed for each api to take, and what kind of information comes with it.

First lets start by installing Jaeger on your local,

docker pull jaegertracing/all-in-one:1.19

docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
		-p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 \
		-p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 \
		jaegertracing/all-in-one:1.19

And create a java project with this pom file,

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.edw</groupId>
    <artifactId>camel-with-jaeger</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <fuse.version>7.2.0.fuse-720020-redhat-00001</fuse.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.jboss.redhat-fuse</groupId>
                <artifactId>fuse-springboot-bom</artifactId>
                <version>${fuse.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-http-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-servlet-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-sleuth</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zipkin</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>

    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.jboss.redhat-fuse</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                    <version>${fuse.version}</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Set connection to Jaeger tracing

# The Camel context name
camel.springboot.name=camel-with-jaeger

# enable all management endpoints
endpoints.enabled=true
management.security.enabled=false
camel.component.servlet.mapping.contextPath=/api/*
logging.level.root=info

# name for tracing
spring.application.name=Camel Hello World
spring.zipkin.base-url=http://localhost:9411/
spring.zipkin.enabled=true
spring.zipkin.sender.type=web
spring.sleuth.enabled=true
spring.sleuth.sampler.probability=1.0

Create a simple helo-world api,

<?xml version="1.0" encoding="UTF-8"?>
<rest path="/hello-world" xmlns="http://camel.apache.org/schema/spring">
    <get>
        <route>
            <setHeader headerName="Content-Type">
                <constant>application/json</constant>
            </setHeader>
            <setBody>
                <simple>{ "hello": "world" }</simple>
            </setBody>
        </route>
    </get>
</rest>

We test our newly created api by using curl command,

curl -kv http://localhost:8080/api/hello-world

And the result can be viewed directly on Jaeger dashboard


Now lets say we have other requirement such as adding a specific information on every trace result on Jaeger. We can achieve that by using Tag inside current Trace Span.

package com.edw.routes;

import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.sleuth.Span;
import org.springframework.cloud.sleuth.Tracer;
import org.springframework.stereotype.Component;

@Component
public class HelloWorldRoute extends RouteBuilder {

    @Autowired
    private Tracer tracer;

    @Override
    public void configure() throws Exception {
        rest()
                .get("hello")
                .route()
                .setHeader(Exchange.HTTP_RESPONSE_CODE, simple("200"))
                .setHeader(Exchange.CONTENT_TYPE, simple("application/json"))
                .process(exchange -> {
                    Span span = tracer.getCurrentSpan();
                    span.tag("something", "whatever");
                    span.tag("pretending to do some queries", "select 1 from dual");
                })
                .setBody(constant("{\"world\":\"world\"}"))
                .endRest()
        ;
    }
}

The result would be displayed on every trace like this,

Full code for this tutorial can be accessed in below url,

https://github.com/edwin/fuse-with-jaeger

Have fun with Jaeger and Camel :)

Google+

Distributed Tracing on Openshift using Jaeger and Spring Sleuth

There is one big issue when we are using microservices environment, that is sometimes we are unable to see messages goes from each microservice goes to which microservice and also unable to see latency for each microservices.

Luckily we have Jaeger to do that. According to its website, Jaeger is an open source, end-to-end distributed tracing for monitor and troubleshoot transactions in complex distributed systems. And it can also be installed easily on Openshift with a very simple oc command,

oc process -f https://raw.githubusercontent.com/jaegertracing/jaeger-openshift/master/all-in-one/jaeger-all-in-one-template.yml | oc create -f -

After installed, you will see Jaeger pod on Openshift project dashboard with several opened ports and a url for accessing query dashboard. See the red box on the image, it is the url for accessing zipkin api from other pods internally.

Next is creating two simple java app, one as backend, and another one as api gateway.
As usual, we’ll start with a simple maven file for our backend service,

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.6.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.edw.test</groupId>
    <artifactId>HelloWorld</artifactId>
    <version>1.0.1</version>
    <name>Hello World</name>
    <description>A Simple Hello World</description>

    <properties>
        <java.version>1.8</java.version>

        <version.fabric8.plugin>3.5.38</version.fabric8.plugin>
        <fabric8.generator.fromMode>istag</fabric8.generator.fromMode>
        <fabric8.generator.from>redhat-openjdk18-openshift:1.0</fabric8.generator.from>

    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-sleuth</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zipkin</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>

            <plugin>
                <groupId>io.fabric8</groupId>
                <artifactId>fabric8-maven-plugin</artifactId>
                <version>${version.fabric8.plugin}</version>

                <executions>
                    <execution>
                        <id>fmp</id>
                        <goals>
                            <goal>resource</goal>
                            <goal>build</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>
</project>

Create a simple java app,

package com.edw.test.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class DemoApplication {
    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }
}
package com.edw.test.demo;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;

@RestController
public class IndexController {

    private Logger logger = LoggerFactory.getLogger(this.getClass());

    @GetMapping("/")
    public HashMap sayHelloApi(@RequestParam String id) {
        logger.debug("say something, anything - {}", id);
        return new HashMap(){{
            put("Message", "Hello My World "+id);
        }};
    }
}

A logback.xml file for logging format,

<configuration>
    <statusListener class="ch.qos.logback.core.status.NopStatusListener"/>
    <springProperty scope="context" name="springAppName" source="spring.application.name"/>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} %-5level [${springAppName},%X{X-B3-SpanId:-}] %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    <logger name="com.edw" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>
    <root level="ERROR" additivity="false">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

And finally, a properties file for storing our configuration.

spring.application.name=Hello World
spring.zipkin.baseUrl: http://zipkin:9411/
spring.sleuth.sampler.probability=1.0

Next is creating our Api Gateway class, we’ll start with a simple POM file.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.6.RELEASE</version>
        <relativePath/>
    </parent>
    <groupId>com.edw</groupId>
    <artifactId>ApiGateway</artifactId>
    <version>1.0</version>

    <name>ApiGateway</name>
    <description>Demo project for Api Gateway</description>

    <properties>
        <java.version>1.8</java.version>

        <version.fabric8.plugin>3.5.38</version.fabric8.plugin>
        <fabric8.generator.fromMode>istag</fabric8.generator.fromMode>
        <fabric8.generator.from>redhat-openjdk18-openshift:1.0</fabric8.generator.from>

    </properties>


    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-sleuth</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zipkin</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>


    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>

            <plugin>
                <groupId>io.fabric8</groupId>
                <artifactId>fabric8-maven-plugin</artifactId>
                <version>${version.fabric8.plugin}</version>

                <executions>
                    <execution>
                        <id>fmp</id>
                        <goals>
                            <goal>resource</goal>
                            <goal>build</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>
</project>

And several java classes,

package com.edw;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}
package com.edw.controller;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

import java.util.Date;
import java.util.UUID;

@RestController
public class IndexController {

    @Autowired
    private RestTemplate restTemplate;

    private Logger logger = LoggerFactory.getLogger(this.getClass());

    @GetMapping(value="/", produces = {MediaType.APPLICATION_JSON_UTF8_VALUE})
    public String indexApi() throws Exception {
        String result = "";
        for (int i = 0; i< 3; i++) {
            logger.debug("firing");
            result = restTemplate.getForObject("http://helloworld:8080/?id="+ UUID.randomUUID().toString()+"&timestamp="+new Date().getTime(), String.class);
            logger.debug("response is {}, MDC is {}", result, MDC.get("X-B3-SpanId"));
        }
        return result;
    }
}
package com.edw.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

/**
 * <pre>
 *     com.edw.config.RestTemplateConfig
 * </pre>
 *
 * @author Muhammad Edwin < emuhamma at redhat dot com >
 * 23 Sep 2019 10:47
 */
@Configuration
public class RestTemplateConfig {
    @Bean
    public RestTemplate getRestTemplate() {
        RestTemplate restTemplate = new RestTemplate();
        return restTemplate;
    }
}

And a logback.xml, and application.properties.

<configuration>
    <statusListener class="ch.qos.logback.core.status.NopStatusListener"/>
    <springProperty scope="context" name="springAppName" source="spring.application.name"/>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} %-5level [${springAppName},%X{X-B3-SpanId:-}] %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    <logger name="com.edw" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>
    <root level="ERROR" additivity="false">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>
spring.application.name=API Gateway
spring.zipkin.baseUrl: http://zipkin:9411/
spring.sleuth.sampler.probability=1.0

We can deploy both project to Openshift using fabric8 command. And this is the result after deployed successfully and hitting Api Gateway url from browser,

And we can see the detail for each request by click on it,

To see a more detailed information, we can click more and see class and method name, and also information span.

And there is one good feature when using Jaeger, is that we can visualize how a message is delivered among different microservices,

And we can search the request SpanId on Kibana,

Well, hopefully it helps. (^)

Google+