fuse Posts

Tracing Red Hat Fuse Transaction with Jaeger

Jaeger is open source software for tracing transactions between distributed services. It’s used for monitoring and troubleshooting complex microservices environments. And in this example, im trying trying to use it for tracing http request on Red Hat Fuse, or its opensource version which is Apache Camel. The purpose is to see how much time is needed for each api to take, and what kind of information comes with it.

First lets start by installing Jaeger on your local,

docker pull jaegertracing/all-in-one:1.19

docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
		-p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 \
		-p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 \
		jaegertracing/all-in-one:1.19

And create a java project with this pom file,

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.edw</groupId>
    <artifactId>camel-with-jaeger</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <fuse.version>7.2.0.fuse-720020-redhat-00001</fuse.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.jboss.redhat-fuse</groupId>
                <artifactId>fuse-springboot-bom</artifactId>
                <version>${fuse.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-http-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-servlet-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-sleuth</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zipkin</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>

    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.jboss.redhat-fuse</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                    <version>${fuse.version}</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Set connection to Jaeger tracing

# The Camel context name
camel.springboot.name=camel-with-jaeger

# enable all management endpoints
endpoints.enabled=true
management.security.enabled=false
camel.component.servlet.mapping.contextPath=/api/*
logging.level.root=info

# name for tracing
spring.application.name=Camel Hello World
spring.zipkin.base-url=http://localhost:9411/
spring.zipkin.enabled=true
spring.zipkin.sender.type=web
spring.sleuth.enabled=true
spring.sleuth.sampler.probability=1.0

Create a simple helo-world api,

<?xml version="1.0" encoding="UTF-8"?>
<rest path="/hello-world" xmlns="http://camel.apache.org/schema/spring">
    <get>
        <route>
            <setHeader headerName="Content-Type">
                <constant>application/json</constant>
            </setHeader>
            <setBody>
                <simple>{ "hello": "world" }</simple>
            </setBody>
        </route>
    </get>
</rest>

We test our newly created api by using curl command,

curl -kv http://localhost:8080/api/hello-world

And the result can be viewed directly on Jaeger dashboard


Now lets say we have other requirement such as adding a specific information on every trace result on Jaeger. We can achieve that by using Tag inside current Trace Span.

package com.edw.routes;

import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.sleuth.Span;
import org.springframework.cloud.sleuth.Tracer;
import org.springframework.stereotype.Component;

@Component
public class HelloWorldRoute extends RouteBuilder {

    @Autowired
    private Tracer tracer;

    @Override
    public void configure() throws Exception {
        rest()
                .get("hello")
                .route()
                .setHeader(Exchange.HTTP_RESPONSE_CODE, simple("200"))
                .setHeader(Exchange.CONTENT_TYPE, simple("application/json"))
                .process(exchange -> {
                    Span span = tracer.getCurrentSpan();
                    span.tag("something", "whatever");
                    span.tag("pretending to do some queries", "select 1 from dual");
                })
                .setBody(constant("{\"world\":\"world\"}"))
                .endRest()
        ;
    }
}

The result would be displayed on every trace like this,

Full code for this tutorial can be accessed in below url,

https://github.com/edwin/fuse-with-jaeger

Have fun with Jaeger and Camel :)

Google+

Logging and Circuit Breaker Mechanism with Hystrix on Red Hat Fuse

Red Hat have a middleware product called Red Hat Fuse, both Red hat Fuse it and Apache Camel as its OpenSource project are well known products for a message-oriented middleware and integration platform. Functionality-wise, there are no huge difference between those two (Fuse and Camel) besides of support, so sometimes it is a good approach to test the opensource version first before going with the supported one.

Right now, we are trying to create a simple Camel script which is doing a proxy request to a third party api. But we are doing some enhancement on the script, so it will not be just a reverse proxy but added with more features such as logging, statistics and a circuit breaker mechanism. We are adding both logging and statistics for seeing the healthiness of our api and third party api endpoints, while circuit braker is being use for handling scenarios where there a latency or slowness happen on our third party api.

The underlying concept is drawn on below diagram,

Okay, so lets start by creating a Fuse application with maven,

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.edw</groupId>
    <artifactId>HttpLoggingAndCircuitBreaker</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <fuse.version>7.2.0.fuse-720020-redhat-00001</fuse.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.jboss.redhat-fuse</groupId>
                <artifactId>fuse-springboot-bom</artifactId>
                <version>${fuse.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-http-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-servlet-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-hystrix-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>com.netflix.archaius</groupId>
            <artifactId>archaius-core</artifactId>
            <version>0.7.1</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-hystrix-dashboard</artifactId>
            <version>1.4.7.RELEASE</version>
            <exclusions>
                <exclusion>
                    <groupId>com.netflix.archaius</groupId>
                    <artifactId>archaius-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-hystrix</artifactId>
            <version>1.4.7.RELEASE</version>
            <exclusions>
                <exclusion>
                    <groupId>com.netflix.archaius</groupId>
                    <artifactId>archaius-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.jboss.redhat-fuse</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                    <version>${fuse.version}</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

And two simple rest api, written in Camel XML format, first is a simple hello world rest api, while the second one is a much more complex app which involved circuit breaking and logging,

<?xml version="1.0" encoding="UTF-8"?>
<rest path="/hello-world" xmlns="http://camel.apache.org/schema/spring">
    <get>
        <route>
            <setHeader headerName="Content-Type">
                <constant>application/json</constant>
            </setHeader>
            <setBody>
                <simple>{ "hello": "world" }</simple>
            </setBody>
        </route>
    </get>
</rest>
<?xml version="1.0" encoding="UTF-8"?>
<rest path="/circuit-breaker" xmlns="http://camel.apache.org/schema/spring">
    <post>
        <route>
            <setHeader headerName="timestamp">
                <simple>${bean:java.lang.System?method=currentTimeMillis}</simple>
            </setHeader>

            <hystrix>
                <!--
                    a message which exceed 1000ms will considered as timeout, and trigger fallback method.
                    It need a 5 timeouts in last ten seconds to trigger a 5000ms circuit break, before trying to re-connect again.
                -->
                <hystrixConfiguration executionTimeoutInMilliseconds="1000"
                                      circuitBreakerSleepWindowInMilliseconds="5000"
									  metricsRollingStatisticalWindowInMilliseconds="10000"
                                      circuitBreakerRequestVolumeThreshold="5" />
                <to uri="https://run.mocky.io/v3/bc6c24ff-d124-401f-bf84-31e0c6366af6?httpClient.soTimeout=1000&amp;bridgeEndpoint=true" />
                <onFallback>
                    <!--  reversal example when a message is considered timeout by hystrix -->
                    <doTry>
                        <to uri="https://run.mocky.io/v3/4eb219a6-a7b1-40b8-90e1-230052fc5823?httpClient.soTimeout=1000&amp;bridgeEndpoint=true" />

                        <doCatch>
                            <exception>java.lang.Exception</exception>
                            <bean ref="logHandler" method="logError('on catch')"/>
                        </doCatch>
                        <doFinally>
                            <setBody>
                                <simple>{ "hello": "fallback" }</simple>
                            </setBody>
                        </doFinally>
                    </doTry>
                </onFallback>
            </hystrix>

            <bean ref="logHandler" method="logTimestamp(${header.timestamp})"/>
        </route>
    </post>
</rest>

The first api is quite self-explanatory, use below curl to do testing

curl -kv http://localhost:8080/api/hello-world

But we are focusing to the second API, which is much more complicated because it consist of a circuit breaker functionality which will do a 5 seconds break if there are more than 5 errors on the last 10 seconds. Perhaps diagram below can help simplifying the basic concept,

And can test that url with this code,

curl -kv http://localhost:8080/api/circuit-breaker -X POST --header "Content-Type: application/json" --data '{"username":"xyz","password":"xyz"}'

Next is creating a surrounding support system for this app, such as a hystrix dashboard for displaying how much transaction is failing, response time average, circuit status and error percentage, and a hystrix stream for providing those data. In order to activate hystrix dashboard and stream, we need to register it on our project.

package com.edw;

import org.apache.camel.component.hystrix.metrics.servlet.HystrixEventStreamServlet;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.ServletRegistrationBean;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.netflix.hystrix.dashboard.EnableHystrixDashboard;
import org.springframework.context.annotation.Bean;

@EnableCircuitBreaker
@EnableHystrixDashboard
@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Bean
    public HystrixEventStreamServlet hystrixServlet() {
        return new HystrixEventStreamServlet();
    }

    @Bean
    public ServletRegistrationBean servletRegistrationBean() {
        return new ServletRegistrationBean(new HystrixEventStreamServlet(), "/hystrix.stream");
    }
}

By default, we can see our hystrix dashboard on this url

http://localhost:8080/hystrix

Fill the data,

And see the result,

One other thing is that we can monitor the statistic and trace http history for our Camel by using our Actuator api,

curl -kv http://localhost:8080/metrics

curl -kv http://localhost:8080/trace

One thing that is worth mentioning is that we can see how much response time for each API from our Actuator trace API, which by default gives last 100 http request. For example below, it took 297ms for our API to give a response.

Lastly, we can see my code on below github url.

https://github.com/edwin/fuse-logging-circuit-breaker

Good luck, and have fun with Camel and Circuit Breaker.

Google+

Create Async HTTP Process with Red Hat Fuse on Top of Openshift 3.11

Got a unique requirement yesterday where one process on Red Hat Fuse consuming a very long time, therefore creating lots of timeout error from frontend. It happen because some process on Fuse performing a very complicated task and we cannot modified existing api flow due to business requirements.

For this example, i want to simulate the same slowness on my Fuse + Spring Boot app,

package com.redhat.edw;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

A simple camel route,

package com.redhat.edw;

import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.rest.RestBindingMode;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

@Component
public class Routes extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        restConfiguration("servlet")
                .bindingMode(RestBindingMode.json)
        ;

        rest()
                .get("hello")
                .route()
                .setBody(method("helloRouteHandler", "setHelloWithName"))
                .endRest()
        ;
    }
}

A placeholder bean,

package com.redhat.edw;

import lombok.*;

@Data
@ToString
@AllArgsConstructor
@NoArgsConstructor
@Builder
public class HelloResponse {
    private String content;
}

And a Handler class,

package com.redhat.edw;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

@Slf4j
@Component("helloRouteHandler")
public class HelloRouteHandler {

    @Value("${name}")
    private String name;

    public HelloResponse setHelloWithName() throws Exception {

        /*
         * simulate a very long process (10second)
         */
        Thread.sleep(10000);

        /*
         * this process will still getting called after 10 second regardless of sync or async
         */
        log.info("calling hello for "+name);

        return HelloResponse.builder().content(name).build();
    }
}

And two simple properties file,

# The Camel context name
camel.springboot.name=FuseHelloWorld

# enable all management endpoints
endpoints.enabled=true
management.security.enabled=false

camel.component.servlet.mapping.contextPath=/api/*

name=Edwin
# OpenShift ConfigMap name
spring.application.name=fuse-hello-world

spring.cloud.kubernetes.reload.enabled=true
spring.cloud.kubernetes.reload.strategy=restart_context
spring.cloud.kubernetes.reload.monitoring-config-maps=true
spring.cloud.kubernetes.reload.monitoring-secrets=true
spring.cloud.kubernetes.reload.mode=polling

Run the project and try calling /api/hello api and see how much time is needed to give response time.

As you can see, 10seconds is not an acceptable response for a regular web user and wee need to improve it. The workaround is quite simple, Spring have an @Async method which we will utilize on Handler class,

package com.redhat.edw;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Component;

@Slf4j
@Component("helloRouteHandler")
public class HelloRouteHandler {

    @Value("${name}")
    private String name;

    @Async
    public HelloResponse setHelloWithName() throws Exception {

        /*
         * simulate a very long process (10second)
         */
        Thread.sleep(10000);

        /*
         * this process will still getting called after 10 second regardless of sync or async
         */
        log.info("calling hello for "+name);

        return HelloResponse.builder().content(name).build();
    }
}

Which will give a faster response,

And we can deploy our code to Openshift by using this command,

mvn fabric8:deploy -Pfabric8

A successful deployment will looks like this,

And finally, the complete code can be downloaded from this url,

https://github.com/edwin/fuse-with-async-http
Google+