Logging and Circuit Breaker Mechanism with Hystrix on Red Hat Fuse
Red Hat have a middleware product called Red Hat Fuse, both Red hat Fuse it and Apache Camel as its OpenSource project are well known products for a message-oriented middleware and integration platform. Functionality-wise, there are no huge difference between those two (Fuse and Camel) besides of support, so sometimes it is a good approach to test the opensource version first before going with the supported one.
Right now, we are trying to create a simple Camel script which is doing a proxy request to a third party api. But we are doing some enhancement on the script, so it will not be just a reverse proxy but added with more features such as logging, statistics and a circuit breaker mechanism. We are adding both logging and statistics for seeing the healthiness of our api and third party api endpoints, while circuit braker is being use for handling scenarios where there a latency or slowness happen on our third party api.
The underlying concept is drawn on below diagram,
Okay, so lets start by creating a Fuse application with maven,
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.edw</groupId>
<artifactId>HttpLoggingAndCircuitBreaker</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<fuse.version>7.2.0.fuse-720020-redhat-00001</fuse.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.jboss.redhat-fuse</groupId>
<artifactId>fuse-springboot-bom</artifactId>
<version>${fuse.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-http-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-servlet-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-hystrix-starter</artifactId>
</dependency>
<dependency>
<groupId>com.netflix.archaius</groupId>
<artifactId>archaius-core</artifactId>
<version>0.7.1</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-hystrix-dashboard</artifactId>
<version>1.4.7.RELEASE</version>
<exclusions>
<exclusion>
<groupId>com.netflix.archaius</groupId>
<artifactId>archaius-core</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-hystrix</artifactId>
<version>1.4.7.RELEASE</version>
<exclusions>
<exclusion>
<groupId>com.netflix.archaius</groupId>
<artifactId>archaius-core</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.jboss.redhat-fuse</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${fuse.version}</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
And two simple rest api, written in Camel XML format, first is a simple hello world rest api, while the second one is a much more complex app which involved circuit breaking and logging,
<?xml version="1.0" encoding="UTF-8"?>
<rest path="/hello-world" xmlns="http://camel.apache.org/schema/spring">
<get>
<route>
<setHeader headerName="Content-Type">
<constant>application/json</constant>
</setHeader>
<setBody>
<simple>{ "hello": "world" }</simple>
</setBody>
</route>
</get>
</rest>
<?xml version="1.0" encoding="UTF-8"?>
<rest path="/circuit-breaker" xmlns="http://camel.apache.org/schema/spring">
<post>
<route>
<setHeader headerName="timestamp">
<simple>${bean:java.lang.System?method=currentTimeMillis}</simple>
</setHeader>
<hystrix>
<!--
a message which exceed 1000ms will considered as timeout, and trigger fallback method.
It need a 5 timeouts in last ten seconds to trigger a 5000ms circuit break, before trying to re-connect again.
-->
<hystrixConfiguration executionTimeoutInMilliseconds="1000"
circuitBreakerSleepWindowInMilliseconds="5000"
metricsRollingStatisticalWindowInMilliseconds="10000"
circuitBreakerRequestVolumeThreshold="5" />
<to uri="https://run.mocky.io/v3/bc6c24ff-d124-401f-bf84-31e0c6366af6?httpClient.soTimeout=1000&bridgeEndpoint=true" />
<onFallback>
<!-- reversal example when a message is considered timeout by hystrix -->
<doTry>
<to uri="https://run.mocky.io/v3/4eb219a6-a7b1-40b8-90e1-230052fc5823?httpClient.soTimeout=1000&bridgeEndpoint=true" />
<doCatch>
<exception>java.lang.Exception</exception>
<bean ref="logHandler" method="logError('on catch')"/>
</doCatch>
<doFinally>
<setBody>
<simple>{ "hello": "fallback" }</simple>
</setBody>
</doFinally>
</doTry>
</onFallback>
</hystrix>
<bean ref="logHandler" method="logTimestamp(${header.timestamp})"/>
</route>
</post>
</rest>
The first api is quite self-explanatory, use below curl to do testing
curl -kv http://localhost:8080/api/hello-world
But we are focusing to the second API, which is much more complicated because it consist of a circuit breaker functionality which will do a 5 seconds break if there are more than 5 errors on the last 10 seconds. Perhaps diagram below can help simplifying the basic concept,
And can test that url with this code,
curl -kv http://localhost:8080/api/circuit-breaker -X POST --header "Content-Type: application/json" --data '{"username":"xyz","password":"xyz"}'
Next is creating a surrounding support system for this app, such as a hystrix dashboard for displaying how much transaction is failing, response time average, circuit status and error percentage, and a hystrix stream for providing those data. In order to activate hystrix dashboard and stream, we need to register it on our project.
package com.edw;
import org.apache.camel.component.hystrix.metrics.servlet.HystrixEventStreamServlet;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.ServletRegistrationBean;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.netflix.hystrix.dashboard.EnableHystrixDashboard;
import org.springframework.context.annotation.Bean;
@EnableCircuitBreaker
@EnableHystrixDashboard
@SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
@Bean
public HystrixEventStreamServlet hystrixServlet() {
return new HystrixEventStreamServlet();
}
@Bean
public ServletRegistrationBean servletRegistrationBean() {
return new ServletRegistrationBean(new HystrixEventStreamServlet(), "/hystrix.stream");
}
}
By default, we can see our hystrix dashboard on this url
http://localhost:8080/hystrix
Fill the data,
And see the result,
One other thing is that we can monitor the statistic and trace http history for our Camel by using our Actuator api,
curl -kv http://localhost:8080/metrics curl -kv http://localhost:8080/trace
One thing that is worth mentioning is that we can see how much response time for each API from our Actuator trace API, which by default gives last 100 http request. For example below, it took 297ms for our API to give a response.
Lastly, we can see my code on below github url.
https://github.com/edwin/fuse-logging-circuit-breaker
Good luck, and have fun with Camel and Circuit Breaker.
