Today i found a very weird condition on one of my end user. Their application’s log file is so big, despite the log4j’s log level is set to INFO. Im using log4j’s DailyRollingFileAppender which would create a new log file everyday. Each file’s size is around 10 to 20MB, and i see the log files is like 3 months old. Which makes total of almost 2GBs of log files.
So my solution to them is to compress everyday’s log into gzip, gz or perhaps into zip file. Basically, im using one of log4j’s companion..
Okay, so here is my solution, first a simple pom.xml to download the libraries needed,
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.edw.compresslogfile</groupId>
<artifactId>CompressLogFile</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>CompressLogFile</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>apache-log4j-extras</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
</dependencies>
</project>
And then my log4j.properties file, please take a look at line 7. It will create a new log file for every second, change it into “yyyyMMdd” if you want to create a daily appender.
log4j.rootLogger=DEBUG, request
log4j.appender.request=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.request.File=msg.log
log4j.appender.request.RollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.request.RollingPolicy.ActiveFileName =msg.log
log4j.appender.request.RollingPolicy.FileNamePattern=msg.%d{yyyyMMdd.HHmmss}.gz
log4j.appender.request.layout = org.apache.log4j.PatternLayout
log4j.appender.request.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n
And my simple java test case
package com.edw.compresslogfile;
import org.apache.log4j.Logger;
/**
* Hello world!
*
*/
public class App {
private static final Logger LOGGER = Logger.getLogger(App.class);
public static void main(String[] args) {
for (int i = 0; i < 10; i++) {
try {
LOGGER.debug("Hello World " + i);
Thread.sleep(5000);
} catch (Exception e) {
}
}
}
}
It will create a compressed log file for every second.
This is the project structure on my Netbeans,
Hope it helped others, cheers (B)
This example does create the.gz files, but they are not actually .gz, just plain log files renames to .gz.
hi Suhas,
ive tried opened my .gz log file directly using notepad, and it seems like its indeed is a .gz file 🙂
Hi Edwin,
I also tried the same thing. Logger creates the .gz file but if I try to open it with winrar then it gives an error. I can directly open the .gz file in notepad. Also the size of actual log file and gz file are same.
It seems that, logger is simply renaming the file name by .gz.
Hi Sushil,
thats weird, could you share your project on github so i could test it from here?
thanks 🙂
Hi Edwin,
here is the sample code.
log4j.properties file
log4j.rootLogger=INFO, loggerId
log4j.appender.loggerId=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.loggerId.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.loggerId.rollingPolicy.maxIndex=5
log4j.appender.loggerId.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.loggerId.triggeringPolicy.MaxFileSize=1000000
log4j.appender.loggerId.rollingPolicy.FileNamePattern=worker-%i.log.tar.gz
log4j.appender.loggerId.rollingPolicy.ActiveFileName=worker.log
log4j.appender.loggerId.layout=org.apache.log4j.PatternLayout
log4j.appender.loggerId.layout.ConversionPattern=%d [%t] %-5p (%F:%L) – %m%n
Sample Java Class
package com.interactcrm.gz.test;
import java.io.IOException;
import java.sql.SQLException;
import org.apache.log4j.Logger;
public class Log4jExample {
static Logger log = Logger.getLogger(Log4jExample.class.getName());
public static void main(String[] args) throws IOException, SQLException {
System.out.println(log.getRootLogger().getAppender(“loggerId”));
for (int i = 0; i < 900000; i++) {
log.debug("Hello this is an debug message");
log.info("Hello this is an info message");
}
}
}
Hi Edwin,
Waiting for your reply.
FixedWindowRollingPolicy != TimeBasedRollingPolicy
FixedWindowRollingPolicy probably can not compress log files
It worked perfectly for me.
Thank you very much.
Hi Edwin, I have to implement archiving of logs in the below manner but I am not able to achieve it despite trying various combinations of TimeBasedRollingPolicy and SizeBasedRollingPolicy. Can you please help me out here?
My requirement(values customized for test purpose):
Log files should be of max size 1KB, beyond which new log file should be generated keeping the previous file untouched. I am not concerned about the max number of files generated by this. It shall have no upper limit.
Every minute, the previous minute’s log files should be converted to .zip. So, if time is 14:00:01, suppose 100 log files got generated for this due to 1 KB restriction. Now at 14:00:02, previous second’s 100 log files should get converted to 100 .zip files, and by end of 14:00:02, I should have another lot of 100 log files.
So, at the start of 14:00:03, i should have 100 .log files for 14:00:02 and 100 .zip files for 14:00:01.
I have tried various combinations of param values and policies but have not been able to achieve this. When I am using both policies, I am getting .zip files as soon as 1KB limit is exceeded. Below is my config. Please let me know where I am going wrong.
%d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n
If am modifying it to below, I am not getting any zip files:
%d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n
The appender creates the file on-demand. The appender only creates the file when a log event passes all filters and is routed to this appender. Defaults to false. A Filter to determine if the event should be handled by this Appender. More than one Filter may be used by using a CompositeFilter.
Hi!
just got the same problem. I solved with the following one-liner, and put it to the crontab as a daily job.
find /opt/tomcat/logs/projectname.log.`date –date=”1 day ago” +%F` -exec gzip -9 {} \;
I have used SizeBasedTriggeringPolicy in my project. It is working fine in Local environment but, when I see the logs in the production it is behaving like filename.log.zip.zip.zip.zip.zip.zip.zip instead of single gz file. Can anyone help me out.
How can I limit the gz file of log?