How to Compress Log4J’s Log File

Today i found a very weird condition on one of my end user. Their application’s log file is so big, despite the log4j’s log level is set to INFO. Im using log4j’s DailyRollingFileAppender which would create a new log file everyday. Each file’s size is around 10 to 20MB, and i see the log files is like 3 months old. Which makes total of almost 2GBs of log files.

So my solution to them is to compress everyday’s log into gzip, gz or perhaps into zip file. Basically, im using one of log4j’s companion..

Okay, so here is my solution, first a simple pom.xml to download the libraries needed,

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.edw.compresslogfile</groupId>
    <artifactId>CompressLogFile</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>CompressLogFile</name>
    <url>http://maven.apache.org</url>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>apache-log4j-extras</artifactId>
            <version>1.0</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>                        
    </dependencies>
</project>

And then my log4j.properties file, please take a look at line 7. It will create a new log file for every second, change it into “yyyyMMdd” if you want to create a daily appender.

log4j.rootLogger=DEBUG, request

log4j.appender.request=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.request.File=msg.log
log4j.appender.request.RollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.request.RollingPolicy.ActiveFileName =msg.log
log4j.appender.request.RollingPolicy.FileNamePattern=msg.%d{yyyyMMdd.HHmmss}.gz
log4j.appender.request.layout = org.apache.log4j.PatternLayout
log4j.appender.request.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n

And my simple java test case

package com.edw.compresslogfile;

import org.apache.log4j.Logger;

/**
 * Hello world!
 *
 */
public class App {

    private static final Logger LOGGER = Logger.getLogger(App.class);

    public static void main(String[] args) {
        for (int i = 0; i < 10; i++) {
            try {
                LOGGER.debug("Hello World " + i);
                Thread.sleep(5000);
            } catch (Exception e) {
            }
        }
    }
}

It will create a compressed log file for every second.

This is the project structure on my Netbeans,

Hope it helped others, cheers (B)

Google+

9 Comments

Suhas

about 3 years ago

This example does create the.gz files, but they are not actually .gz, just plain log files renames to .gz.

Reply

edwin

about 3 years ago

hi Suhas, ive tried opened my .gz log file directly using notepad, and it seems like its indeed is a .gz file :)

Sushil

about 3 years ago

Hi Edwin, I also tried the same thing. Logger creates the .gz file but if I try to open it with winrar then it gives an error. I can directly open the .gz file in notepad. Also the size of actual log file and gz file are same. It seems that, logger is simply renaming the file name by .gz.

Reply

edwin

about 3 years ago

Hi Sushil, thats weird, could you share your project on github so i could test it from here? thanks :)

Sushil

about 3 years ago

Hi Edwin, here is the sample code. log4j.properties file log4j.rootLogger=INFO, loggerId log4j.appender.loggerId=org.apache.log4j.rolling.RollingFileAppender log4j.appender.loggerId.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.loggerId.rollingPolicy.maxIndex=5 log4j.appender.loggerId.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.loggerId.triggeringPolicy.MaxFileSize=1000000 log4j.appender.loggerId.rollingPolicy.FileNamePattern=worker-%i.log.tar.gz log4j.appender.loggerId.rollingPolicy.ActiveFileName=worker.log log4j.appender.loggerId.layout=org.apache.log4j.PatternLayout log4j.appender.loggerId.layout.ConversionPattern=%d [%t] %-5p (%F:%L) - %m%n Sample Java Class package com.interactcrm.gz.test; import java.io.IOException; import java.sql.SQLException; import org.apache.log4j.Logger; public class Log4jExample { static Logger log = Logger.getLogger(Log4jExample.class.getName()); public static void main(String[] args) throws IOException, SQLException { System.out.println(log.getRootLogger().getAppender("loggerId")); for (int i = 0; i < 900000; i++) { log.debug("Hello this is an debug message"); log.info("Hello this is an info message"); } } }

Reply

Sushil

about 3 years ago

Hi Edwin, Waiting for your reply.

oczek

about 2 years ago

FixedWindowRollingPolicy != TimeBasedRollingPolicy FixedWindowRollingPolicy probably can not compress log files

Anubhav Singhal

about 6 months ago

It worked perfectly for me. Thank you very much.

Reply

Tamanna Lekhwani

about 1 month ago

Hi Edwin, I have to implement archiving of logs in the below manner but I am not able to achieve it despite trying various combinations of TimeBasedRollingPolicy and SizeBasedRollingPolicy. Can you please help me out here? My requirement(values customized for test purpose): Log files should be of max size 1KB, beyond which new log file should be generated keeping the previous file untouched. I am not concerned about the max number of files generated by this. It shall have no upper limit. Every minute, the previous minute's log files should be converted to .zip. So, if time is 14:00:01, suppose 100 log files got generated for this due to 1 KB restriction. Now at 14:00:02, previous second's 100 log files should get converted to 100 .zip files, and by end of 14:00:02, I should have another lot of 100 log files. So, at the start of 14:00:03, i should have 100 .log files for 14:00:02 and 100 .zip files for 14:00:01. I have tried various combinations of param values and policies but have not been able to achieve this. When I am using both policies, I am getting .zip files as soon as 1KB limit is exceeded. Below is my config. Please let me know where I am going wrong. %d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n If am modifying it to below, I am not getting any zip files: %d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n

Reply

Leave a Comment

Please be polite. We appreciate that.
Your email address will not be published and required fields are marked


:-[ (B) (^) (P) (@) (O) (D) :-S ;-( (C) (&) :-$ (E) (~) (K) (I) (L) (8) :-O (T) (G) (F) :-( (H) :-) (*) :-D (N) (Y) :-P (U) (W) ;-)