How to Compress Log4J’s Log File

Today i found a very weird condition on one of my end user. Their application’s log file is so big, despite the log4j’s log level is set to INFO. Im using log4j’s DailyRollingFileAppender which would create a new log file everyday. Each file’s size is around 10 to 20MB, and i see the log files is like 3 months old. Which makes total of almost 2GBs of log files.

So my solution to them is to compress everyday’s log into gzip, gz or perhaps into zip file. Basically, im using one of log4j’s companion..

Okay, so here is my solution, first a simple pom.xml to download the libraries needed,

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.edw.compresslogfile</groupId>
    <artifactId>CompressLogFile</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>CompressLogFile</name>
    <url>http://maven.apache.org</url>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>apache-log4j-extras</artifactId>
            <version>1.0</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>                        
    </dependencies>
</project>

And then my log4j.properties file, please take a look at line 7. It will create a new log file for every second, change it into “yyyyMMdd” if you want to create a daily appender.

log4j.rootLogger=DEBUG, request

log4j.appender.request=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.request.File=msg.log
log4j.appender.request.RollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.request.RollingPolicy.ActiveFileName =msg.log
log4j.appender.request.RollingPolicy.FileNamePattern=msg.%d{yyyyMMdd.HHmmss}.gz
log4j.appender.request.layout = org.apache.log4j.PatternLayout
log4j.appender.request.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n

And my simple java test case

package com.edw.compresslogfile;

import org.apache.log4j.Logger;

/**
 * Hello world!
 *
 */
public class App {

    private static final Logger LOGGER = Logger.getLogger(App.class);

    public static void main(String[] args) {
        for (int i = 0; i < 10; i++) {
            try {
                LOGGER.debug("Hello World " + i);
                Thread.sleep(5000);
            } catch (Exception e) {
            }
        }
    }
}

It will create a compressed log file for every second.

This is the project structure on my Netbeans,

Hope it helped others, cheers (B)

Google+

8 Comments

Suhas

about 3 years ago

This example does create the.gz files, but they are not actually .gz, just plain log files renames to .gz.

Reply

edwin

about 3 years ago

hi Suhas, ive tried opened my .gz log file directly using notepad, and it seems like its indeed is a .gz file :)

Sushil

about 2 years ago

Hi Edwin, I also tried the same thing. Logger creates the .gz file but if I try to open it with winrar then it gives an error. I can directly open the .gz file in notepad. Also the size of actual log file and gz file are same. It seems that, logger is simply renaming the file name by .gz.

Reply

edwin

about 2 years ago

Hi Sushil, thats weird, could you share your project on github so i could test it from here? thanks :)

Sushil

about 2 years ago

Hi Edwin, here is the sample code. log4j.properties file log4j.rootLogger=INFO, loggerId log4j.appender.loggerId=org.apache.log4j.rolling.RollingFileAppender log4j.appender.loggerId.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.loggerId.rollingPolicy.maxIndex=5 log4j.appender.loggerId.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.loggerId.triggeringPolicy.MaxFileSize=1000000 log4j.appender.loggerId.rollingPolicy.FileNamePattern=worker-%i.log.tar.gz log4j.appender.loggerId.rollingPolicy.ActiveFileName=worker.log log4j.appender.loggerId.layout=org.apache.log4j.PatternLayout log4j.appender.loggerId.layout.ConversionPattern=%d [%t] %-5p (%F:%L) - %m%n Sample Java Class package com.interactcrm.gz.test; import java.io.IOException; import java.sql.SQLException; import org.apache.log4j.Logger; public class Log4jExample { static Logger log = Logger.getLogger(Log4jExample.class.getName()); public static void main(String[] args) throws IOException, SQLException { System.out.println(log.getRootLogger().getAppender("loggerId")); for (int i = 0; i < 900000; i++) { log.debug("Hello this is an debug message"); log.info("Hello this is an info message"); } } }

Reply

Sushil

about 2 years ago

Hi Edwin, Waiting for your reply.

oczek

about 2 years ago

FixedWindowRollingPolicy != TimeBasedRollingPolicy FixedWindowRollingPolicy probably can not compress log files

Anubhav Singhal

about 4 months ago

It worked perfectly for me. Thank you very much.

Reply

Leave a Comment

Please be polite. We appreciate that.
Your email address will not be published and required fields are marked


:-[ (B) (^) (P) (@) (O) (D) :-S ;-( (C) (&) :-$ (E) (~) (K) (I) (L) (8) :-O (T) (G) (F) :-( (H) :-) (*) :-D (N) (Y) :-P (U) (W) ;-)