Changing the Logging Configuration

For DAGAppMasters and ContainerWorkers of MR3, Hive-MR3 uses by default a logging configuration file mr3-container-log4j.properties included in the Hive-MR3 distribution:

log4j.rootLogger=${mr3.root.logger}

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n
log4j.appender.console.Threshold=ERROR

log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=100MB
log4j.appender.file.MaxBackupIndex=100
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n
log4j.appender.file.File=${yarn.app.container.log.dir}/run.log
log4j.appender.file.Append=false
log4j.appender.file.Threshold=ALL

By default, ${mr3.root.logger} expands to INFO,file, and ${yarn.app.container.log.dir} expands to a logging directory determined by Yarn at runtime.

The user can easily change the logging level by resetting the environment variable LOG_LEVEL in env.sh:

LOG_LEVEL=INFO

The user can also use a new logging configuration file in two steps. First the user should update command-line options for DAGAppMasters and ContainerWorkers. In mr3-site.xml, mr3.am.launch.cmd-opts is the key for specifying command-line options for DAGAppMasters, and mr3.container.launch.cmd-opts is the key for specifying command-line options for ContainerWorkers. Thus the user should append a new command-line option -Dlog4j.configuration=<name of the new logging configuration file> to the values for these keys, as shown in the following example:

<property>
  <name>mr3.am.launch.cmd-opts</name>
  <value>-server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -Dlog4j.configuration=custom-log4j.properties</value>
  <description>
    Command-line options for launching DAGAppMaster
  </description>
</property>
<property>
  <name>mr3.container.launch.cmd-opts</name>
  <value>-server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -Dlog4j.configuration=custom-log4j.properties</value>
  <description>
    Command-line options for launching ContainerWorkers
  </description>
</property>

Second the user should register the new logging configuration file as a local resource so that it appears in the classpath. Here is an example of registering a logging configuration file custom-log4j.properties as a local resource:

  • copy the file to /tmp/custom-log4j.properties on HDFS.
  • append the path /tmp/custom-log4j.properties to the value of mr3.aux.uris in mr3-site.xml:

    <property>
      <name>mr3.aux.uris</name>
      <value>${auxuris},/tmp/custom-log4j.properties</value>
      <description>
        URIs for the MR3 auxiliary jar files
      </description>
    </property>
    

Similarly to the one included in the Hive-MR3 distribution, the new logging configuration file may use ${mr3.root.logger} and ${yarn.app.container.log.dir} which automatically expand at runtime.