Changing the Logging Configuration

For DAGAppMasters and ContainerWorkers of MR3, Hive on MR3 uses by default a logging configuration file included in the MR3 distribution:

status = INFO
name = MR3ContainerLog4j2

property.mr3.root.logger = INFO
property.mr3.log.dir = ${}
property.mr3.log.file = run.log

appenders = console, DRFA

appender.console.type = Console = consoleLogger = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d [%t] %-5p %c %x - %m%n

appender.DRFA.type = RollingRandomAccessFile = DRFALogger
appender.DRFA.immediateFlush = true
appender.DRFA.fileName = ${sys:mr3.log.dir}/${sys:mr3.log.file}
appender.DRFA.filePattern = ${sys:mr3.log.dir}/${sys:mr3.log.file}.%d{yyyy-MM-dd}.%i
appender.DRFA.layout.type = PatternLayout
appender.DRFA.layout.pattern = %d [%t] %-5p %c %x - %m%n
appender.DRFA.policies.type = Policies
appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
appender.DRFA.policies.time.interval = 1
appender.DRFA.policies.time.modulate = true
appender.DRFA.policies.size.type = SizeBasedTriggeringPolicy
appender.DRFA.policies.size.size = 250MB
appender.DRFA.strategy.type = DefaultRolloverStrategy
appender.DRFA.strategy.max = 30

rootLogger.level = ${sys:mr3.root.logger}
rootLogger.appenderRef.file.ref = DRFALogger
rootLogger.appenderRef.console.ref = consoleLogger
rootLogger.appenderRef.console.level = FATAL
rootLogger.includeLocation = false

By default, ${sys:mr3.root.logger} expands to INFO, and ${} expands to a logging directory determined by Yarn at runtime.

The user can easily change the logging level by resetting the environment variable LOG_LEVEL in


The user can also use a new logging configuration file in two steps. First the user should update command-line options for DAGAppMasters and ContainerWorkers. In mr3-site.xml, is the key for specifying command-line options for DAGAppMasters, and mr3.container.launch.cmd-opts is the key for specifying command-line options for ContainerWorkers. Thus the user should append a new command-line option -Dlog4j.configurationFile=<name of the new logging configuration file> to the values for these keys, as shown in the following example:

  <value>-server -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB</value>
    Command-line options for launching DAGAppMaster
  <value>-server -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB</value>
    Command-line options for launching ContainerWorkers

Second the user should register the new logging configuration file as a local resource so that it appears in the classpath. Here is an example of registering a logging configuration file as a local resource:

  • copy the file to /tmp/ on HDFS.
  • append the path /tmp/ to the value of mr3.aux.uris in mr3-site.xml:

        URIs for the MR3 auxiliary jar files

Similarly to the one included in the MR3 distribution, the new logging configuration file may use ${sys:mr3.root.logger} and ${} which automatically expand at runtime.