MR3-UI

MR3-UI is a module for visualizing the progress of DAGs. It is a slight variant of TEZ-UI included in the Tez distribution, so the usage is similar.

Before running MR3-UI

MR3-UI requires a Timeline Server in order to retrieve data on DAGs (both currently active and previously finished). Hence the user should update yarn-site.xml properly before submitting DAGs. For example, if a Timeline Server is running on node0, the user can add the following properties to yarn-site.xml:

<configuration>
  <property>
    <name>yarn.timeline-service.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.timeline-service.hostname</name>
    <value>node0</value>
  </property>
  <property>
    <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.timeline-service.generic-application-history.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.timeline-service.http-cross-origin.enabled</name>
    <value>true</value>
  </property>
</configuration>

In a secure cluster with Kerberos, the user should specify the principal and service keytab file for the Timeline Server itself and also for Timeline HTTP authentication. The following example shows which configuration keys should be set in yarn-site.xml.

<configuration>
  <property>
    <name>yarn.timeline-service.principal</name>
    <value>yarn/_HOST@datamonad.com</value>
  </property>
  <property>
    <name>yarn.timeline-service.keytab</name>
    <value>/etc/security/keytabs/yarn.service.keytab</value>
  </property>
  <property>
    <name>yarn.timeline-service.http-authentication.type</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>yarn.timeline-service.http-authentication.kerberos.principal</name>
    <value>HTTP/_HOST@datamonad.com</value>
  </property>
  <property>
    <name>yarn.timeline-service.http-authentication.kerberos.keytab</name>
    <value>/etc/security/keytabs/spnego.service.keytab</value>
  </property>
</configuration>

If you update yarn-site.xml, restart both the Yarn Resource Manager and the Timeline Server.

Installing and configuring MR3-UI

Find mr3-ui/mr3-ui-0.8.5.tar.gz in the MR3 distribution and extract it to a MR3-UI directory that can be accessed by a web server. Then update config/configs.env in the MR3-UI directory to specify the addresses of the Timeline Server, the Yarn Resource Manager, and optionally the Resource Manager web proxy, as illustrated in the following example:

hosts: {
  /*
  * Timeline Server Address:
  * By default MR3 UI looks for timeline server at http://localhost:8188, uncomment
  * and change the following value for pointing to a different address.
  */
  timeline: "node0:8188",
  /*
  * Resource Manager Address:
  * By default RM REST APIs are expected to be at http://localhost:8088, uncomment
  * and change the following value to point to a different address.
  */
  rm: "node0:8088",
  /*
  * Resource Manager Web Proxy Address:
  * Optional - By default, value configured as RM host will be taken as proxy address
  * Use this configuration when RM web proxy is configured at a different address than RM.
  */
  rmProxy: "node0:8088",
}

By default, a DAGAppMaster does not communicate with the Timeline Server and thus MR3-UI displays nothing. In order to activate MR3-UI, the following keys should be set in mr3-site.xml.

  • mr3.app.history.logging.enabled should be set to true in order to display states of Applications and ContainerWorkers in MR3-UI.

  • mr3.dag.history.logging.enabled should be set to true in order to display states of DAGs and Vertexes in MR3-UI.

  • mr3.task.history.logging.enabled should be set to true in order to display states of Tasks and TaskAttempts in MR3-UI.

Using MR3-UI

The user can check the progress of DAGs by visiting the MR3-UI address specified by its installation directory.

This page shows the list of DAGs that are running or have been completed: mr3ui1

This page shows details of a DAG: mr3ui2

This page shows a graphical view of a DAG: mr3ui3

This page shows details of a Task: mr3ui4