Compiling Hive on MR3

The user can rebuild all the components of Hive running on top of MR3 from the source code (except MR3 which is currently distributed in binary form only). The current MR3 release is built with Java 1.8 for Hive and Scala 2.11 for MR3, but any version of Java that compiles Hive and Tez should also work.

To rebuild Hive on MR3, download a minimal MR3 release and the latest source code:

Then set the following environment variables in env.sh to specify the directories of the source code and the revision numbers:

# Used with --tezsrc1 option in scripts 
TEZ1_SRC=~/tez-mr3
TEZ1_REV=0.7.0.mr3.0.1
# Used with --tezsrc3 option in scripts 
TEZ3_SRC=~/tez3-mr3
TEZ3_REV=0.9.1.mr3.0.1

# Used with --hivesrc1 option in scripts 
HIVE1_SRC=~/hive1-mr3
HIVE1_REV=1.2.2
# Used with --hivesrc2 option in scripts 
HIVE2_SRC=~/hive2-mr3
HIVE2_REV=2.3.3
# Used with --hivesrc5 option in scripts
HIVE5_SRC=~/hive5-mr3
HIVE5_REV=3.0.0

Because of the compilation dependency between Hive and Tez, the user should rebuild first Tez and then Hive. To compile Tez, execute tez/compile-tez.sh with the following options:

--tezsrc1                 # Choose tez1-mr3 (based on Tez 0.7.0) (default).
--tezsrc3                 # Choose tez3-mr3 (based on Tez 0.9.1).
<mvn option>              # Add a Maven option.

To compile Hive, execute hive/compile-hive.sh with the following options:

--hivesrc1                # Choose hive1-mr3 (based on Hive 1.2.2) (default).
--hivesrc2                # Choose hive2-mr3 (based on Hive 2.3.3).
--hivesrc5                # Choose hive5-mr3 (based on Hive 3.0.0).
<mvn option>              # Add a Maven option.

The user can append as many Maven options as necessary to the command. These scripts invoke Maven to compile the source code, and automatically update the local Maven repository as well as hive/hivejar and tez/tezjar directories. They also upload the new jar files to HDFS, so the user does not need to execute mr3/upload-hdfslib-mr3.sh and tez/upload-hdfslib-tez.sh again.