The user can rebuild all the components of Hive-MR3 from the source code (except MR3 which is currently distributed in binary form only).
The current Hive-MR3 release is built with Java 1.8 (and Scala 2.11 for MR3), but
any version of Java that compiles Hive and Tez should also work.
To rebuild Hive-MR3, download a minimal Hive-MR3 release and the latest source code:
- Hive-MR3 0.1 based on Hive 1.2.2 [Source code]
- Hive-MR3 0.1 based on Hive 2.1.1 [Source code]
- Hive-MR3 0.1 based on Hive 2.3.2 [Source code]
- Tez 0.7.0 for MR3 [Source code]
- Tez 0.8.4 for MR3 [Source code]
- Tez 0.9.1 for MR3 [Source code]
Then set the following environment variables in
env.sh to specify the directories of the source code and the revision numbers:
# Used with --tezsrc1 option in scripts TEZ1_SRC=~/tez-mr3 TEZ1_REV=0.7.0.mr3.0.1 # Used with --tezsrc2 option in scripts TEZ2_SRC=~/tez2-mr3 TEZ2_REV=0.8.4.mr3.0.1 # Used with --tezsrc3 option in scripts TEZ3_SRC=~/tez3-mr3 TEZ3_REV=0.9.1.mr3.0.1 # Used with --hivesrc1 option in scripts HIVE1_SRC=~/hive1-mr3 HIVE1_REV=1.2.2 # Used with --hivesrc2 option in scripts HIVE2_SRC=~/hive2-mr3 HIVE2_REV=2.3.2 # Used with --hivesrc3 option in scripts HIVE3_SRC=~/hive3-mr3 HIVE3_REV=2.1.1
Because of the compilation dependency between Hive-MR3 and Tez, the user should rebuild first Tez for MR3 and then Hive-MR3.
To compile Tez for MR3, execute
tez/compile-tez.sh with the following options:
--tezsrc1 # Choose tez1-mr3 (based on Tez 0.7.0, default for --hivesrc1). --tezsrc2 # Choose tez2-mr3 (based on Tez 0.8.4, default for --hivesrc3). --tezsrc3 # Choose tez3-mr3 (based on Tez 0.9.1, default for --hivesrc2). <mvn option> # Add a Maven option.
Before proceeding to compile Hive-MR3, the user should compile all three versions of Tez for MR3 separately.
To compile Hive-MR3, execute
hive/compile-hive.sh with the following options:
--hivesrc1 # Choose hive1-mr3 (based on Hive 1.2.2) (default). --hivesrc2 # Choose hive2-mr3 (based on Hive 2.3.2). --hivesrc3 # Choose hive3-mr3 (based on Hive 2.1.1). <mvn option> # Add a Maven option.
The user can append as many Maven options as necessary to the command.
These scripts invoke Maven to compile the source code, and automatically update the local Maven repository as well as
They also upload the new jar files to HDFS, so the user does not need to execute