Installing Hive on MR3
Compiling Hive on MR3
Configuring Hive on MR3
Running the TPC-DS Benchmark
Using the Shuffle Handler
Enabling High Availability
Changing the Logging Configuration
Enabling ACID Transactions
Using User Defined Functions
Integrating Apache Ranger
Accessing from Spark
Compiling Hive on MR3
The user can rebuild all the components of Hive running on top of MR3 from the source code (except MR3 which is currently distributed in binary form only). The current MR3 release is built with Java 1.8 for Hive and Scala 2.11 for MR3, but any version of Java that compiles Hive and Tez should also work.
To rebuild Hive on MR3, download a minimal MR3 release and the latest source code:
- Hive based on Hive 1.2.2 [Source code]
- Hive based on Hive 2.3.5 [Source code]
- Hive based on Hive 3.1.1 [Source code]
- Hive based on Hive 4.0.0 [Source code]
- Tez for MR3 [Source code]
Then set the following environment variables in
env.sh to specify the directories of the source code and the revision numbers:
# Tez-MR3 source directory and its revision number (for compiling Tez-MR3) TEZ_SRC=~/tez3-mr3 TEZ_REV=0.9.1.mr3.0.1 # Used with --hivesrc1 option in scripts HIVE1_SRC=~/hive1-mr3 HIVE1_REV=1.2.2 # Used with --hivesrc2 option in scripts HIVE2_SRC=~/hive2-mr3 HIVE2_REV=2.3.5 # Used with --hivesrc3 option in scripts HIVE3_SRC=~/hive5-mr3 HIVE3_REV=3.1.1 # Used with --hivesrc4 option in scripts HIVE4_SRC=~/hive4-mr3 HIVE4_REV=4.0.0-SNAPSHOT
Because of the compilation dependency between Hive and Tez, the user should rebuild first Tez and then Hive.
To compile Tez, execute
tez/compile-tez.sh with additional Maven options:
<mvn option> # Add a Maven option.
To compile Hive, execute
hive/compile-hive.sh with the following options:
--hivesrc1 # Choose hive1-mr3 (based on Hive 1.2.2). --hivesrc2 # Choose hive2-mr3 (based on Hive 2.3.5). --hivesrc3 # Choose hive3-mr3 (based on Hive 3.1.1) (default). --hivesrc4 # Choose hive4-mr3 (based on Hive 4.0.0-SNAPSHOT). <mvn option> # Add a Maven option.
The user can append as many Maven options as necessary to the command.
These scripts invoke Maven to compile the source code, and automatically update the local Maven repository as well as
They also upload the new jar files to HDFS, so the user does not need to execute