Mean Partitions Block-Block Matrix Multiplication of Sparse Matrix Fall 2018 Project
Siddhant Benadikar, Sarthak Kothari, Sahil Gandhi, Mustafa Kapadia.
These components are installed:
- JDK 1.8
- Hadoop 2.9.1
- Maven
- AWS CLI (for EMR execution)
-
Example ~/.bash_aliases: export JAVA_HOME=$(/usr/libexec/java_home) export HADOOP_HOME=/usr/local/opt/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
-
Explicitly set JAVA_HOME in $HADOOP_HOME/etc/hadoop/hadoop-env.sh: export JAVA_HOME=/usr/lib/jvm/java-8-oracle
Use the python script GenData.py with 4 arguments.
python GenData.py fly dim1 dim2 density
It is recommended to keep the first argument as fly for dimensions > 10,000.
All of the build & execution commands are organized in the Makefile.
- Unzip project file.
- Open command prompt.
- Navigate to directory where project files unzipped.
- We have two Makefiles, one for Cannons and BB respectively. Please change the name from Makefile_BB and Makefile_Cannon to Makefile and run make.
- To change the partitions size you need to make the following changes to BBAlgorithm.java and CannonAlgorithm.java: In BBAlgorithm.java, change values of A_ROW, A_COL, B_ROW, B_COL, P_ROW, P_MID, P_COL Where P_ROW * P_MID * P_COL == the number of partitions you want to keep. (eg: For p=100, P_ROW=5, P_MID=4, P_COL=4)
- Standalone Hadoop: make switch-standalone -- set standalone Hadoop environment (execute once) make local
- AWS EMR Hadoop: (you must configure the aws.* config parameters at top of Makefile) make upload-input-aws -- only before first execution make aws -- check for successful execution with web interface (aws.amazon.com) download-output-aws -- after successful execution & termination