CentOS7 install spark+ipython-nodebook

Time:2021-4-7

ipython-nodebook

  1. IPython notebook has become an important tool for teaching, computing and scientific research with Python.

  2. IPython notebook uses the browser as the interface to send requests to the background IPython server and display the results.

  3. In the browser interface, cell is used to save all kinds of information. There are many types of cell, including markdown unit for formatted text and code unit for code.


This paper mainly introduces the process of installing IPython nodebook on centos7 minimal

1. install ifconfig

yum search ifconfig
yum install net-tools.x86_64

2. install vim

yum search vim
yum install vim-enhanced

3. install wget

[[email protected] x]$ yum search wget
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.skyshe.cn
* extras: mirrors.163.com
* updates: mirrors.163.com
============================================================================================ N/S matched: wget =============================================================================================
wget.x86_64 : A utility for retrieving files using the HTTP or FTP protocols

 Name and summary matches only, use "search all" for everything.

[[email protected] x]$ yum install wget.x86_64

4. install Jdk

# green install jdk-7u80-linux-x64.gz
# edit /etc/profile add
## libin add ##

### JAVA ###
JAVA_HOME=/home/x/jdk
JAVA_BIN=$JAVA_HOME/bin
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/jre/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
"/etc/profile" 86L, 2035C

#/ etc / Profile: This is the first file used by the operating system to customize the user environment when the user logs in. It is applied to every user who logs in to the system. Valid for all users##

5. install Scala

# green install scala-2.10.4.tgz
# edit /etc/profile add

### Scala ###
#export SCALA_HOME=/usr/local/xSoft/scala
export SCALA_HOME=/home/x/scala
export PATH=${SCALA_HOME}/bin:$PATH

6. install Spark (Standalone)

green install spark-1.5.2-bin-hadoop2.6.tgz
cp conf/spark-env.sh.template conf/spark-env.sh

edit conf/spark-env.sh add

export JAVA_HOME=/home/x/jdk
export SCALA_HOME=/home/x/scala
export SPARK_HOME=/home/x/spark
export SPARK_MASTER_IP=192.168.181.113
export MASTER=spark://192.168.181.113:7077

export SPARK_EXECUTOR_INSTANCES=2
export SPARK_EXECUTOR_CORES=1

export SPARK_WORKER_MEMORY=1000m
export SPARK_EXECUTOR_MEMORY=300m

export SPARK_LIBRARY_PATH=${SPARK_HOME}/lib

#export SPARK_LAUNCH_WITH_SCALA=0
#export SCALA_LIBRARY_PATH=${SPARK_HOME}/lib


#export SPARK_LIBRARY_PATH=/home/deploy/spark/spark-1.5.2-bin-hadoop2.6/lib

7. install ipython-nodebook

openssh、zlib

yum -y install openssh-clients
yum install zlib

setuptools、pip

tar xvf setuptools-18.1.tar.gz
cd setuptools-18.1
sudo python setup.py build
sudo python setup.py install

tar xvf pip-8.1.0.tar.gz
cd pip-8.1.0
sudo python setup.py build
sudo python setup.py install

ipython、matplotlib

sudo pip install ipython
sudo pip install matplotlib

python-dev、g++

Sudo Yum install Python devel
sudo yum install gcc-c++

install python-notebook

#The various py of the previous install are related to this step

sudo pip install notebook

8. start-up notebook

PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook --ip=192.168.181.113" /home/x/spark/bin/pyspark

Browser accesshttp://192.168.181.113:8888/notebooks

CentOS7 install spark+ipython-nodebook

9. spark-notebook example1

%pylab inline
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

data =[33,25,20,12,10]
plt.figure(num=1, figsize=(6,6))
plt.axes(aspect=1)
plt.title('Plot 3', size=14)
plt.pie(data, labels=('Group 1','Group 2','Group 3','Group 4','Group 5'))
plt.savefig('/home/x/spark/test_libin/plot3.png', format='png')

CentOS7 install spark+ipython-nodebook

maybe attention point

python -V

#If the system is python2.6 by default, you need to upgrade to 2.7
tar xvf Python-2.7.tgz
./configure --with-zlib=/usr/include --prefix=/usr/local/python27 --prefix=/usr/local/python27

make
make install
mv /usr/bin/python /usr/bin/python_old
ln -s /usr/local/python27/bin/python /usr/bin/
python
You can use python2.7 normally here
But because Yum is used in version 2.6, it needs to be modified
[[email protected] Python-2.7.2]# vim /usr/bin/yum
#! / usr / bin / Python #