Data analysis and prediction of Taobao double 11 — “pit point” summary

Time:2020-1-22

During the National Day holiday, I stayed in school to do Dr. Lin Ziyu’s big data experiment — Taobao double 11 data analysis and prediction. Although the tutorial was written in detail, I still stepped on many holes. Now I record it and hope to give others some help.

Taobao double 11 data analysis and prediction experiment link: click here

  • Problem 1: error in installing Hadoop

Error content:

Starting namenodes on [localhost]
Localhost: MKDIR: unable to create directory '/ usr / local / Hadoop / logs': insufficient permissions
Localhost: chown: unable to access' / usr / local / Hadoop / logs': no file or directory
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-dblab-namenode-dblab-virtual-machine.out
Localhost: / usr / local / Hadoop / SBIN / hadoop-daemon.sh: Line 159: / usr / local / Hadoop / logs / Hadoop dblab namenode dblab virtual machine.out: no file or directory
Localhost: head: unable to open "/ usr / local / Hadoop / logs / Hadoop dblab namenode dblab virtual machine. Out" read data: no file or directory
Localhost: / usr / local / Hadoop / SBIN / hadoop-daemon.sh: Line 177: / usr / local / Hadoop / logs / Hadoop dblab namenode dblab virtual machine.out: no file or directory
Localhost: / usr / local / Hadoop / SBIN / hadoop-daemon.sh: Line 178: / usr / local / Hadoop / logs / Hadoop dblab namenode dblab virtual machine.out: no file or directory
Localhost: MKDIR: unable to create directory '/ usr / local / Hadoop / logs': insufficient permissions
Localhost: chown: unable to access' / usr / local / Hadoop / logs': no file or directory
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-dblab-datanode-dblab-virtual-machine.out
Localhost: / usr / local / Hadoop / SBIN / hadoop-daemon.sh: Line 159: / usr / local / Hadoop / logs / Hadoop dblab datanode dblab virtual machine.out: no file or directory
Localhost: head: unable to open "/ usr / local / Hadoop / logs / Hadoop dblab datanode dblab virtual machine. Out" read data: no file or directory
Localhost: / usr / local / Hadoop / SBIN / hadoop-daemon.sh: Line 177: / usr / local / Hadoop / logs / Hadoop dblab datanode dblab virtual machine.out: no file or directory
Localhost: / usr / local / Hadoop / SBIN / hadoop-daemon.sh: Line 178: / usr / local / Hadoop / logs / Hadoop dblab datanode dblab virtual machine.out: no file or directory
Starting secondary namenodes [0.0.0.0]
0.0.0.0: MKDIR: could not create directory '/ usr / local / Hadoop / logs': insufficient permissions
0.0.0.0: chown: unable to access' / usr / local / Hadoop / logs': no file or directory
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-dblab-secondarynamenode-dblab-virtual-machine.out
0.0.0.0: / usr / local / Hadoop / SBIN / hadoop-day.sh: Line 159: / usr / local / Hadoop / logs / Hadoop dblab secondarynamenode dblab virtual machine.out: no file or directory
0.0.0.0: head: unable to open "/ usr / local / Hadoop / logs / Hadoop dblab secondarynamenode dblab virtual machine. Out" read data: no file or directory
0.0.0.0: / usr / local / Hadoop / SBIN / hadoop-day.sh: Line 177: / usr / local / Hadoop / logs / Hadoop dblab secondarynamenode dblab virtual machine.out: no file or directory
0.0.0.0: / usr / local / Hadoop / SBIN / hadoop-day.sh: Line 178: / usr / local / Hadoop / logs / Hadoop dblab secondarynamenode dblab virtual machine.out: no file or directory
[email protected]:/usr/local/hadoop$ sudo chown -R niugen /usr/local/hadoop

Error reason: insufficient authority

resolvent:

Sudo chown - r your own computer to username / usr / local / Hadoop
  • Problem 2: error: java_home is not set and could not be found

Solution: Java home environment variable is not set, please set Java home in the. Bashrc file according to the tutorial. If
Error: java_home is not set and could not be found. Go to the installation directory of Hadoop to modify the configuration file “/ usr / local / Hadoop / etc / Hadoop / Hadoop env. Sh”, find the line “export java_home = ${java_home}”, and then modify it to the specific address of the Java installation path, for example,
“Export java_home = / usr / lib / JVM / go to Java version yourself”, and then start Hadoop again.

  • Question 3: use sqoop to import hive data into MySQL

Error content:

ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: YES)
...
ERROR tool.ExportTool: Encountered IOException running export job: java.io.IOException: No columns to generate for ClassWriter

Error reason: insufficient operation authority
resolvent:
First, use the command service mysql start to ensure that MySQL has been started. Enter the operation interface MySQL – U root – P, and enter the MySQL password as prompted. Then execute the following command to modify the remote login permission

mysql> grant all privileges on *.* to [email protected] identified by 'root';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.04 sec)
  • Problem 4: error in importing hive data to MySQL

Error content:

ERROR:"Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient"

Error reason: failed to communicate with hive successfully when changing database.
Solution: on the premise of ensuring that hive and MySQL configuration steps are executed correctly, enter the following command at the terminal

Source / usr / local / hive / scripts / Metastore / upgrade / MySQL / hive-schema-3.0.0. Mysql. SQL // make this version of the configuration file useful! Pay attention to changing the file path 

Schematool - dbtype MySQL - initschema // initialize it. After running successfully, you can communicate normally.
  • Question 5: when creating a new Java Web project, no dynamic web project was found

Reason for the problem: the installation is dedicated to developing Java projects, while the dynamic web project belongs to J2EE technology, so you need to download an eclipse (eclipse Jee Helios Sr2 Win32) that integrates J2EE plug-ins
resolvent:
Method 1: uninstall the previous version of eclipse, and download the corresponding version of IDE on the Eclipse Official Website
Official website installation link: click here

Method 2: view the blog and click here(If you fail to execute this method in the original eclipse version, please go to method 1

  • Question 6: Eclipse imports external jar packages

PS: it’s not so much a problem as a consideration. Before that day, I thought that it was OK to import the jar package as long as I drag it in with the left mouse button. In fact, this is wrong.

Correct procedure: view this link

Recommended Today

RCAST 35: add type to currency

– font ALT: Simsun; MSO font charset: 134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} @font-face {font-family:”Cambria Math”; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:1; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:Calibri; Variable; Ose-1: 216301111; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} /\* Style […]