• Client cannot authenticate XXX:[TOKEN, KERBEROS]


    Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate xxx:[TOKEN, KERBEROS] The security authentication failed, and the cause analysis is as follows:1. Check whether the Kerberos address can be connected normally // kdc’s ip System.setProperty(“java.security.krb5.kdc”, “”); // realm System.setProperty(“java.security.krb5.realm”, “XXX”); 2. The configuration file failed to be read successfully UserGroupInformation.loginUserFromKeytab(user name, path to keytab file); It is recommended […]

  • Can’t get Kerberos realm


    When using HDFS to connect to Hadoop, Keytab authentication is required, but an error is reported: Can’t get Kerberos realm Solution: Add two lines of code to the configuration code System.setProperty(“java.security.krb5.realm”, “XXX.COM”);System.setProperty(“java.security.krb5.kdc”, “XXX.COM”); XXX.COM is obtained from the krb5.cof configuration file screenshot:

  • Getting to know the Hadoop architecture


    Getting to know hadoop Google’s “troika” GFS MapReduce BigTable HDFS HDFSwhat is it HDFSIt is a distributed file system based on streaming data access mode, supports the storage of massive data, and allows users to form storage clusters with hundreds or thousands of computers. Advantages: can handle very large files, support streaming data access (write […]

  • Vivo’s 10,000-scale HDFS cluster upgrade HDFS 3.x practice


    vivo Internet Big Data Team-Lv Jia The first stable version of Hadoop 3.x was released at the end of 2017 with many major improvements. In terms of HDFS, new features such as Erasure Coding, More than 2 NameNodes, Router-Based Federation, Standby NameNode Read, FairCallQueue, and Intra-datanode balancer are supported. These new features bring many benefits […]

  • Tech Talk | The Evolution of HDFS in Shopee


    Topics of this issue This issue of sharing will introduce how the distributed file system HDFS (Hadoop Distributed File System) developed from a small cluster to a federated cluster consisting of thousands of nodes in Shopee, as well as the problems and solutions we encountered during this process. Through this sharing, you will be able […]

  • One Side of Data: Architecture Design and Practice of Hadoop Migration on the Cloud


    background Founded in 2014, Yifang Data is a leading data intelligence solution provider that provides real-time and comprehensive data insights by interpreting massive data from e-commerce platforms and social media channels. Long-term service for global FMCG giants (Procter & Gamble, Unilever, Mars, etc.), and has been widely recognized by the industry. The company has cooperated […]

  • JuiceFS Metadata Engine Selection Guide


    The file system is our common storage form, which is mainly composed of data and metadata. The data is the specific content of the file, which is usually displayed directly to the user; and the metadata is the data describing the data, which is used to record the file attributes, directory structure, data storage location, […]