DWQA QuestionsCategory: ServerAppend HDFS error is already the current lease holder
chick asked 3 weeks ago

When Kafka is used to write HDFS, the logic is as follows:

  1. Writing HDFS the first time creates a file
  2. The second time and the next step will be to append data to this file

In fact, there are two errors reported

  1. The first error message

    Caused by: org.apache.hadoop.ipc.RemoteException: append: lastBlock=blk_1073851558_23293749 of src=/logs/4_CS.JCY/2019/06/03/T_LOG_RECORD_CS/web-Thread-3 is not sufficiently replicated yet.

    The prompt seems to be due to the problem of the number of backups written. After inquiring about the offline problem, someone really encountered it. I changed the number of HDFS backups written by this consumer to 1, that is, only one copy would be written. The next error really did not happen, if there is a better way to deal with it, please let me know!

  2. Then there was a second error

    Caused by:  org.apache.hadoop . ipc.RemoteException : Failed to APPEND_ FILE /logs/4_ CS.JCY/2019/07/18/T_ LOG_ RECORD_ CS/web-Thread-3 for DFSClient_ NONMAPREDUCE_ -558981961_ 18 on IP address because dfsclient_ NONMAPREDUCE_ -558981961_ 18 is already the current lease holder.
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2970)
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2766)
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3073)
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3042)
            at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:760)
            at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:429)
            at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
            at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
            at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
            at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
            at org.apache.hadoop.ipc.Client.call(Client.java:1475)
            at org.apache.hadoop.ipc.Client.call(Client.java:1412)
            at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
            at com.sun.proxy.$Proxy60.append(Unknown Source)
            at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.append(ClientNamenodeProtocolTranslatorPB.java:328)
            at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
            at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
            at com.sun.proxy.$Proxy61.append(Unknown Source)
            at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1808)
            at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1877)
            at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1847)
            at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:340)
            at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:336)
            at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
            at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:348)
            at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:318)
            at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1166)
            at com.sharing.hdfs.HdfsHelper$.getFSDataOutputSteam(HdfsHelper.scala:225)
            at com.sharing.hdfs.HdfsHelper$.writeLines(HdfsHelper.scala:149)

    Error information should be the problem of lease expiration. We tried the method of updating lease introduced on the Internet, but there will be error information every day. Although it can be recovered for a period of time after the error is reported, it is not a very comfortable way to deal with it!

Related codes

/**
    *Create fsdataoutputstream
    *
    * @param path
    * @return
    */
  def getFSDataOutputSteam(path: String): FSDataOutputStream = {
    val pathIsExist = isExist(path)
    if (pathIsExist) {
      fileSystem.append(path)
    } else {
      fileSystem.create(path)
    }
  }
/**
    *Write multiple lines
    *
    * @param lines
    * @param path
    */
  def writeLines(lines: java.util.List[String], path: String): Unit = {


    var outputStream: FSDataOutputStream = null
    try {

      outputStream = getFSDataOutputSteam(path)
      lines.foreach({
        line =>
          outputStream.write(line.trim.getBytes("UTF-8"))
          outputStream.write("\n".getBytes("UTF-8"))
      })
      LOGGER.info ("HDFS write succeeded! ""
    }
    catch {
      case e: IOException =>
        LOGGER.error(e.getMessage, e)
        updateLease(path)
        throw new IOException(e)
    } finally {
      if (Objects.nonNull(outputStream)) {
        outputStream.close()
      }
    }
  }
//I don't know if it's useful to renew the lease
  def updateLease(path: String): Unit = {

    LOGGER.error (start of lease renewal)
    fileSystem.close()
    fileSystem = FileSystem.get(uri, conf, userName).asInstanceOf[DistributedFileSystem]
    fileSystem.recoverLease(path)
    TimeUnit.SECONDS.sleep(3)
  }

What are the results you are looking forward to?
Hope to have a more regular and effective method to write HDFS. At present, I feel that using HDFS append API will always report errors! At present, we encounter the above two problems.

Li Hao replied 3 weeks ago

Hello, you have solved this problem. I also have a problem with this lease. I don’t know how to solve it

Li Hao replied 3 weeks ago

I didn’t solve it.. It will be reported several times a day, and then a filesystem will be reconstructed to update the lease. The process will be executed more than 10 times…

Li Hao replied 3 weeks ago

Every day will report a mistake! However, through the Kafka mechanism, the message is not repeatedly consumed or lost

1 Answers
user_PlihwqYX answered 3 weeks ago

Brother, do you have a good way? I’m similar to your scenario. Kafka reads and writes HDFS, uses springboot to initialize a filesystem bean, and then runs and reports an error. Already the current lease holder