Golang implements HTTP server to provide compressed file download function

Time:2022-5-24

Recently, I encountered a need to download static HTML reports. I need to complete the download function in the form of compressed package. In the process of implementation, I found that the relevant documents are very miscellaneous, so I summarize my implementation.

Development environment:

System environment: MacOS + Chrome
Frame: beego
Compression function: Tar + gzip
Target compressed file: static HTML file with its own data and all packages

When downloading a file, the header of the server will be directly set to the format of the downloaded file. In fact, when we download the file, the header of the server will be directly set to the format of the downloaded file. In this way, when we download the file, we will return to the front end first
The format of data header is as follows:

func (c *Controller)Download() {
  //... Generation logic of file information
  //
  //RW is responsewriter
  rw := c.Ctx.ResponseWriter
  //Specify the file name after downloading
  rw. Header(). Set ("content disposition", "attachment; filename =" + "(file name)")
  rw.Header().Set("Content-Description", "File Transfer")
  //Indicates the type of transfer file
  //For other types, please refer to: https://www.runoob.com/http/http-content-type.html
  rw.Header().Set("Content-Type", "application/octet-stream")
  rw.Header().Set("Content-Transfer-Encoding", "binary")
  rw.Header().Set("Expires", "0")
  rw.Header().Set("Cache-Control", "must-revalidate")
  rw.Header().Set("Pragma", "public")
  rw.WriteHeader(http.StatusOK)
  //The file transfer is of byte slice type. In this example, B is a byte Buffer, you need to call b.bytes()
  http. Servercontent (RW, c.ctx. Request, "(file name)", time Now(), bytes. NewReader(b.Bytes()))
}

In this way, the beego back end will send the data packet marked as the download file in the header to the front end, and the front end will automatically start the download function after receiving it.

However, this is only the last step. How to compress our files first and then send them to the front end for download?

If more than one file needs to be downloaded, it needs to be packaged with tar and compressed with gzip. The implementation is as follows:

//The innermost layer uses bytes Buffer to store files
  var b bytes.Buffer
  //Writer of nested tar package and writer of gzip package
  gw := gzip.NewWriter(&b)
  tw := tar.NewWriter(gw)


  dataFile := //... File generation logic, datafile is file type
  info, _ := dataFile.Stat()
  header, _ := tar.FileInfoHeader(info, "")
  //Path setting of current file after downloading
  header.Name = "report" + "/" + header.Name
  err := tw.WriteHeader(header)
  if err != nil {
    utils.LogErrorln(err.Error())
    return
  }
  _, err = io.Copy(tw, dataFile)
  if err != nil {
    utils.LogErrorln(err.Error())
  }
  //... You can continue adding files
  //The closing order of tar writer and gzip writer must not be reversed
  tw.Close()
  gw.Close()

After the final and intermediate steps are completed, we only have the generation logic of file file. Because it is a static HTML file, we need to write all the dependency packages referenced by HTML completely under the < script > and < style > tags in the generated file. In addition, in this example, the report part also needs some static JSON data to fill the table and image, which is stored in memory as a map. Of course, you can save it as a file before packaging and compressing it in the above step, but this will cause concurrency problems. Therefore, we need to write all dependent package files and data into a byte In the buffer, the byte Buffer returns to file format.

There is no written byte in golang The function of buffer to file can be used, so we need to implement it ourselves.

The implementation is as follows:


type myFileInfo struct {
  name string
  data []byte
}

func (mif myFileInfo) Name() string    { return mif.name }
func (mif myFileInfo) Size() int64    { return int64(len(mif.data)) }
func (mif myFileInfo) Mode() os.FileMode { return 0444 }    // Read for all
func (mif myFileInfo) ModTime() time.Time { return time.Time{} } // Return whatever you want
func (mif myFileInfo) IsDir() bool    { return false }
func (mif myFileInfo) Sys() interface{}  { return nil }

type MyFile struct {
  *bytes.Reader
  mif myFileInfo
}

func (mf *MyFile) Close() error { return nil } // Noop, nothing to do

func (mf *MyFile) Readdir(count int) ([]os.FileInfo, error) {
  return nil, nil // We are not a directory but a single file
}

func (mf *MyFile) Stat() (os.FileInfo, error) {
  return mf.mif, nil
}

Dependent package and data write logic:

func testWrite(data map[string]interface{}, taskId string) http.File {
  //Finally generated HTML, open the HTML template
  tempfileP, _ := os.Open("views/traffic/generatePage.html")
  info, _ := tempfileP.Stat()
  html := make([]byte, info.Size())
  _, err := tempfileP.Read(html)
  //Write data to HTML
  var b bytes.Buffer
  //Create JSON encoder
  encoder := json.NewEncoder(&b)

  err = encoder.Encode(data)
  if err != nil {
    utils.LogErrorln(err.Error())
  }
  
  //Add JSON data to the HTML template
  //The method is to insert a special replacement field in the HTML template. In this case, {data_json_source}
  html = bytes.Replace(html, []byte("{Data_Json_Source}"), b.Bytes(), 1)

  //Add static files to HTML
  //If so CSS, add < style > < / style > tags before and after
  //If so JS, add < script > < script > tags before and after
  allStaticFiles := make([][]byte, 0)
  //JQuery needs to be added first
  tempfilename := "static/report/jquery.min.js"

  tempfileP, _ = os.Open(tempfilename)
  info, _ = os.Stat(tempfilename)
  curFileByte := make([]byte, info.Size())
  _, err = tempfileP.Read(curFileByte)

  allStaticFiles = append(allStaticFiles, []byte("<script>"))
  allStaticFiles = append(allStaticFiles, curFileByte)
  allStaticFiles = append(allStaticFiles, []byte("</script>"))
  //All remaining static files
  staticFiles, _ := ioutil.ReadDir("static/report/")
  for _, tempfile := range staticFiles {
    if tempfile.Name() == "jquery.min.js" {
      continue
    }
    tempfilename := "static/report/" + tempfile.Name()

    tempfileP, _ := os.Open(tempfilename)
    info, _ := os.Stat(tempfilename)
    curFileByte := make([]byte, info.Size())
    _, err := tempfileP.Read(curFileByte)
    if err != nil {
      utils.LogErrorln(err.Error())
    }
    if isJs, _ := regexp.MatchString(`\.js$`, tempfilename); isJs {
      allStaticFiles = append(allStaticFiles, []byte("<script>"))
      allStaticFiles = append(allStaticFiles, curFileByte)
      allStaticFiles = append(allStaticFiles, []byte("</script>"))
    } else if isCss, _ := regexp.MatchString(`\.css$`, tempfilename); isCss {
      allStaticFiles = append(allStaticFiles, []byte("<style>"))
      allStaticFiles = append(allStaticFiles, curFileByte)
      allStaticFiles = append(allStaticFiles, []byte("</style>"))
    }
    tempfileP.Close()
  }
  
  //Convert to http Return in file format
  mf := &MyFile{
    Reader: bytes.NewReader(html),
    mif: myFileInfo{
      name: "report.html",
      data: html,
    },
  }
  var f http.File = mf
  return f
}

OK! So far, the back-end file generation – > Packaging – > compression have been completed. Let’s string them together:

func (c *Controller)Download() {
  var b bytes.Buffer
  gw := gzip.NewWriter(&b)

  tw := tar.NewWriter(gw)

  //Dynamically compress the report and add it
  //Call the testwrite method above
  dataFile := testWrite(responseByRules, strTaskId)
  info, _ := dataFile.Stat()
  header, _ := tar.FileInfoHeader(info, "")
  header.Name = "report_" + strTaskId + "/" + header.Name
  err := tw.WriteHeader(header)
  if err != nil {
    utils.LogErrorln(err.Error())
    return
  }
  _, err = io.Copy(tw, dataFile)
  if err != nil {
    utils.LogErrorln(err.Error())
  }

  tw.Close()
  gw.Close()
  rw := c.Ctx.ResponseWriter
  rw.Header().Set("Content-Disposition", "attachment; filename="+"report_"+strTaskId+".tar.gz")
  rw.Header().Set("Content-Description", "File Transfer")
  rw.Header().Set("Content-Type", "application/octet-stream")
  rw.Header().Set("Content-Transfer-Encoding", "binary")
  rw.Header().Set("Expires", "0")
  rw.Header().Set("Cache-Control", "must-revalidate")
  rw.Header().Set("Pragma", "public")
  rw.WriteHeader(http.StatusOK)
  http.ServeContent(rw, c.Ctx.Request, "report_"+strTaskId+".tar.gz", time.Now(), bytes.NewReader(b.Bytes()))
}

The back-end part has been fully implemented. How can the front-end part receive it? In this example, I made a button nested < a > tag to make a request:

<a href="/traffic/download_indicator?task_id={{$.taskId}}&task_type={{$.taskType}}&status={{$.status}}&agent_addr={{$.agentAddr}}&glaucus_addr={{$.glaucusAddr}}" rel="external nofollow" >
   <button style="font-family: 'SimHei';font-size: 14px;font-weight: bold;color: #0d6aad;text-decoration: underline;margin-left: 40px;"  Type = "button" > Download report < / button >
</a>

In this way, after clicking the download Report button in the current end page, the download will be automatically started to download the report returned from our back end tar. GZ file.

This is the end of this article about golang’s implementation of HTTP server’s compressed file download function. For more compressed download content of golang HTTP server, please search the previous articles of developeppaer or continue to browse the relevant articles below. I hope you can support developeppaer in the future!