The implementation of downloading files with HTTP client in golang

Time:2019-12-5

Before using beego’s HTTP library, sometimes you need to download files. Beego can be implemented, but there is a problem: it does not support callback and cannot display download speed, which is intolerable in daily development.

I have seen that the implementation of beego mainly uses the io.copy function, so I have a deep look at the implementation principle and found that it is quite simple, so I implemented a simple downloader according to the io.copy principle

//Define files to download
var durl = "https://dl.google.com/go/go1.10.3.darwin-amd64.pkg";
// parse URL
uri, err := url.ParseRequestURI(durl)
if err != nil {
 Panic ("URL error")
}

In a normal process, first check if the URL has any errors


filename := path.Base(uri.Path)
log.Println("[*] Filename " + filename)

Use path.base to get the file name of the URL. Here’s a possible bug: if the URL is 302, you can’t get the file name after the jump.

In this step, you can send the HTTP request again. When you define a function called client.checkredirect, you can retrieve the file name.

client := http.DefaultClient;
Client. Timeout = time. Second * 60 // set timeout
resp, err := client.Get(durl)

Create an httpclient. In addition, the timeout of this client is to set the time-out when reading data.

I prefer to use the do method to pass a reqeust in the past, because some URLs need to check the HTTP header and so on. Why don’t you ask me to be lazy.


raw := resp.Body
defer raw.Close()
reader := bufio.NewReaderSize(raw, 1024*32);

In fact, it’s not clear that bufio can really speed up HTPP reading speed. There’s no comparison with no bufio. But it’s added for psychological comfort


file, err := os.Create(filename)
if err != nil {
 panic(err)
}
writer := bufio.NewWriter(file)

The same as above is not sure how fast bufio can increase file writing speed


buff := make([]byte, 32*1024)
written := 0
go func() {
 for {
  nr, er := reader.Read(buff)
  if nr > 0 {
   nw, ew := writer.Write(buff[0:nr])
   if nw > 0 {
    written += nw
   }
   if ew != nil {
    err = ew
    break
   }
   if nr != nw {
    err = io.ErrShortWrite
    break
   }
  }
  if er != nil {
   if er != io.EOF {
    err = er
   }
   break
  }
 }
 if err != nil {
  panic(err)
 }
}()

This source code is that I directly copy the io.copybuffer function. I just made some simple modifications to understand the general meaning

//Interval time
spaceTime := time.Second * 1
// timer
ticker := time.NewTicker(spaceTime)
//Last read data size
lastWtn := 0
stop := false

for {
 select {
 case <-ticker.C:
  //File size read this time - last read data size = speed
  speed := written - lastWtn
  log.Printf("[*] Speed %s / %s \n", bytesToSize(speed), spaceTime.String())
  if written-lastWtn == 0 {
   ticker.Stop()
   stop = true
   break
  }
  lastWtn = written
 }
 if stop {
  break
 }
}

This code blocks the program, and the timer calculates the speed based on the interval time. The possible bug here is to jump out of the loop directly when no data is read in the interval. I’m very sleepy at this time. I’m so sleepy to write a blog. I’ll be free to solve this bug later


func bytesToSize(length int) string {
 var k = 1024 // or 1024
 var sizes = []string{"Bytes", "KB", "MB", "GB", "TB"}
 if length == 0 {
  return "0 Bytes"
 }
 i := math.Floor(math.Log(float64(length)) / math.Log(float64(k)))
 r := float64(length) / math.Pow(float64(k), i)
 return strconv.FormatFloat(r, 'f', 3, 64) + " " + sizes[int(i)]
}

I converted this function from my personal PHP project


2018/08/17 00:24:50 [*] Filename go1.10.3.darwin-amd64.pkg
2018/08/17 00:24:51 [*] Speed 9.000 MB / 1s 
2018/08/17 00:24:52 [*] Speed 11.125 MB / 1s 
2018/08/17 00:24:53 [*] Speed 11.125 MB / 1s 
2018/08/17 00:24:54 [*] Speed 10.562 MB / 1s 
2018/08/17 00:24:55 [*] Speed 11.187 MB / 1s 
2018/08/17 00:24:56 [*] Speed 11.109 MB / 1s 
2018/08/17 00:24:57 [*] Speed 11.109 MB / 1s 
2018/08/17 00:24:58 [*] Speed 11.141 MB / 1s 
2018/08/17 00:24:59 [*] Speed 11.172 MB / 1s 
2018/08/17 00:25:00 [*] Speed 11.141 MB / 1s 
2018/08/17 00:25:01 [*] Speed 8.453 MB / 1s 
2018/08/17 00:25:02 [*] Speed 6.385 MB / 1s 
2018/08/17 00:25:03 [*] Speed 0 Bytes / 1s 

This is the final result of the operation, and then I put all the source code on the next to go to sleep.

package main

import (
 "net/http"
 "log"
 "time"
 "net/url"
 "path"
 "os"
 "io"
 "bufio"
 "math"
 "strconv"
)

var durl = "https://dl.google.com/go/go1.10.3.darwin-amd64.pkg";

func main() {
 uri, err := url.ParseRequestURI(durl)
 if err != nil {
  Panic ("URL error")
 }

 filename := path.Base(uri.Path)
 log.Println("[*] Filename " + filename)

 client := http.DefaultClient;
 Client. Timeout = time. Second * 60 // set timeout
 resp, err := client.Get(durl)
 if err != nil {
  panic(err)
 }
 if resp.ContentLength <= 0 {
  log.Println("[*] Destination server does not support breakpoint download.")
 }
 raw := resp.Body
 defer raw.Close()
 reader := bufio.NewReaderSize(raw, 1024*32);


 file, err := os.Create(filename)
 if err != nil {
  panic(err)
 }
 writer := bufio.NewWriter(file)

 buff := make([]byte, 32*1024)
 written := 0
 go func() {
  for {
   nr, er := reader.Read(buff)
   if nr > 0 {
    nw, ew := writer.Write(buff[0:nr])
    if nw > 0 {
     written += nw
    }
    if ew != nil {
     err = ew
     break
    }
    if nr != nw {
     err = io.ErrShortWrite
     break
    }
   }
   if er != nil {
    if er != io.EOF {
     err = er
    }
    break
   }
  }
  if err != nil {
   panic(err)
  }
 }()

 spaceTime := time.Second * 1
 ticker := time.NewTicker(spaceTime)
 lastWtn := 0
 stop := false

 for {
  select {
  case <-ticker.C:
   speed := written - lastWtn
   log.Printf("[*] Speed %s / %s \n", bytesToSize(speed), spaceTime.String())
   if written-lastWtn == 0 {
    ticker.Stop()
    stop = true
    break
   }
   lastWtn = written
  }
  if stop {
   break
  }
 }
}

func bytesToSize(length int) string {
 var k = 1024 // or 1024
 var sizes = []string{"Bytes", "KB", "MB", "GB", "TB"}
 if length == 0 {
  return "0 Bytes"
 }
 i := math.Floor(math.Log(float64(length)) / math.Log(float64(k)))
 r := float64(length) / math.Pow(float64(k), i)
 return strconv.FormatFloat(r, 'f', 3, 64) + " " + sizes[int(i)]
}

The above implementation method of download file of golang using HTTP client is all the content shared by Xiaobian. I hope it can give you a reference, and I hope you can support developpaer more.