Workflow of logagent


Project architecture design

Workflow of logagent

The workflow of logagent is as follows:

  1. Read log–tailfThird party Library

func main() {

fileName := "./my.log"
config := tail.Config{
    Reopen: true, // reopen
    Follow: true, // follow
    Location:  & tail.SeekInfo {offset: 0, when: 2}, // where to start reading the file
    Mustexist: false, // no error will be reported if the file does not exist
    Poll:      true,
tails, err := tail.TailFile(fileName, config)
if err != nil {
    fmt.Println("tail file failed, err:", err)
var (
    line *tail.Line
    ok   bool
for {
    line, ok = <-tails.Lines
    if !ok {
        fmt.Printf("tail file close reopen, filename:%s\n", tails.Filename)
    fmt.Println("line:", line.Text)

2. Write a log to Kafka–saramaThird party Library

The version after Sarama v1.20 adds the zstd compression algorithm, which needs to go to CGO. When compiling on the windows platform, you will be prompted something like this
The following error occurred:
exec: "gcc":executable file not found in %PATH%

So on the windows platform, please make the version of Sarama of v1.19.

Workflow of logagent

//Kafka client based on the third Library of Sarama

func main() {
config := sarama.NewConfig()
//Tail package
config.Producer.RequiredAcks  =  sarama.WaitForAll  //After sending data, both leader and follow need to confirm
config.Producer.Partitioner  =  sarama.NewRandomPartitioner  //New partition
config.Producer.Return . successes = true // successfully delivered messages will be returned in success channel

//Construct message
msg := &sarama.ProducerMessage{}
msg.Topic = "web_log"
msg.Value = sarama.StringEncoder("this is a test log")

//Connect Kafka
client, err := sarama.NewSyncProducer([]string{""}, config)
if err != nil {
    fmt.Println("producer closed, err:", err)
fmt.Println "Connect Kafka successfully! ")
defer client.Close()
//Send message
pid, offset, err := client.SendMessage(msg)
if err != nil {
    fmt.Println("send msg failed, err:", err)
fmt.Printf("pid:%v offset:%v\n", pid, offset)
fmt.Println "Send successfully! ")}

Kafka and zookeeper

Introduction to tail . V1 third party documents……

Instance code
Workflow of logagent