Usage details of golang elasticsearch client

Time:2021-9-23

The client of elasticsearch establishes a connection through newclient, sets the access address through set.url in newclient, and setsniff sets the cluster

After obtaining the connection, insert the data through the index method, and obtain the data through the get method (the get method of elasticsearch client will be used in the final test case)

func Save(item interface{}) {
    client, err := elastic.NewClient(
  elastic.SetURL("https://192.168.174.128:9200/"),
  // Must turn off sniff in docker
  elastic.SetSniff(false),
 )
 
 if err != nil {
  panic(err)
 }
 
 resp, err := client.Index().
  Index("dating_profile").
  Type("zhenai").
  BodyJson(item).
  Do (context. Background()) // context needs context package
 if err != nil {
  panic(err)
 }
 
 fmt.Printf("%+v", resp)
 
}

Test program, define a data structure profile to test

func TestSave(t *testing.T) {
 profile := model.Profile{
  Age:        34,
  Height:     162,
  Weight:     57,
  Income: "3001-5000 yuan",
  Gender: "female",
  Name: "quiet snow",
  Xingzuo: "Aries",
  Occupation: "personnel / Administration",
  Marriage: "divorce",
  House: "purchased",
  Hukou: "Heze, Shandong",
  Education: "undergraduate course",
  Car: "no car purchased",
 }
 
 Save(profile)
}

Go test succeeded

Check whether the data exists in elasticsearch through the get method

We use panic in test and error return in function. Take out the stored data from elasearch and compare it with the data defined by us,

Therefore, the ID of the inserted data needs to be returned in save


func Save(item interface{}) (id string, err error) {
	client, err := elastic.NewClient(
		elastic.SetURL("https://192.168.174.128:9200/"),
		// Must turn off sniff in docker
		elastic.SetSniff(false),
	)
 
	if err != nil {
		return "", err
	}
 
	resp, err := client.Index().
		Index("dating_profile").
		Type("zhenai").
		BodyJson(item).
		Do(context.Background())
	if err != nil {
		return "", err
	}
 
	return resp.Id, nil
 
}

test case

package persist
 
import (
	"context"
	"encoding/json"
	"my_crawler_single/model"
	"testing"
 
	elastic "gopkg.in/olivere/elastic.v5"
)
 
func TestSave(t *testing.T) {
	expected := model.Profile{
		Age:        34,
		Height:     162,
		Weight:     57,
		Income: "3001-5000 yuan",
		Gender: "female",
		Name: "quiet snow",
		Xingzuo: "Aries",
		Occupation: "personnel / Administration",
		Marriage: "divorce",
		House: "purchased",
		Hukou: "Heze, Shandong",
		Education: "undergraduate course",
		Car: "no car purchased",
	}
 
	id, err := Save(expected)
	if err != nil {
		panic(err)
	}
 
	client, err := elastic.NewClient(
		elastic.SetURL("https://192.168.174.128:9200/"),
		elastic.SetSniff(false),
	)
	if err != nil {
		panic(err)
	}
 
	resp, err := client.Get().
		Index("dating_profile").
		Type("zhenai").
		ID (ID). // find the piece of data with the specified ID
		Do(context.Background())
	if err != nil {
		panic(err)
	}
 
	t.Logf("%+v", resp)
	//From the print, we can know that the data is in resp.source, and we can also know from the screenshot of rest client
 
	var actual model.Profile
	//Check * resp.source to see that its data type is [] byte
	err = json.Unmarshal(*resp.Source, &actual)
	if err != nil {
		panic(err)
	}
 
	if actual != expected {
		t.Errorf("got %v;expected %v", actual, expected)
	}
}

Supplement: go elastic search: the official go language client of elastic

explain

Elastic officials encourage you to try this package in the project, but please remember the following points:

The work of this project is still in progress. Not all planned functions and the standards in the official elasticsearch client (failure retry, automatic node discovery, etc.) have been implemented.

API stability cannot be guaranteed. Although public APIs are carefully designed, they can be changed in an incompatible manner based on further exploration and user feedback.

The target of the client is elasticsearch version 7. X. Support for 6. X and 5. X APIs will be added later.

install

Install this package with go get:


go get -u github.com/elastic/go-elasticsearch

Or add this package to the go.mod file:


require github.com/elastic/go-elasticsearch v0.0.0

Or clone the warehouse:


git clone https://github.com/elastic/go-elasticsearch.git \u0026amp;\u0026amp; cd go-elasticsearch

A complete example:


mkdir my-elasticsearch-app \u0026amp;\u0026amp; cd my-elasticsearch-appcat \u0026gt; go.mod \u0026lt;\u0026lt;-END  module my-elasticsearch-app  require github.com/elastic/go-elasticsearch v0.0.0ENDcat \u0026gt; main.go \u0026lt;\u0026lt;-END  package main  import (    \u0026quot;log\u0026quot;    \u0026quot;github.com/elastic/go-elasticsearch\u0026quot;  )  func main() {    es, _ := elasticsearch.NewDefaultClient()    log.Println(es.Info())  }ENDgo run main.go

usage

Elasticsearch package is bound with the other two packages. Esapi is used to call elasticsearch API, and estansport transmits data through HTTP.

Use the elasticsearch. Newdefaultclient() function to create a client with the following default settings:


es, err := elasticsearch.NewDefaultClient()if err != nil {  log.Fatalf(\u0026quot;Error creating the client: %s\u0026quot;, err)}res, err := es.Info()if err != nil {  log.Fatalf(\u0026quot;Error getting response: %s\u0026quot;, err)}log.Println(res)// [200 OK] {//   \u0026quot;name\u0026quot; : \u0026quot;node-1\u0026quot;,//   \u0026quot;cluster_name\u0026quot; : \u0026quot;go-elasticsearch\u0026quot;// ...

Note: when exporting elasticsearch_ When the URL environment variable, it will be used as the cluster endpoint.

Configure the client using the elasticsearch. Newclient() function (for demonstration only):


cfg := elasticsearch.Config{  Addresses: []string{    \u0026quot;http://localhost:9200\u0026quot;,    \u0026quot;http://localhost:9201\u0026quot;,  },  Transport: \u0026amp;http.Transport{    MaxIdleConnsPerHost:   10,    ResponseHeaderTimeout: time.Second,    DialContext:           (\u0026amp;net.Dialer{Timeout: time.Second}).DialContext,    TLSClientConfig: \u0026amp;tls.Config{      MaxVersion:         tls.VersionTLS11,      InsecureSkipVerify: true,    },  },}es, err := elasticsearch.NewClient(cfg)// ...

The following example shows a more complex usage. It gets the elastic search version from the cluster, indexes several documents at the same time, and prints the search results using a lightweight wrapper around the response body.


// $ go run _examples/main.gopackage mainimport (  \u0026quot;context\u0026quot;  \u0026quot;encoding/json\u0026quot;  \u0026quot;log\u0026quot;  \u0026quot;strconv\u0026quot;  \u0026quot;strings\u0026quot;  \u0026quot;sync\u0026quot;  \u0026quot;github.com/elastic/go-elasticsearch\u0026quot;  \u0026quot;github.com/elastic/go-elasticsearch/esapi\u0026quot;)func main() {  log.SetFlags(0)  var (    r  map[string]interface{}    wg sync.WaitGroup  )  // Initialize a client with the default settings.  //  // An `ELASTICSEARCH_URL` environment variable will be used when exported.  //  es, err := elasticsearch.NewDefaultClient()  if err != nil {    log.Fatalf(\u0026quot;Error creating the client: %s\u0026quot;, err)  }  // 1. Get cluster info  //  res, err := es.Info()  if err != nil {    log.Fatalf(\u0026quot;Error getting response: %s\u0026quot;, err)  }  // Deserialize the response into a map.  if err := json.NewDecoder(res.Body).Decode(\u0026amp;r); err != nil {    log.Fatalf(\u0026quot;Error parsing the response body: %s\u0026quot;, err)  }  // Print version number.  log.Printf(\u0026quot;~~~~~~~\u0026gt; Elasticsearch %s\u0026quot;, r[\u0026quot;version\u0026quot;].(map[string]interface{})[\u0026quot;number\u0026quot;])  // 2. Index documents concurrently  //  for i, title := range []string{\u0026quot;Test One\u0026quot;, \u0026quot;Test Two\u0026quot;} {    wg.Add(1)    go func(i int, title string) {      defer wg.Done()      // Set up the request object directly.      req := esapi.IndexRequest{        Index:      \u0026quot;test\u0026quot;,        DocumentID: strconv.Itoa(i + 1),        Body:       strings.NewReader(`{\u0026quot;title\u0026quot; : \u0026quot;` + title + `\u0026quot;}`),        Refresh:    \u0026quot;true\u0026quot;,      }      // Perform the request with the client.      res, err := req.Do(context.Background(), es)      if err != nil {        log.Fatalf(\u0026quot;Error getting response: %s\u0026quot;, err)      }      defer res.Body.Close()      if res.IsError() {        log.Printf(\u0026quot;[%s] Error indexing document ID=%d\u0026quot;, res.Status(), i+1)      } else {        // Deserialize the response into a map.        var r map[string]interface{}        if err := json.NewDecoder(res.Body).Decode(\u0026amp;r); err != nil {          log.Printf(\u0026quot;Error parsing the response body: %s\u0026quot;, err)        } else {          // Print the response status and indexed document version.          log.Printf(\u0026quot;[%s] %s; version=%d\u0026quot;, res.Status(), r[\u0026quot;result\u0026quot;], int(r[\u0026quot;_version\u0026quot;].(float64)))        }      }    }(i, title)  }  wg.Wait()  log.Println(strings.Repeat(\u0026quot;-\u0026quot;, 37))  // 3. Search for the indexed documents  //  // Use the helper methods of the client.  res, err = es.Search(    es.Search.WithContext(context.Background()),    es.Search.WithIndex(\u0026quot;test\u0026quot;),    es.Search.WithBody(strings.NewReader(`{\u0026quot;query\u0026quot; : { \u0026quot;match\u0026quot; : { \u0026quot;title\u0026quot; : \u0026quot;test\u0026quot; } }}`)),    es.Search.WithTrackTotalHits(true),    es.Search.WithPretty(),  )  if err != nil {    log.Fatalf(\u0026quot;ERROR: %s\u0026quot;, err)  }  defer res.Body.Close()  if res.IsError() {    var e map[string]interface{}    if err := json.NewDecoder(res.Body).Decode(\u0026amp;e); err != nil {      log.Fatalf(\u0026quot;error parsing the response body: %s\u0026quot;, err)    } else {      // Print the response status and error information.      log.Fatalf(\u0026quot;[%s] %s: %s\u0026quot;,        res.Status(),        e[\u0026quot;error\u0026quot;].(map[string]interface{})[\u0026quot;type\u0026quot;],        e[\u0026quot;error\u0026quot;].(map[string]interface{})[\u0026quot;reason\u0026quot;],      )    }  }  if err := json.NewDecoder(res.Body).Decode(\u0026amp;r); err != nil {    log.Fatalf(\u0026quot;Error parsing the response body: %s\u0026quot;, err)  }  // Print the response status, number of results, and request duration.  log.Printf(    \u0026quot;[%s] %d hits; took: %dms\u0026quot;,    res.Status(),    int(r[\u0026quot;hits\u0026quot;].(map[string]interface{})[\u0026quot;total\u0026quot;].(map[string]interface{})[\u0026quot;value\u0026quot;].(float64)),    int(r[\u0026quot;took\u0026quot;].(float64)),  )  // Print the ID and document source for each hit.  for _, hit := range r[\u0026quot;hits\u0026quot;].(map[string]interface{})[\u0026quot;hits\u0026quot;].([]interface{}) {    log.Printf(\u0026quot; * ID=%s, %s\u0026quot;, hit.(map[string]interface{})[\u0026quot;_id\u0026quot;], hit.(map[string]interface{})[\u0026quot;_source\u0026quot;])  }  log.Println(strings.Repeat(\u0026quot;=\u0026quot;, 37))}// ~~~~~~~\u0026gt; Elasticsearch 7.0.0-SNAPSHOT// [200 OK] updated; version=1// [200 OK] updated; version=1// -------------------------------------// [200 OK] 2 hits; took: 7ms//  * ID=1, map[title:Test One]//  * ID=2, map[title:Test Two]// =====================================

As shown in the above example, the esapi package allows the elasticsearch API to be called in two different ways: by creating a structure (such as indexrequest) and passing the context and client to it to call its do() method, or by directly calling the search() function on it through a function available on the client (such as withindex()). See the package documentation for more information.

The estransport package handles data transfer between and elasticsearch. Currently, this implementation takes up only a small space: it loops only on the configured cluster endpoints. More functions will be added later: retry failed requests, ignore some status codes, automatically discover nodes in the cluster, and so on.

Examples

_ The examples folder contains many comprehensive examples to help you use the client, including client configuration and customization, simulating the transmission of unit tests, embedding the client into custom types, building queries, executing requests and parsing responses.

licence

Follow Apache license version 2.0.

The above is my personal experience. I hope I can give you a reference, and I hope you can support developpaer. If you have any mistakes or don’t consider completely, please don’t hesitate to comment.