Elasticsearch advanced use – dynamic template

Time:2022-11-24

total progress

Elasticsearch 8.1 Certified Engineer Learning Path

today we learnElasticsearchIn fact, we have already learned similar dynamic templates in chapter 2.2.3 of the first lesson, the link is as follows

Create indexes based on given requirements

But today let’s learn something different. Last time we just used it briefly. This time we need to understand it deeply. Perfect control is the first goal. Let’s start now without further ado.

What is a dynamic template

Dynamic templates allow us to control data outside of dynamic field mapping rules

The mapping of dynamic fields we can setdynamicThe parameter istrueorruntimeTo enable dynamic mapping, you can also customize the dynamic mapping template to customize the mapping. The customized template can be applied to dynamically added fields according to the matching rules. In the previous section, the dynamic mapping rules have already been mentioned, and this section is directly moved over, as follows

JSON data type "dynamic":"true" "dynamic":"runtime"
null no added no added
true or false boolean boolean
double float double
long long long
object object no added
array Judging by the first non-empty value in the array Judging by the first non-empty value in the array
string of date type date date
numeric string float or long double or long
A string that is neither a date nor a number texttype and.keywordword type keyword

Among them, the custom dynamic template matching rules are as follows

  • match_mapping_typerightElasticsearchOperate on the data type detected in , refer to the chart above
  • matchandunmatchField names can be matched using patterns
  • path_matchandpath_unmatchOperates on the full dotted path of the field
  • If the dynamic template is not definedmarch_mapping_typematchorpath_match, then the template will not match any of the fields. but_bulk The template can be referenced by the template name when requesting

can also use{name}and{dynamic_type}Template variables are used as placeholders. For example, placeholders are used later to specify tokenizers, etc.

The definition of a dynamic template is an array of objects

"dynamic_templates":[
  {
    "my_template_name":{#1
      ... matching condition...#2
      "mapping":{...}#3
    }
  },
  ...
]
  1. Custom template name, which can be any string
  2. The use matching conditions of the template can be:match_mapping_type,match,match_pattern,unmatch,path_match,path_unmatch
  3. The index map that the match field should use

Through the above study, we know the definition of the dynamic template. Now that it is defined, it should be verified. After all, whether the defined template can be used and whether the definition is correct still needs to be verified.

Validate dynamic templates

An error is returned if the defined template contains invalid mapping fragments. existindexThe dynamic template is used for verification during operation, but in most cases it will be verified when the dynamic template is updated. Providing invalid mapping fragments may cause dynamic template validation to fail during updates or under certain conditions, such as:

  • if not specifiedmatch_mapping_type, but this template provides at least one valid mapping, then this mapping fragment is valid. However, if the field matching the template is mapped to another type, a validation error will be returned during indexing. For example: configure a dynamic template that does not containmatch_mapping_type, which is a valid string type, but if there is a field matching the dynamic template it is matched aslong, then inindexwill return a validation error. The suggestion is tomatch_mapping_typeconfigured as expectedJSONtype (refer to the mapping relationship table at the beginning) or inmappingConfigure the required type in
  • if we are inmappingused in the fragment of{name}placeholder, then the verification is skipped when the dynamic template is updated. This is because the field name at that time is still unknown, so inindexVerify when

If there are multiple templates matching at the same time, they will be processed according to the sequential matching rules, and the first matching template has the highest priority; when passedupdate mapping APIWhen updating dynamic templates, all existing templates will be overwritten, allowing dynamic templates to be reordered or deleted after the initial creation

Map runtime fields in dynamic templates

In the previous section, we used this as an example of the content of this section. If you are interested, you can take a look back and put the link at the beginning of the article.

if we wantElasticsearchTo dynamically map a certain type of new field to a runtime field, then we can set"dynamic":"runtime", these fields are not indexed and are queried from_sourceload

Or we use the default dynamic mapping rules, then create a dynamic template, and map specific fields as runtime fields. we need to be inindex mappingset in"dynamic:"true",Then create a dynamic template and map a new field of a certain type as a runtime field

As an example, suppose we have a set of data where each field is_ipAt the beginning, according to the dynamic mapping rules,ElasticsearchAny strings detected by the value are mapped tofloatorlong, at this point we can create a dynamic template that maps this new string toipruntime field of type

The following is an example of ours, which roughly means that whenElasticsearchUse match pattern isip*, it maps the fields asiptype of runtime field. Because the fields are not dynamically mapped, we can use"dynamic":"true"or"dynamic":"runtime"In conjunction with

PUT my-dynamic-template-001/
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_ip": {
          "match_mapping_type": "string",
          "match": "ip*",
          "runtime": {
            "type": "ip"
          }
        }
      }
    ]
  }
}

combine"dynamic":"true"or"dynamic":"runtime"use

PUT my-dynamic-template-001/
{
  "mappings": {
    "dynamic":"runtime",
    "dynamic_templates": [
      {
        "strings_as_ip": {
          "match_mapping_type": "string",
          "match": "ip*",
          "runtime": {
            "type": "ip"
          }
        }
      }
    ]
  }
}

In the above statement, we will match the matching patternip*The new field of is mapped as a runtime field, but since we set the"dynamic":"runtime", so we will set the following new fields as runtime fields, which is the following statement, whereip_req,ip_res, conforming to the dynamic templatedynamic_templatesmatching rules forip*,andmy_ipset with index beginning"dynamic":"runtime"Also added to the runtime field

PUT my-dynamic-template-001/_doc/1
{
  "ip_req":"127.0.0.1",
  "ip_res":"127.0.0.1",
  "my_ip":"locahost"
}

At this point we look at the index as follows

{
  "my-dynamic-template-001" : {
    "mappings" : {
      "dynamic" : "runtime",
      "dynamic_templates" : [
        {
          "strings_as_ip" : {
            "match" : "ip*",
            "match_mapping_type" : "string",
            "runtime" : {
              "type" : "ip"
            }
          }
        }
      ],
      "runtime" : {
        "ip_req" : {
          "type" : "ip"
        },
        "ip_res" : {
          "type" : "ip"
        },
        "my_ip" : {
          "type" : "keyword"
        }
      }
    }
  }
}

The above is a simple use, where

  • match_mapping_typeYesstring, that is, the value of the field is a string
  • matchYesip*That is, the field namedipat the beginning
  • runtimeDefines the field type to be mapped. In the above example, it is mapped asruntime, of typeip

match_mapping_type

match_mapping_typeYesJSONThe data type detected by the parser. becauseJSONDoes not distinguish betweenlongandinteger, nor distinguishdoubleandfloat, so when parsingdoubleandfloatwould be considered to bedoubleintegerandlongwould be considered to belong

NOTE: When using dynamic mapping, Elasticsearch will always choose the wider data type, with the exception offloattype, which requires less storage space thandouble, and is accurate enough for most applications. But the runtime field does not supportfloattype, so this is"dynamic":"runtime"usedoubles reason

Elasticsearch will automatically detect the data type, the detection rule is the content of the table at the beginning of the article, and we can also usematch_mapping_typeuse wildcards in*to match all data types

For example, if we want to map an integer field asintegerinstead oflongtype, string fields match astextandkeywordtype, we can use the following template

  • create a template
PUT my-dynamic-template-002
{
  "mappings": {
    "dynamic_templates": [
      {
        "integers": {
          "match_mapping_type": "long",
          "mapping": {
            "type": "integer"
          }
        }
      },
      {
        "strings": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "text",
            "fields": {
              "raw": {
                "type":  "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    ]
  }
}


  • Insert a piece of test data

    PUT my-dynamic-template-002/_doc/1
    {
      "my_integer": 5, 
      "my_string": "Some string" 
    }
  • view generatedmapping

    GET my-dynamic-template-002/_mapping

    The returned results are as follows,my_integerwill be mapped asintegermy_stringwill be mapped astextandkeyword

    {
      "my-dynamic-template-002" : {
        "mappings" : {
          "dynamic_templates" : [
            {
              "integers" : {
                "match_mapping_type" : "long",
                "mapping" : {
                  "type" : "integer"
                }
              }
            },
            {
              "strings" : {
                "match_mapping_type" : "string",
                "mapping" : {
                  "fields" : {
                    "raw" : {
                      "ignore_above" : 256,
                      "type" : "keyword"
                    }
                  },
                  "type" : "text"
                }
              }
            }
          ],
          "properties" : {
            "my_integer" : {
              "type" : "integer"
            },
            "my_string" : {
              "type" : "text",
              "fields" : {
                "raw" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
    

match and unmatch

matchUse pattern matching on field names,unmatchUse a pattern to exclude matching fields

match_patternCan be adjusted by setting this parameter valuematchparameter behavior so thatmatchParameters support fullJavaRegular expressions to replace simple wildcards, as follows

  "match_pattern": "regex",
  "match": "^profit_\d+$"

In the example below, we match names tolong_All string fields beginning with_textString fields ending in , and map them aslongtype field


PUT my-dynamic-template-003
{
  "mappings": {
    "dynamic_templates": [
      {
        "longs_as_strings": {
          "match_mapping_type": "string",
          "match":   "long_*",
          "unmatch": "*_text",
          "mapping": {
            "type": "long"
          }
        }
      }
    ]
  }
}

PUT my-dynamic-template-003/_doc/1
{
  "long_num": "5", 
  "long_text": "foo" 
}

GET my-dynamic-template-003/_mapping

In the example above,long_numis mapped tolongfield of type, becausematch_mapping_typeforstringtype, and islong_at the beginning.long_textAlthough it islong_at the beginning, but also_textending, in line withunmatchconditions, suchlong_textjust follow the default rulesstringto map, generatetextandkeywordfield

path_match and path_unmatch

path_matchandpath_unmatchandmatchandunmatchworks similarly, but matches the full dotted path of the field instead of just the final name, e.g.some_object.*.some_field

PUT my-dynamic-template-004
{
  "mappings": {
    "dynamic_templates": [
      {
        "full_name": {
          "path_match":   "name.*",
          "path_unmatch": "*.middle",
          "mapping": {
            "type":       "text",
            "copy_to":    "full_name"
          }
        }
      }
    ]
  }
}

PUT my-dynamic-template-004/_doc/1
{
  "name": {
    "first":  "John",
    "middle": "Winston",
    "last":   "Lennon"
  }
}

As above, anynameAny field under themiddleExcept for the fields at the end, they will be mapped astexttype, andcopy_towithfull_name

Execute the following command to view

GET my-dynamic-template-004/_mapping

The display results are as follows

{
  "my-dynamic-template-004" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "full_name" : {
            "path_match" : "name.*",
            "path_unmatch" : "*.middle",
            "mapping" : {
              "copy_to" : "full_name",
              "type" : "text"
            }
          }
        }
      ],
      "properties" : {
        "full_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "name" : {
          "properties" : {
            "first" : {
              "type" : "text",
              "copy_to" : [
                "full_name"
              ]
            },
            "last" : {
              "type" : "text",
              "copy_to" : [
                "full_name"
              ]
            },
            "middle" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}

From the above results, we can see that,nameundermiddleNocopy_to, which is mapped according to the defaultstringmapping rules, generatetextas well askeyword

It should be noted that, in addition toleafoutside the field,path_matchandpath_unmatchThe parameter matches the object path, and the following will generate an error for the document index, becausename.titlecannot be mapped to a text type

PUT my-dynamic-template-004/_doc/2
{
  "name": {
    "first":  "Paul",
    "last":   "McCartney",
    "title": {
      "value": "Sir",
      "category": "order of chivalry"
    }
  }
}

The error is as follows

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [name.title] of type [text] in document with id '2'. Preview of field's value: '{category=order of chivalry, value=Sir}'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [name.title] of type [text] in document with id '2'. Preview of field's value: '{category=order of chivalry, value=Sir}'",
    "caused_by" : {
      "type" : "illegal_state_exception",
      "reason" : "Can't get text on a START_OBJECT at 5:14"
    }
  },
  "status" : 400
}

template variables (template variables)

{name}and{dynamic_type}Placeholders are replaced in the mapping with field names and detected dynamic types

The following example uses an analyzer with the same name as the field for all string fields, and disables all fields that are not stringsdoc_values


PUT my-dynamic-template-005
{
  "mappings": {
    "dynamic_templates": [
      {
        "named_analyzers": {
          "match_mapping_type": "string",
          "match": "*",
          "mapping": {
            "type": "text",
            "analyzer": "{name}"
          }
        }
      },
      {
        "no_doc_values": {
          "match_mapping_type":"*",
          "mapping": {
            "type": "{dynamic_type}",
            "doc_values": false
          }
        }
      }
    ]
  }
}

PUT my-dynamic-template-005/_doc/1
{
  "english": "Some English text", 
  "count":   5 
}
GET my-dynamic-template-005/_mapping

In the example above,{name} is replaced byfield name,and{dynamic_type}is replaced withJSONThe data types detected by the parser

englishThe field analyzer is set toenglish,andcountdetected aslongtype, anddoc_valuesSet asfalse

Examples of dynamic templates

structured search

when we set"dynamic":"true"Elasticsearch maps string fields astexttype, with akeywordtype of subfield, if we are only searching for structures and do not need full-text retrieval, then we can let Elasticsearch map askeywordfields, but in doing so, the fields must be searched for exactly the same values ​​as the index, i.e. an exact match

PUT my-dynamic-template-006
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_keywords": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

plaintext map of strings

Contrary to the structured search above, if we only care about full-text retrieval, and do not aggregate, sort, and precisely search fields, then we let Elasticsearch map strings astexttype

PUT my-dynamic-template-007
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_text": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "text"
          }
        }
      }
    ]
  }
}

For the runtime fields added in recent versions, we can also create a dynamic template to map strings to runtime fieldskeywordType, although the field will not be indexed, but their values ​​exist_source, and can be used for searching, aggregating, filtering and sorting

For example, the following example will create a dynamic template that willstringFields are mapped tokeywordruntime field, althoughruntimeThe definition is empty, but Elasticsearch will use the matching rules at the beginning of the article to add, any string that fails the time or number detection will be mapped tokeywordtype

PUT my-dynamic-template-008
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_keywords": {
          "match_mapping_type": "string",
          "runtime": {}
        }
      }
    ]
  }
}

At this point we index a document

PUT my-dynamic-template-008/_doc/1
{
  "english": "Some English text",
  "count":   5
}

When viewing the mapping relationship, you can seeenglishis mapped tokeywordruntime field of type

GET my-dynamic-template-008/_mapping
{
  "my-dynamic-template-008" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "strings_as_keywords" : {
            "match_mapping_type" : "string",
            "runtime" : { }
          }
        }
      ],
      "runtime" : {
        "english" : {
          "type" : "keyword"
        }
      },
      "properties" : {
        "count" : {
          "type" : "long"
        }
      }
    }
  }
}

Disable norms

If we are not sorting by rating then we can disable the rating factor in the index to save space

PUT my-dynamic-template-009
{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_keywords": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    ]
  }
}

appearing in the template abovekeywordThe keyword field is consistent with the default rules of dynamic mapping. If we don’t need it, we can delete it according to the above example

sequentially

When using Elasticsearch for time series analysis, there are usually many numeric fields, but these numeric fields are generally not filtered and usually aggregated. In this case we can disable indexing on these fields to save disk space and maybe also gain some indexing speed

PUT my-dynamic-template-010
{
  "mappings": {
    "dynamic_templates": [
      {
        "unindexed_longs": {
          "match_mapping_type": "long",
          "mapping": {
            "type": "long",
            "index": false
          }
        }
      },
      {
        "unindexed_doubles": {
          "match_mapping_type": "double",
          "mapping": {
            "type": "float", 
            "index": false
          }
        }
      }
    ]
  }
}

As with the default dynamic rules,doubleis mapped tofloat, because he can satisfy most requests. and requires only half the disk space

Summarize

Well, this is the end of the knowledge about dynamic templates. From the beginning, we will introduce what dynamic templates are, followed by how to use dynamic templates, and the last common examples of dynamic templates. So how much have you mastered? Go and try it out. Please look forward to a preview “Defining Index Lifecycle Strategy ILM for Time Series Indexes”

This article is written bymdniceMulti-platform publishing