47. Elastic search (search engine) uses Django to realize the automatic completion function of search

Time:2021-5-6
Baidu cloud search, search all kinds of information: http://www.lqkweb.com
Search the Internet disk for all kinds of information: http://www.swpan.cn

47. Elastic search (search engine) uses Django to realize the automatic completion function of search

Elasticsearch (search engine) provides automatic completion interface

Official note:https://www.elastic.co/guide/…

1. Create the search auto complete field suggest

Automatic completion requires a field with the name of suggestion and the type of completion

So we need to use the previous elasticsearch DSL operation elasticsearch (search engine) to add the suggestion type to completion

Note: due to the source code problem of elasticsearch DSL, we need to rewrite the customanalyzer class

Only the completion type is used. Other types are not used. Other types can directly specify the word participator

#!/usr/bin/env python

from datetime import datetime
from elasticsearch_dsl import DocType, Date, Nested, Boolean, \
    analyzer, InnerObjectWrapper, Completion, Keyword, Text, Integer

#   For more field types, see section 364 mapping management of elastic search
from   elasticsearch_ dsl.analysis   import   CustomAnalyzer   as  _ CustomAnalyzer     # Import customanalyzer class

from   elasticsearch_ dsl.connections   import   connections                        #  Import the method of connecting to the elasticsearch server
connections.create_connection(hosts=['127.0.0.1'])

class   CustomAnalyzer(_ CustomAnalyzer):                                       #  Customize the customanalyzer class to override the customanalyzer class
    def get_analysis_definition(self):
        return {}

ik_ analyzer  =  CustomAnalyzer("ik_ max_ word",   filter=["lowercase"])            #  Instantiate the overridden customanalyzer class, pass in word segmentation and case conversion, and convert upper case to lower case

class   lagouType(DocType):                                                    #  Customize a class to inherit the DOCTYPE class
    suggest = Completion(analyzer=ik_analyzer)
    #   Text type needs word segmentation, so we need to know Chinese word segmentation, IK_ max_ Wordwai is a Chinese word segmentation device
    title  =  Text(analyzer="ik_ max_ word")                                     #  Set, field name = field type, text is string type, and word segmentation can be used to establish inverted index
    description = Text(analyzer="ik_max_word")
    keywords = Text(analyzer="ik_max_word")
    url  =  Keyword()                                                          #  Set, field name = field type, keyword is ordinary string type, no word segmentation
    riqi  =  Date()                                                            #  Set, field name = field type, date date type

    class   Meta:                                                              #  Meta is a fixed writing method
        index  = " lagou"                                                      #  Set index name (equivalent to database name)
        doc_ type  = ' biao'                                                    #  Set table name

if  __ name__  == "__ main__":           #  Judge in this code file to execute the method inside, other page calls will not execute the method inside
    lagouType.init()                 #  Generate the index, table, field and other information of elastic search (search engine)

#   Instructions for use:
#   On the page where you want to operate elastic search, import this module
#   lagou  =  lagouType()            # Instantiation class
#   lagou.title  = ' 'value'             # To write a field = value
#   lagou.description  = ' 'value'
#   lagou.keywords  = ' 'value'
#   lagou.url  = ' 'value'
#   lagou.riqi  = ' 'value'
#   lagou.save()                   # Write data to elastic search (search engine)

2. Search auto complete field suggest write data

Search the word segmentation data of the field to be searched received by the automatic completion field suggest. See the user-defined word segmentation function below for details

Elasticsearch DSL operation elasticsearch (search engine)

#!/usr/bin/env python
# -*- coding:utf8 -*-
#!/usr/bin/env python

from datetime import datetime
from elasticsearch_dsl import DocType, Date, Nested, Boolean, \
    analyzer, InnerObjectWrapper, Completion, Keyword, Text, Integer
from   elasticsearch_ dsl.connections   import   connections                        #  Import the method of connecting to the elasticsearch server
#   For more field types, see section 364 mapping management of elastic search
from   elasticsearch_ dsl.analysis   import   CustomAnalyzer   as  _ CustomAnalyzer     # Import customanalyzer class

connections.create_connection(hosts=['127.0.0.1'])

class   CustomAnalyzer(_ CustomAnalyzer):                                       #  Customize the customanalyzer class to override the customanalyzer class
    def get_analysis_definition(self):
        return {}

ik_ analyzer  =  CustomAnalyzer("ik_ max_ word",   filter=["lowercase"])            #  Instantiate the overridden customanalyzer class, pass in word segmentation and case conversion, and convert upper case to lower case

class   lagouType(DocType):                                                    #  Customize a class to inherit the DOCTYPE class
    suggest = Completion(analyzer=ik_analyzer)
    #   Text type needs word segmentation, so we need to know Chinese word segmentation, IK_ max_ Wordwai is a Chinese word segmentation device
    title  =  Text(analyzer="ik_ max_ word")                                     #  Set, field name = field type, text is string type, and word segmentation can be used to establish inverted index
    description = Text(analyzer="ik_max_word")
    keywords = Text(analyzer="ik_max_word")
    url  =  Keyword()                                                          #  Set, field name = field type, keyword is ordinary string type, no word segmentation
    riqi  =  Date()                                                            #  Set, field name = field type, date date type

    class   Meta:                                                              #  Meta is a fixed writing method
        index  = " lagou"                                                      #  Set index name (equivalent to database name)
        doc_ type  = ' biao'                                                    #  Set table name

def gen_suggest(index, info_tuple):
    #   Generate search suggestion array based on string
    """
    This function is mainly used to connect elasticsearch (search engine) and use IK_ max_ Word word segmentation, the incoming string segmentation, return the results after segmentation
    This function requires two arguments:
    The first parameter: index to call elasticsearch (search engine) segmentation, generally (index operation class)_ doc_ type.index)
    The second parameter: it is a tuple, and the element of Yuanzu is also a tuple. There are two values in the element Yuanzu. One is the string to be segmented, and the second is the weight of the segmentation. Multiple participles are passed to multiple Yuanzu as follows
    Writing format:
    gen_ suggest(lagouType._ doc_ type.index,  ((' String ',   10) , ('string ',   8)))
    """
    es  =  connections.create_ connection(lagouType._ doc_ type.using)        #  Connect elasticsearch (search engine) and use the_ doc_ Type. Using connection
    used_words = set()
    suggests = []
    for text, weight in info_tuple:
        if text:
            #   Call the analyze interface of ES to analyze the string,
            words = es.indices.analyze(index=index, analyzer="ik_max_word", params={'filter':["lowercase"]}, body=text)
            anylyzed_words = set([r["token"] for r in words["tokens"] if len(r["token"])>1])
            new_words = anylyzed_words - used_words
        else:
            new_words = set()

        if new_words:
            suggests.append({"input":list(new_words), "weight":weight})

    #   Return to the list after word segmentation, which is the dictionary,
    #   For example: [{input ':  [' Recording ',  ' Advertising '],  ' weight':   10},  {' input':  [' New energy ',  ' Cars',],  ' weight':   8}]
    return suggests

if  __ name__  == "__ main__":           #  Judge in this code file to execute the method inside, other page calls will not execute the method inside
    lagouType.init()                 #  Generate the index, table, field and other information of elastic search (search engine)
#   Instructions for use:
#   On the page where you want to operate elastic search, import this module
#   lagou  =  lagouType()            # Instantiation class
#   lagou.title  = ' 'value'             # To write a field = value
#   lagou.description  = ' 'value'
#   lagou.keywords  = ' 'value'
#   lagou.url  = ' 'value'
#   lagou.riqi  = ' 'value'
#   lagou.save()                   # Write data to elastic search (search engine)

The suggest field writes data

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
#   Items. Py, the file is specially used to receive the data information obtained by the crawler, which is equivalent to a container file

import scrapy
from scrapy.loader.processors import MapCompose, TakeFirst
from   scrapy.loader   import   ItemLoader                             #  Importing the itemloader class loads the items container class to fill in the data
from   adc.models.elasticsearch_ orm   import   lagouType,   gen_ suggest   #  Import elasticsearch operation module

class   LagouItemLoader(ItemLoader):                   #  The custom loader inherits the itemloader class, which is called to fill the data into the item class on the crawler page
    default_ output_ processor  =  TakeFirst()           #  By default, the itemloader class is used to load the items container class to fill in the data. It is a list type, and the contents of the list can be obtained through the takefirst() method

def   tianjia(value):                                  #  Custom data preprocessing function
    return   value                                     #  Return the processed data to the item

class   LagouItem(scrapy.Item):                        #  Set the information container class obtained by crawler
    title  =  scrapy.Field(                            #  Receive the title information obtained by the crawler
        input_ processor=MapCompose(tianjia),         #  Pass the name of data preprocessing function into mapcompose method for processing, and the formal parameter value of data preprocessing function will automatically receive the field title
    )
    description = scrapy.Field()
    keywords = scrapy.Field()
    url = scrapy.Field()
    riqi = scrapy.Field()

    def save_to_es(self):
        lagou  =  lagouType()                          #  Instantiate elasticsearch (search engine object)
        lagou.title  =  self['title']                  #  Field name = value
        lagou.description = self['description']
        lagou.keywords = self['keywords']
        lagou.url = self['url']
        lagou.riqi = self['riqi']
        #   Pass the data of title and keywords into the word segmentation function, combine the words, and return to write the search suggestion field suggestion
        lagou.suggest = gen_suggest(lagouType._doc_type.index, ((lagou.title, 10),(lagou.keywords, 8)))
        lagou.save()                                 #  Write data to elasticsearch (search engine object)
        return

The situation after writing elasticsearch (search engine)

{
    "_index": "lagou",
    "_type": "biao",
    "_id": "AV5MDu0NXJs9MkF5tFxW",
    "_version": 1,
    "_score": 1,
    "_source": {
        "title":  " Advertising recording of LED photocatalytic mosquito killing lamp_ Advertising recording network - red advertising recording_ Peddle recording Download_ Voice advertising production ",
        "keywords":  " All kinds of small commodities, advertising recording, Hawking recording, red advertising recording,
        "url": "http://www.luyin.org/post/2486.html",
        "suggest": [
            {
                "input": [
                    "Advertising"
                    ,
                    "Fiery red"
                    ,
                    "Making"
                    ,
                    "Hawking"
                    ,
                    "Mosquito lamp"
                    ,
                    "Voice"
                    ,
                    "Download"
                    ,
                    "led"
                    ,
                    "Recording"
                    ,
                    "Mosquito control"
                    ,
                    "Photocatalysis"
                    ,
                    "Catalysis"
                ],
                "weight": 10
            }
            ,
            {
                "input": [
                    "Small commodities"
                    ,
                    "Advertising"
                    ,
                    "All kinds"
                    ,
                    "Fiery red"
                    ,
                    "Hawking"
                    ,
                    "Commodity"
                    ,
                    "Small business"
                    ,
                    "Recording"
                ],
                "weight": 8
            }
        ],
        "riqi": "2017-09-04T16:43:20",
        "description":  " Advertising recording of LED photocatalytic mosquito killing lamp   It's an article about advertising recording   All kinds of small commodities   Welcome to read and comment, professional Hawking recording - advertising recording - Voice advertising production“
    }
}

 47. Elastic search (search engine) uses Django to realize the automatic completion function of search

Using Django to realize the function of automatic completion of search

1. Bind an event to the search box, trigger the event with each input word, get the content of the input box, and request the input words to Django’s logic processing function with Ajax.

2. In the logic processing function, use the fuzzy fuzzy query of elastic search (search engine) to query the data of the request word in the suggest field, and add the data to the automatic completion

HTML code:

<!DOCTYPE html >
<html xmlns="http://www.w3.org/1999/xhtml">
{import static file path}
{% load staticfiles %}
<head>
<meta http-equiv="X-UA-Compatible" content="IE=emulateIE7" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>lcv-search   Search engine < / Title >
<link href="{% static 'css/style.css'%}" rel="stylesheet" type="text/css" />
<link href="{% static 'css/index.css'%}" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="container">
    <div id="bd">
        <div id="main">
            <h1 class="title">
                <div class="logo large"></div>
            </h1>
            <div class="nav ue-clear">
                <ul class="searchList">
                    <li   class="searchItem   current"   Data type = "article" > Article</li>
                    <li   class="searchItem"   Data type = "question" > Q & A</li>
                    <li   class="searchItem"   Data type = "job" > position</li>
                </ul>
            </div>
            <div class="inputArea">
                {% csrf_token %}
                <input type="text" class="searchInput" />
                <input type="button" class="searchButton" onclick="add_search()" />
                <ul class="dataList">
                    <li>How to learn design well</li>
                    <li>Interface design</li>
                    <li>How much is UI design training</li>
                    <li>Designer learning</li>
                    <li>Where is a good website</li>
                </ul>
            </div>

            <div class="historyArea">
                <p class="history">
                    < label > popular search: < / label >

                </p>
                <p class="history mysearch">
                    < label > My Search: < / label >
                    <span class="all-search">
                        <a   href="javascript:;"> Focus on interface design website</a>
                        <a   href="javascript:;"> User experience</a>
                        <a   href="javascript:;"> Internet</a>
                        <a   href="javascript:;"> Tariff package</a>
                    </span>

                </p>
            </div>
        </div><!-- End of main -->
    </div><!--End of bd-->

    <div class="foot">
        <div class="wrap">
            <div   class="copyright">Copyright  & copy; uimaker.com   copyright    E-mail:[email protected] </div>
        </div>
    </div>
</div>
</body>
<script type="text/javascript" src="{% static 'js/jquery.js'%}"></script>
<script type="text/javascript" src="{% static 'js/global.js'%}"></script>
<script type="text/javascript">
    var suggest_url = "/suggest/"
    var search_url = "/search/"

    $('.searchList').on('click', '.searchItem', function(){
        $('.searchList .searchItem').removeClass('current');
        $(this).addClass('current');
    });

    function removeByValue(arr, val) {
      for(var i=0; i<arr.length; i++) {
        if(arr[i] == val) {
          arr.splice(i, 1);
          break;
        }
      }
    }

    //   Search for suggestions
    $(function(){
        $('.searchInput').bind(' input propertychange ',function(){
            var searchText = $(this).val();
            var tmpHtml = ""
            $.ajax({
                cache: false,
                type: 'get',
                dataType:'json',
                url:suggest_url+"?s="+searchText+"&s_type="+$(".searchItem.current").attr('data-type'),
                async: true,
                success: function(data) {
                    for (var i=0;i<data.length;i++){
                        tmpHtml += '<li><a href="'+search_url+'?q='+data[i]+'">'+data[i]+'</a></li>'
                    }
                    $(".dataList").html("")
                    $(".dataList").append(tmpHtml);
                    if (data.length == 0){
                        $('.dataList').hide()
                    }else {
                        $('.dataList').show()
                    }
                }
            });
        } );
    })

    hideElement($('.dataList'), $('.searchInput'));

</script>
<script>
    var searchArr;
    //Define a search to determine whether the browser has data storage (Search History)
    if(localStorage.search){
    //If so, convert to   The form of array is stored in the array of searcharr (localstorage is stored in the form of string, so it should be converted to the form of array)
        searchArr= localStorage.search.split(",")
    }else{
    //If not, define searcharr as an empty array
        searchArr = [];
    }
    //Display the stored data as search history
    MapSearchArr();

    function add_search(){
        var val = $(".searchInput").val();
        if (val.length>=2){
            //When you click the search button, remove the duplicate
            KillRepeat(val);
            //After de duplication, the array is stored in the browser localstorage
            localStorage.search = searchArr;
            //Then display the search content
            MapSearchArr();
        }

        window.location.href=search_url+'?q='+val+"&s_type="+$(".searchItem.current").attr('data-type')

    }

    function MapSearchArr(){
        var tmpHtml = "";
        var arrLen = 0
        if (searchArr.length >= 5){
            arrLen = 5
        }else {
            arrLen = searchArr.length
        }
        for (var i=0;i<arrLen;i++){
            tmpHtml += '<a href="'+search_url+'?q='+searchArr[i]+'">'+searchArr[i]+'</a>'
        }
        $(".mysearch .all-search").html(tmpHtml);
    }
    //De duplication
    function KillRepeat(val){
        var kill = 0;
        for (var i=0;i<searchArr.length;i++){
            if(val===searchArr[i]){
                kill ++;
            }
        }
        if(kill<1){
            searchArr.unshift(val);
        }else {
            removeByValue(searchArr, val)
            searchArr.unshift(val)
        }
    }

</script>
</html>

Django routing mapping

"""pachong URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/1.10/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  url(r'^/pre>, views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  url(r'^/pre>, Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.conf.urls import url, include
    2. Add a URL to urlpatterns:  url(r'^blog/', include('blog.urls'))
"""
from django.conf.urls import url
from django.contrib import admin
from app1 import views

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^/pre>, views.indexluoji),
    url(r'^index/', views.indexluoji),
    url(r'^suggest//pre>,   views.suggestluoji,name="suggest"),      #  Search field completion request

]

Django static file configuration

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.10/howto/static-files/
#Configure static file prefix
STATIC_URL = '/static/'
#Configure static file directory
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'static')
]

Note: search automatically completes fuzzy query

#Search automatically complete fuzzy query
POST lagou/biao/_search?pretty
{
  "Suggest": {# field name
    "my_ Suggest ": {# custom variable
      "Text": "advertisement", "search term"
      "completion":{
        "Field": "suggest", # search field
        "fuzzy":{
          "Fuzziness": 1 # edit distance
        }
      }
    }
  },
  "_source":"title"
}

Django logic processing file

from django.shortcuts import render

# Create your views here.
from django.shortcuts import render,HttpResponse
from django.views.generic.base import View
from   app1.models   import   lagouType    # Import operation elasticsearch (search engine) class
import json

def indexluoji(request):
    print(request.method)   #  Get the path of user request
    return render(request, 'index.html')

def   suggestluoji(request):                                       #  Search automatic completion logic processing
    key_ words  =  request.GET.get('s',  '')                        #  Get request word
    re_datas = []
    if key_words:
        s  =  lagouType.search()                                   #  Instantiate the search query of elasticsearch class
        s = s.suggest('my_suggest', key_words, completion={
            "field": "suggest", "fuzzy": {
                "fuzziness": 2
            },
            "size": 5
        })
        suggestions = s.execute_suggest()
        for match in suggestions.my_suggest[0].options:
            source = match._source
            re_datas.append(source["title"])
    return HttpResponse(json.dumps(re_datas), content_type="application/json")

47. Elastic search (search engine) uses Django to realize the automatic completion function of search

Final completion

47. Elastic search (search engine) uses Django to realize the automatic completion function of search

Recommended Today

Detailed steps for installing Perl and Komodo IDE for windows

Perl official website: https://www.perl.org/Perl document: https://perldoc.perl.org/Download address: https://www.perl.org/get.html The installation package of Perl Windows version is divided into activestate Perl and strawberry Perl. For the difference between the two, see: http://www.zzvips.com/article/202134.htm Note: the download speed of activestate Perl is slow. You may need KX to surf the Internet I have uploaded all the versions of […]