Solr cache best practices for easy tuning

Time:2020-3-25

Background

Apache Solr is a widely used open-source search engine. Greenplum text, the full-text search component of Greenplum dB, is based on it: Greenplum text is abbreviated to gptext, which combines Greenplum database with Apache Solrcloud enterprise search and madlib analysis library are closely integrated to provide customers with large-scale analysis processing and business decision support. The main functions include free text search and support for text analysis.

Generally speaking, the cache in Solr can be divided into two parts: the internal cache of Solr system and the file cache left to the operating system. For the former, if it is set improperly, not only the performance will not be improved, but also a lot of memory will be wasted.

Based on the long-term experience of tuning for customers, the author will briefly introduce Solr cache and state its best practices.

Two, classification

Solr internal cache is divided into four types:

  • Queryresult cache: cache the docid zipper corresponding to query, data structure: query = > docid list
  • Filter cache: the bitmap index corresponding to the cache filter. Data structure: filter = > bitmap (total doc number bit)
  • Doc cache: cache the stored doc document content corresponding to doc ID, data structure: Doc id = > DOC content
  • Custom cache, custom cache, closed by default, will not be introduced in this document.

What you need to notice is that the data structure given here is just plain implementation, and the actual internal implementation of Solr is more sophisticated than plain implementation (such as saving memory), especially for filter cache, see Solr community discussion for details.

Three, configuration

Configure in solrconfig.xml of each index, and configure the maximum number of indexes of each type (size), for example:

<filterCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />

<queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />

<documentCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />

For configuration, you need to pay attention to two points:

  • The first two can also set the maxrammb parameter (version requirements: it can’t be configured in Solr 6.6, and it can be configured in 7.3.1): the upper memory limit of this type of cache, in MB. If configured, the size and initialsize parameters are ignored
  • In addition, the effective time of cache is affected by the auto / autosoft commit parameter. Currently, the latest gptext code is configured as follows. Autocommit will not cause cache failure (because opensearcher is set to false)

<autoCommit>

<maxDocs>10000</maxDocs>

<maxTime>60000</maxTime>

<openSearcher>false</openSearcher>
</autoCommit>

Cache invalidation caused by autosoftcommit: cache invalidation will be caused when every docs / time cycle arrives

<autoSoftCommit>

<maxDocs>100000</maxDocs>

<maxTime>600000</maxTime>

</autoSoftCommit>

4、 Memory overhead

Here we use an example to illustrate. Suppose a user’s usage scenario is as follows:

  • 10 bytes per query / filter
  • Average number of results: 250000 (each result ID is represented by an 8 byte integer)
  • Total number of documents: 1 billion (using a bitmap, 1 bit for each document)
  • Average stored document size: 32 bytes (4 8byte fields)
  • The cache and autocommit parameters are both default values

The maximum size of each type of cache can be obtained:

  • queryresult cache:(10+2500008) 512 = 1GB (worst case)
  • Filter cache: (1000000000 / 8) * 512 = 60GB (worst case)
  • doc cache:32 * 512 = 16KB

Obviously, the filter cache is too large, but the above is only a qualitative analysis to find out the possible memory bottleneck, and the actual memory overhead will be smaller.

Five. Monitoring
If you can access the Solr UI, you can view the cache metrics statistics on each core:

http://localhost:18983/solr/#/demo.public.test_wiki_shard1_replica_n2/plugins?type=cache

If the Solr UI cannot be accessed, view it directly through the following URL:

  • http://localhost:18983/solr/admin/metrics?group=core&prefix=CACHE.searcher.queryResultCache
  • http://localhost:18983/solr/admin/metrics?group=core&prefix=CACHE.searcher.filterCache
  • http://localhost:18983/solr/admin/metrics?group=core&prefix=CACHE.searcher.documentCache

For example, viewing metrics projects through the Solr UI

Solr cache best practices for easy tuning

Focus on the need for xxx.cumulative_hitratio: it is the cumulative hit rate after the Solr node is started, and will not be affected by the cache emptying.

6、 Configuration practice

This article will give a general setting method for each type of cache size:

First, run the system continuously for a period of time (such as a day / a week, etc.), and then collect the following system parameters: 1. Whether oom (out of memory) has occurred; 2. Collect the cumulative ﹣ hitratio (usually only the largest index needs to be focused on)

If no oom occurs, follow the flowchart below:

Solr cache best practices for easy tuning

If oom occurs, we should turn off or limit the size of a certain type of cache:

1.filter cache

  • As shown in the previous example, it is generally the cache type with the highest demand for space, and it is likely that oom is caused by it
  • Check cumulative_hitratioa. If it is high (indicating that cache is very effective), set an acceptable maxrammb value for it. B. if it is low, close it directly (set size to 0)

2. query result cache

  • Usually it should not be turned off (often useful, such as repeatedly searching for a query) value
  • If high, set an acceptable maxrammb for it. If low, set a smaller maxrammb value

3. document cache

Generally, it is not a memory bottleneck, so please handle the first two types of cache first

Seven, reference

  • https://teaspoon-consulting.c…
  • Scaling Lucene and Solr:https://lucidworks.com/post/s…
  • https://blog.cloudera.com/apa…
  • https://blog.cloudera.com/sol…
  • https://lucene.apache.org/sol…
  • https://blog.csdn.net/john_ho…