My little sister teaches you to customize a logstash Java filter~

LogstashIt is a processing engine used to collect data, analyze and process data, and finally output data to storage components. The data processing flow is as follows:

Logstash Java FilternamelyFilter extension API based on logstashDevelop aFilter implemented by java language, and thenFilter code is packaged to build logstash filter lib on your own serverIn the middle. You can be in theIn the data flow configuration file (that is, the configuration file specified by logstash - F)Use this custom logstash Java filter.

The customization steps include the following five steps:

1. Prepare logstash environment

Because logstash Java filter needs to rely on logstash API, we need to download and build logstash source code

1.1. Download logstash source code

git clone --branch <branch_name> --single-branch https://github.com/elastic/logstash.git <target_folder>

among<branch_name>You need to replace it with the version of logstash you want to use, just use the GA version after 7.1. <target_ Folder > should be replaced with the parent directory of the logstash code you want to download. If not specified, download it to the logstash folder of the current directory. I’m using version 7.6:

git clone --branch 7.6  --single-branch https://github.com/elastic/logstash.git 

1.2. Build logstash source code

Enter the logstash directory of the current directory (that is, the logstash source directory, which is later called:$LS_HOME）Next, execute

./gradlew assemble

If it is a Windows systemgradlew.bat assemble

This step will take a long time. If you can’t download it, try adding a domestic image of gradle.
vim $LS_HOME/build.gradle, and then add it to the file

    repositories {
           maven { url 'https://maven.aliyun.com/repository/google/' }
           maven { url 'https://maven.aliyun.com/repository/jcenter/'}
           mavenCentral()
   
           maven {
               url 'https://plugins.gradle.org/m2/'
           }
      }

After the build is successful, check the$LS_HOME/logstash-core/build/libs/Generate in directorylogstash-core-x.y.z.jar. Where x, y and Z are the version numbers of logstash you downloaded. What I want is

/Users/xx/corprepo/logstash/logstash-core/build/libs/logstash-core-7.6.3.jar

2. Write logstash Java filter code

2.1. Download the official demo

The official provides a demo, which we can download and modify based on.

2.2. Designated logstash_ CORE_ PATH

After downloading the demo, create it in the project root directorygradle.propertiesFile, add a line of data:

LOGSTASH_CORE_PATH=<target_folder>/logstash-core

2.3. Develop filter code

We need to inherit logstash’s filter API to implement our own Java filter function. A well implemented filter is as follows:

import co.elastic.logstash.api.Configuration;
import co.elastic.logstash.api.Context;
import co.elastic.logstash.api.Event;
import co.elastic.logstash.api.Filter;
import co.elastic.logstash.api.FilterMatchListener;
import co.elastic.logstash.api.LogstashPlugin;
import co.elastic.logstash.api.PluginConfigSpec;
import org.apache.commons.lang3.StringUtils;

import java.util.Collection;
import java.util.Collections;
//The class name must match the underline annotation name according to the hump name, javafilterexample > java_ filter_ example
@LogstashPlugin(name = "java_filter_example")
public class JavaFilterExample implements Filter {
    //Define a setting configuration supported by the filter. The name is source and the default value is message
    //It can be seen from the filter method that source is used_ The value of config is used as the name of field
    public static final PluginConfigSpec<String> SOURCE_CONFIG =
            PluginConfigSpec.stringSetting("source", "message");

    private String id;
    private String sourceField;

    public JavaFilterExample(String id, Configuration config, Context context) {
        // constructors should validate configuration options
        this.id = id;
        this.sourceField = config.get(SOURCE_CONFIG);
    }

    /**
     *The filtering logic of the filter can perform various CRUD operations on the input event data
     * @param events
     * @param matchListener
     *@ return finally flows to the data of the next pipeline. If there are events that meet the conditions, they must be returned
     */
    @Override
    public Collection<Event> filter(Collection<Event> events, FilterMatchListener matchListener) {
        for (Event e : events) {
            Object f = e.getField(sourceField);
            if (f instanceof String) {
                e.setField(sourceField, StringUtils.reverse((String)f));
                matchListener.filterMatched(e);
            }
        }
        return events;
    }
    /**
     *
     *@ return returns all the settings supported by the filter
     */
    @Override
    public Collection<PluginConfigSpec<?>> configSchema() {
        // should return a list of all configuration options for this plugin
        return Collections.singletonList(SOURCE_CONFIG);
    }

    /**
     *
     *@ return the ID of the filter. Logstash will assign it to us
     */
    @Override
    public String getId() {
        return this.id;
    }
}

Two points should be paid attention to

• @LogstashPluginAnnotatednameMust be highly consistent with the class name. Such as Java_ filter_ Example – > javafilterexample
• Need to achieveco.elastic.logstash.api.FilterClass, if you don’t import successfully, it isgradle.propertiesFailed to configure or build logstash source code. There are three ways to rewrite it

Getid method

Return the ID of the filter. Logstash will assign the value for us. We just need to define oneMember variable，In the construction methodJust fine.

Configschema method

Returns the set of all settings supported by the filter.PluginConfigSpecDefinedSetting configurationThis is the parameter that can be passed when we use the filter in the logstash configuration file, such as the parameter passed in when we use the grok filterpatterns_dirandmatch。

filter {
      grok {
        patterns_dir => ["./patterns"]
        match => { "message" => "%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}" }
      }
    }

This setting configurationPluginConfigSpecThe supported configuration parameters areName, type, precondition status, required status, and default value。

In our filter class, we define ` pluginconfigspec < string > source_ CONFIG =

PluginConfigSpec.stringSetting ("source", "message"); ` Where 'name = source, default value = message'`

Filter method

Filters, of course, are about filtering logic. One of them is the referenceCollection<Event> eventsIt’s the input data we need to process. We can do some curd operations for the logic. Enter the referenceFilterMatchListener matchListenerFor example, the implementation of matchlistener in logstash isDecoratingFilterMatchListener. What it can do, for exampleADD_FIELD
Similarly, we need to define pluginconfigspec first, and then configure it when using the filteradd_fieldParameters. For example, grok filter supports this parameter and the decoratingfiltermatchlistener

 filter {
      grok {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }

You don’t need to call the matchlistener without informing the matchlistener of its needsmatchListener.filterMatched(e)It’s too late.

3. Unit test

There are also test classes in demo, and run is over..

4. Package and deploy filter

We need to use gradle to transform our filter project into Ruby gem package, so it’s better to modify it based on the gradle configuration file in the demo project.

4.1. Configure gradle packaging task
Edit project root pathbuild.gradlefile

plugin infoPart of it is the information of our filter. I have used the special points that need to be modifiedTODOIt’s marked.
4.2. Run gradle package task
Execute in the project root directory

./gradlew gem

Windows system executiongradlew.bat gem

After the execution is successful, you will see that it is generated in the root directory of the projectlogstash-{plugintype}-<pluginName>-<version>.gemfile

4.3. Install filter gem package in logstash
Go to logstash directory ($LS)_ Home)

bin/logstash-plugin install --no-verify --local /path/to/javaPlugin.gem

among/path/to/javaPlugin.gemThis is the gem absolute path generated in step 4.2.

5. Use our Java filter to run logstash

5.1. At $LS_ Create logstash running configuration file in home / config directoryjava_filter.conf

input {
  generator { message => "Hello world!" count => 1 }
}
filter {
# java_ filter_ Example: the name of the @ logstashplugin annotation in our filter
  java_filter_example {}
}
output {
  stdout { codec => rubydebug }
}

5.2. Start logstash
At $LS_ Home running

bin/logstash -f  config/java_filter.conf

So far, it’s a success

{
       "message" => "!dlrow olleH",
      "sequence" => 0,
      "@version" => "1",
          "host" => "xxdeMacBook-Pro.local",
    "@timestamp" => 2020-04-12T13:15:30.376Z
}

Thank you for reading. I’m monica23334 | monica2333. Little sister who writes an original article flag every week, pay attention to me and look forward to slapping face

Refer to official documents: https://www.elastic.co/guide/en/logstash/7.6/java-filter-plugin.html