In the recent work, there is a scenario in which elasticsearch (hereinafter referred to as ES) is written. Flink processes the data and writes it to es in real time. Now you need to load a batch of historical data into es through Flink. There are two points to ensure:
- For historical data, if the ES has documents, the old data will be discarded, and if the ES has no documents, the historical data will be inserted.
- For new data, the existing es data can be updated.
reference resourcesElasticsearch advanced level (I) — version controlYou can use the ES version to develop this requirement.
2、 Code implementation and verification
Add version and version when requesting to write data_ Type parameter. The main codes are as follows:
IndexRequest indexRequest = Requests.indexRequest() .index(indexName) .id("1") //Specify the business fields for version comparison. The specific business is analyzed. Generally, the timestamp is appropriate .version(Long.parseLong(dataMap1.get("create").toString())) //Specify to use external version number .versionType(VersionType.EXTERNAL) .source(dataMap);
The verification demo can use the timestamp of the current time as the basis for version comparison. The verification idea is as follows:
- Run the demo program, insert a piece of data under the current timestamp, and verify whether the data is successfully inserted through kibana and other tools. And record the current timestamp.
- Change some field values to update the data. Run the program again to verify whether the data is updated successfully.
- Fix the field value of time version comparison as the timestamp of the first program execution, and verify whether the data is updated successfully.
The verification results are as follows:
It can be seen from the screenshot that both step 1 and step 2 can be executed successfully, and the exception of version conflict will appear in step 3. It is easy to identify the cause according to the prompt, that isElasticsearch advanced level (I) — version controlUsing version and version_ Type=external for version control, only the version number of the document to be written is greater than the version number of the existing document can be updated successfully.
Case code reference:elasticsearch_demo