Proficient in log query: how to turn pages to get log and calculation results

Time:2019-10-28

Proficient in log query: how to turn pages to get log and calculation results

Log service provides one-stop log collection, storage, query and calculation functions. Interactive log collection experience, release the pressure of user’s operation and maintenance, liberate users’ hands; interactive query and analysis experience, let users freely build data model, exploratory analysis and mining data depth.

Proficient in log query: how to turn pages to get log and calculation results

Users can use the query and analysis ability of log service, not only in the console interactive query, but also in the program using the SDK. When the calculation results are relatively large, how to read the full results to the local is a headache. Fortunately, the log service provides the function of page turning, which can not only read the original log content, but also read the calculation results of SQL locally. Developers can use the SDK or cli provided by the log service to turn pages through the read data interface to read logs.

Query and analysis use different pagination methods

Proficient in log query: how to turn pages to get log and calculation results

The log service provides a unified query log entry: getlogstorelogs. It can query the original log content according to the keyword, or submit SQL calculation to get the calculation result.

Use keyword query to get the original content. You can page through the offset and lines parameters in the API to get all the content.

Use SQL calculation to get the calculation results of SQL, and use SQL limit syntax to turn pages.

Query page turning use cases

In the getlogstorelogs API, there are two parameters: offset and lines.

• offset: used to specify to read the log from the first line

• lines: used to specify how many lines the current request reads. The maximum number of lines for this parameter is 100. If the parameter is set to be greater than 100 lines, 100 lines will still be returned.

During page turning reading, increase the offset continuously until the number of result rows obtained is 0 after reading an offset, and the progress of the result is in the complete state, then it is considered that all data has been read and can be ended.

Page code example

Paged pseudo code:

Proficient in log query: how to turn pages to get log and calculation results

Example of Python page reading

More detailed case references:

Proficient in log query: how to turn pages to get log and calculation results

Java page flipping reading example

More detailed case references

Proficient in log query: how to turn pages to get log and calculation results

Page reading of SQL analysis results

In SQL analysis, the offset and lines in getlogstorelogs API parameters are invalid. Fill in. That is to say, if you follow the way of page turning to read the original content and traverse the offset page turning, the result of each SQL execution is the same. Theoretically, we can get all the calculation results in one call, but if the result set is too large, the following problems may occur:

• the transmission of large amounts of data on the network has a high latency.

• the memory of the client needs to store a large number of results for further processing.

In order to solve the problem of SQL page turning, we provide the limit page turning syntax of standard SQL.

Proficient in log query: how to turn pages to get log and calculation results

Offset indicates which line to read the result from

• line indicates how many lines are read, and line has no size limit; however, if too many lines are read at a time, network delay and client processing will be affected.

• a typical case, if the following SQL generates 2000 logs in total

Proficient in log query: how to turn pages to get log and calculation results

Then you can turn the page, read 500 lines each time, and read 4 times in total:

Proficient in log query: how to turn pages to get log and calculation results

SQL page flipping example

In the program, the pseudo code of SQL page turning is as follows:

Proficient in log query: how to turn pages to get log and calculation results

Python program example:

Proficient in log query: how to turn pages to get log and calculation results

Java program example:

Proficient in log query: how to turn pages to get log and calculation results

Author: Yunlei

Read the original text

This is the original content of yunqi community, which can not be reproduced without permission.