Databend community actively embraces rust Ecology (202111 review)

Time:2021-12-4

Databend aims to become an open-source, flexible and reliable serverless data warehouse, query as fast as lightning, and organically combine with elastic, simple and low-cost cloud services. The construction of data cloud has never been so simple!
Databend community actively embraces rust Ecology (202111 review)

Significance of databend to rust community

Most of the team members of databend have rich experience in database / data platform development and open source collaboration. We choose rust to develop next-generation data cloud services with high performance and high reliability.
Databend community actively embraces rust Ecology (202111 review)

  1. Build the number one seed player in the field of rust warehouse, show the charm of rust and promote the further application of rust in the field of database.
  2. Continue to promote rust’s efforts in data governance / database / cloud native and other fields, and improve ecology and tools.
  3. Provide more learning / training / employment opportunities for the majority of rustaceans, and cultivate a number of database development experts for the community.

The close relationship between databend and rust ecology

From the first day of the establishment of databend community, we have held the belief of “coming from the community and going to the community”, and actively embraced open source and rust ecology.

curriculum

Since August, databend, together with rust Chinese community and zhishutang, has launched an open class program for rustacean and database developers.
Databend community actively embraces rust Ecology (202111 review)

Rust training and improvement program

upper reaches

The success of databend is inseparable from the prosperity of rust ecology and high-quality upstream projects. Databend community will also provide patches to the upstream from time to time to strive to build a better rust ecology.
Databend community actively embraces rust Ecology (202111 review)

  • Advisory DB – Security Advisory database for rust crite published in crites.io
  • Add flatbuffers patch version for rustsec-2020-0009
  • Arrow RS & arrow flight & parquet – the official rust implementation of Apache arrow / parquet, which is an important dependency of databend in the early stage

    • Add the limit option for sort
    • Affected by rustsec-2020-0009, update flatbuffers to the latest version
    • Prettytable RS is not maintained and replaced with comfy table
    • Update Prost to 0.8 and the corresponding tonic update
    • Update Prost to 0.9 and the corresponding tonic update to fix the error
  • Arrow2 & arrow format & parquet2 – the implementation of arrow and the fastest and safest parquet rewritten with the transform free operation

    • Add the limit option for merge sort
    • Improved sort
    • Add wrapping_ Cast improves cast performance
    • Add computing support for binaryarray
    • Prettytable RS is not maintained and replaced with comfy table
    • Update Prost to 0.8 and the corresponding tonic update
    • Update Prost to 0.9 and the corresponding tonic update to fix error and clip lints
  • Async raft – Implementation of asynchronous raft consensus algorithm based on Tokio

    • Fix: when restarting, nonvoter should maintain its identity instead of becoming a follower
    • A series of additional fixes are listed inhttps://github.com/async-raft…
    • Because the maintenance status of the original async raft library is not good, especially the maintenance of Datafuse extras / async raft
  • Cargo license – utility for viewing dependencies on adopted license agreements

    • Update dependencies to avoid conflicts and fix clip lints
  • Clickhouse SRV – used to write bindings that can act as Clickhouse servers

    • The project was developed and maintained by members of the databend team.
    • It has been merged into the databend project, which is located in common / Clickhouse SRV
  • Float CMP – floating point approximation comparison

    • Specify the minimum version of num traits and fix clip lints to help upstream dependencies pass minver test
  • Hashbrown – high performance HashMap, which is also part of the standard library

    • Update Rand to 0.8.3
  • LRU RS – LRU cache implementation

    • Update hashbrown to 0.11.2 and fix clip lints
  • Mockall – a useful library for writing tests

    • Update dependencies affected by rustsec-2020-0095
  • Msql SRV – used to write bindings that can act as MySQL / MariaDB servers

    • Repair com_ FIELD_ List corresponding
    • Synchronize dependencies to the latest version and fix dependencies in the test
    • Add basic authplugin support
    • Query progress information statistics
    • Some patches are trying to be merged upstream. At present, they can be tried in Datafuse extras / msql SRV
  • Pprof – CPU / application performance testing tool

    • Update Prost to 0.8 and the corresponding tonic update
    • Update Prost to 0.9 and the corresponding tonic update to repair clip lints
  • Ritelinked – Fast & easy to use implementation of LinkedHashMap & linkedhashset

    • The project was developed and maintained by members of the databend team.
    • Currently located in ritelabs / ritelinked
  • Sqlparser RS – extensible SQL lexical analyzer and parser

    • Replace panic with error returned
    • Add a fuzzy testing tool based on Hongg fuzzy
    • Support tinyint
    • Datafuse extras / sqlparser RS currently includes some databend customization features

      • Support insert format
      • Support cast (a, b)
      • Make the function support arguments
      • Substring (string, start, length) syntax is supported
  • Structopt toml – toml default value loader for structopt

    • Failure is not maintained, replace with anyhow

Databend’s community building plan

Databend community is a platform jointly built by every follower, developer and user. In the future, we will continue to improve the feedback mechanism to help each community member grow and progress in the process of community construction.
Databend community actively embraces rust Ecology (202111 review)

  1. Establish and improve the developer growth system to provide rustacean with a platform for learning, practice and innovation
  2. Continue to output high-quality rust + database courses, invite industry experts to share development practical experience and publicize high-quality projects in databend ecology
  3. Provide sponsorship, training and other opportunities for important upstream and downstream projects and high-quality developers to encourage their contribution to open source and ecology

The name databend comes from the theory of relativity. Due to the existence of matter, time and space will bend, which is the time bend of relativity. We expect that the emergence of databend can enable users to re-examine the data, so as to mine the greater value in the data, and let users in the field of big data: make life easier.

Welcome to join the databend community and work with us to build the next generation of data cloud services with rust.

Recommended Today

Hive built-in function summary

1. Related help operation functions View built-in functions: Show functions; Display function details: desc function ABS; Display function extension information: desc function extended concat; 2. Learn the ultimate mental method of built-in function Step 1: carefully read all the functions of the show functions command to establish an overall understanding and impression Step 2: use […]