Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

use net.Buffers to reduce memory inuse bytes.. #421

Closed

Conversation

zdyj3170101136
Copy link
Contributor

Summary

use net.Buffers to reduce memory inuse bytes

Detail

In a typical clickhouse insert scenario, in order to improve the insert efficiency, the client usually inserts in batches, inserting every 10w:

1, convert []struct to proto.Input
2, use ch.Client.Do to convert proto.Input to proto.Block
3, send proto.Block to clickhouse server.

The third step usually takes a long time, so in order to reduce memory usage, you can set []struct to nil after the first step to release []struct.

Since proto.Block is []byte, the proto.Input data will actually be copied to proto.Block, which results in double memory overhead.

Now set proto.Block to net.Buffers, which allows a series of memory areas to be written to the network connection at once, reducing the memory overhead by half.

Only Support *Col.Str for now.

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

@ernado
Copy link
Collaborator

ernado commented Sep 18, 2024

BTW there is also WIP PR #413 that aims to reduce memory footprint

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants