Feature: read_parquet_mergetree

[akvlad]

read_parquet_mergetree

Description

The read_parquet_mergetree chsql function provides a familiar interface for ClickHouse users by emulating aspects of the MergeTree engine strategy. Its primary purpose is to efficiently merge multiple parquet files using a specified primary SORT key - without consuming excessive memory and facilitating fast range queries on the resulting file.

TLDR; A memory efficient parquet file merge/compact feature with sorting capabilities.

Syntax

COPY (SELECT * FROM read_parquet_mergetree([PARQUET_FILE_ARRAY], {PRIMARY_SORT_KEY} )) TO `{MERGED.PARQUET}`

Features

• Merge data from multiple files, similar to how ClickHouse combines data from different parts
• Use a specified sort key to order data, analogous to the primary key in ClickHouse MergeTree tables
• Maintain sorted order within the merged dataset, facilitating fast range queries
• Support glob patterns and wildcards in file array

Parameters

• FILE_ARRAY[]: An array of file paths to merge
• PRIMARY_SORT_KEY: Specifies the column(s) used as the primary sort key for merging and ordering data

Benchmark

COPY (SELECT * FROM read_parquet(['/folder/*.parquet']) ORDER BY some_key) TO 'sorted.parquet'
// USAGE: ~64GB RAM

COPY (SELECT * FROM read_parquet_mergetree(['/folder/*.parquet'], 'some_key') TO 'sorted.parquet'
// USAGE: ~800MB RAM

[lmangani]

Hey @carlopi any chance you or someone in the team knows how to get around the windows build error? 🙏

[carlopi]

Can you try to reduce the diff?

Or try to copy the setup of extensions like duckdb_delta.

[akvlad]

@carlopi Is it enough if I tell you that the real change is only in the file https://github.com/lmangani/duckdb-extension-clickhouse-sql/pull/13/files#diff-c5bffd6b887e2ced50224f44652dab784c9c7f7ab8c46a390410cc58490391ed ?

The other changes are just internal insignificant file moves.

Or do you need a separate PR with the function implementation?

[carlopi]

Then it's likely either a #pragma once is needed in chsql.hpp or maybe Chsql::Name & co can stay in the main header with the main extension mechanics, and the rest of the function registration should be moved to a secondary header.

[akvlad]

@carlopi Aaah . It's about the windows build problem.

From the MSVC++ linker logs I see that somehow the linker wants to link chsql_extension.obj file more than once:

chsql_extension.lib(chsql_extension.obj) : error LNK2005: "public: virtual void __cdecl duckdb::ChsqlExtension::Load(class duckdb::DuckDB &)" (?Load@ChsqlExtension@duckdb@@UEAAXAEAVDuckDB@2@@Z) already defined in chsql_extension.obj [D:\a\duckdb-extension-clickhouse-sql\duckdb-extension-clickhouse-sql\build\release\extension\chsql\chsql_loadable_extension.vcxproj]

chsql_extension.lib(chsql_extension.obj )error LNK2005:  .... already defined in chsql_extension.obj

Have no idea why it wants to link the same .obj twice. Have you encountered the similar problem anywhere?

[lmangani]

screenshot of @akvlad kicking the windows builder where it hurts 😄

[lmangani]

amazing work @akvlad lets merge and proceed with some field testing 🎉