GH-44515: [D] Add Initial Support

[kassane]

Rationale for this change

Add Dlang support. Initial implementation based on c_glib, using gir-to-d generator.

Tools used

• ldc2 (based on LLVM backend + dmd-frontend) toolchain - works in macOS ARM64, too.
• dub (pkg-manager, like cargo [rust])
• gtkd-developers/gir-to-d

What changes are included in this PR?

Add D implementation only. No modify any other language support.

Are these changes tested?

Unit tests will also be added during development.

Are there any user-facing changes?

It adds a new programming language alternative for attracting more users (the small D community).

References

• GIR in Apache Arrow - Error: OutputStream matches conflicting symbols gtkd-developers/gir-to-d#45
  C++
• https://dlang.org/spec/cpp_interface.html
• https://dlang.org/articles/template-comparison.html
• https://dlang.org/articles/cpptod.html
  C
• https://dlang.org/spec/interfaceToC.html
• https://dlang.org/articles/ctod.html

cc: @kou

• GitHub Issue: [D] D programming language Implementation of Arrow #44515

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #44515 has been automatically assigned in GitHub to PR creator.

[kou]

Could you remove auto generated files?
Can we add a shell script that generates files and fix them (by sed or something)?

[kassane]

Could you remove auto generated files?

Ok.

Can we add a shell script that generates files and fix them (by sed or something)?

It's possible, but is it viable?
Because there are so many lines and the fixes vary ( looking at previous commits).

Note that it is possible to use meson to configure the build of D projects (officially supported). However, it may be interesting to make build.d with D (std/phobos2) features to solve.

[kassane]

Now, base build.d done. (try run dub build to test)

• cpp:cmake (config, build): 🆗 - the flags can be revised later!
• c_glib:meson (setup, build): 🆗 - the flags can be revised later!

Right now, I will review the issue of gir-to-d. If needed, it requires an improved fork to fix the D bindings generation.
Note: Still need to fix D's CI later.

[kou]

I'm not familiar with D but why do we need to create a D program just build the D bindings?

Can we simplify this?

• Can we detect already installed Apache Arrow GLib by pkgconf --cflags --libs arrow-glib instead of building Apache Arrow C++ and GLib?
• Can we create a shell script that just runs dub run girtod -- --gir-directory $(pkgconf --variable=girdir arrow-glib) --input $(pkgconf --variable=girdir arrow-glib/Arrow-1.0.gir --output source --user-runtime-linker?

[kassane]

I'm not familiar with D but why do we need to create a D program just build the D bindings?

Like cargo (Rust), dub only get D files (all *.d in rootpath) by default.

build.d will only be applied for building non-D project.
For this use case, it is assumed that the user does not have Arrow installed.

Can we simplify this?

• Can we detect already installed Apache Arrow GLib by pkgconf --cflags --libs arrow-glib instead of building Apache Arrow C++ and GLib?

dub.json: libs = [...] use pkg-config by default on posix.

Also is possible add specialized libraries like: libs-posix|libs-windows

• Can we create a shell script that just runs dub run girtod -- --gir-directory $(pkgconf --variable=girdir arrow-glib) --input $(pkgconf --variable=girdir arrow-glib/Arrow-1.0.gir --output source --user-runtime-linker?

Yeah! dub.json can add multi commands in preBuildCommands. Like dub.json from glibD project.

[kou]

it is assumed that the user does not have Arrow installed.

Can we require users to install Apache Arrow GLib before they install the D bindings?

FYI: We have the APT/Yum repositories for Apache Arrow GLib: https://arrow.apache.org/install/

• Can we create a shell script that just runs dub run girtod -- --gir-directory $(pkgconf --variable=girdir arrow-glib) --input $(pkgconf --variable=girdir arrow-glib/Arrow-1.0.gir --output source --user-runtime-linker?

Yeah! dub.json can add multi commands in preBuildCommands. Like dub.json from glibD project.

Could you try this approach?

[kou]

Could you rename this to linux-apt-d.dockerfile and use ubuntu-c-glib as the base image like ubuntu-ruby does?

arrow/docker-compose.yml

Lines 853 to 875 in 33e8cbb

	ubuntu-ruby:
	# Usage:
	# docker compose build ubuntu-cpp
	# docker compose build ubuntu-c-glib
	# docker compose build ubuntu-ruby
	# docker compose run --rm ubuntu-ruby
	# Parameters:
	# ARCH: amd64, arm64v8, ...
	# UBUNTU: 20.04, 22.04
	image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-ruby
	build:
	context: .
	dockerfile: ci/docker/linux-apt-ruby.dockerfile
	cache_from:
	- ${REPO}:${ARCH}-ubuntu-${UBUNTU}-ruby
	args:
	base: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-c-glib
	shm_size: *shm-size
	ulimits: *ulimits
	environment:
	<<: [*common, *ccache]
	volumes: *ubuntu-volumes
	command: *ruby-command

[kou]

Can we use out-of-source build?

[kassane]

generator tools replaced (gtk4 compat): https://github.com/Kymorphia/gidgen

Can we use out-of-source build?

Considerable. Maybe add packages folder.

[kassane]

What's report: https://github.com/Kymorphia/gidgen#summary

Summary

Package	Structs (Act Dis Uns Perc)	Funcs (Act Dis Uns Perc)	Signals (Act Dis Uns Perc)	Fields (Act Dis Uns Perc)
Arrow	592 0 0 100	978 54 9 99.1	0 0 0 100	589 0 1 99.8
GLib	86 8 0 100	1342 643 6 99.6	0 0 0 100	213 23 35 85.9
GModule	3 0 0 100	8 6 0 100	0 0 0 100	0 0 0 100
GObject	64 2 0 100	306 158 3 99.0	3 0 0 100	231 8 10 95.9
Gdk	80 0 0 100	478 34 5 99.0	33 0 2 94.3	32 0 1 97.0
GdkPixbuf	16 0 0 100	95 12 2 97.9	4 0 0 100	42 4 2 95.5
Gid	3 0 0 100	0 0 0 100	0 0 0 100	0 0 0 100
Gio	374 2 0 100	1729 162 17 99.0	80 0 1 98.8	885 195 32 96.5
Graphene	21 0 0 100	367 43 0 100	0 0 0 100	40 0 0 100
Gsk	57 0 0 100	266 24 4 98.5	0 0 0 100	13 0 2 86.7
Gtk	542 0 0 100	3593 115 10 99.7	342 1 3 99.1	601 95 9 98.5
HarfBuzz	35 0 0 100	407 52 42 90.6	0 0 0 100	71 28 11 86.6
Pango	54 0 0 100	388 44 12 97.0	0 0 0 100	162 11 8 95.3
PangoCairo	4 0 0 100	26 5 0 100	0 0 0 100	0 0 0 100
cairo	20 0 1 95.2	327 56 3 99.1	0 0 0 100	38 0 1 97.4
freetype2	5 0 0 100	0 0 0 100	0 0 0 100	0 0 0 100

full: report.txt

[kassane]

External enhancement for Arrow support:

GiD bindings

• Kymorphia/gid@afd41bd
• Kymorphia/gid@0e4ecfb