Improve documentation for format `OPTIONS` clause #15708

marvelshan · 2025-04-14T12:58:48Z

Which issue does this PR close?

Closes #.Document CREATE EXTERNAL TABLE ... OPTIONS #10451

Rationale for this change

This PR adds documentation for the OPTIONS clause, including generic options and format-specific options, to ensure users have clear guidance on available settings.

What changes are included in this PR?

Renamed write_options.md to format_options.md to reflect its scope for both reading and writing.
Added examples for each format (JSON, CSV, Parquet) in the OPTIONS clause.
Documented generic options and format-specific options in a structured format.
Updated heading levels to ensure all sections appear in the table of contents.

Are these changes tested?

Do not require automated tests

Are there any user-facing changes?

Yes, the documentation now includes detailed examples and descriptions of the OPTIONS clause for CREATE EXTERNAL TABLE and COPY queries.

…e for both reading and writing

alamb · 2025-04-14T20:09:26Z

Run extended tests

alamb · 2025-04-14T20:10:21Z

Run extended tests

FYI, I am using this PR as a way to test ci: fix workflow triggering extended tests from pr comments. #15704 from @ashdnazg. It is not anything related to this PR

alamb

Thank you so much @marvelshan -- this is a nice improvement

I left a few comments -- let me know what you think. It would be great to put the NULL option into the csv options and fix the examples so they work in this PR

alamb · 2025-04-15T11:07:03Z

docs/source/user-guide/sql/format_options.md

+
+# Format Options
+
+DataFusion supports customizing how data is read from or written to disk as a result of a `COPY`, `INSERT INTO`, or `CREATE EXTERNAL TABLE` query. There are a few special options, file format (e.g., CSV or Parquet) specific options, and Parquet column-specific options. Options can also in some cases be specified in multiple ways with a set order of precedence.


Suggested change

DataFusion supports customizing how data is read from or written to disk as a result of a `COPY`, `INSERT INTO`, or `CREATE EXTERNAL TABLE` query. There are a few special options, file format (e.g., CSV or Parquet) specific options, and Parquet column-specific options. Options can also in some cases be specified in multiple ways with a set order of precedence.

DataFusion supports customizing how data is read from or written to disk as a result of a `COPY`, `INSERT INTO`, or `CREATE EXTERNAL TABLE` statements. There are a few special options, file format (e.g., CSV or Parquet) specific options, and Parquet column-specific options. In some cases, Options can be specified in multiple ways with a set order of precedence.

alamb · 2025-04-15T11:15:35Z

docs/source/user-guide/sql/format_options.md

+Format-related options can be specified in the following ways:
+
+- Session-level config defaults
+- `CREATE EXTERNAL TABLE` options
+- `COPY` option tuples
+


I think it would be helpful to explicit specify the order of precedence here. Something like

Suggested change

Format-related options can be specified in the following ways:

- Session-level config defaults

- `CREATE EXTERNAL TABLE` options

- `COPY` option tuples

Format-related options can be specified in three ways, in decreasing order of precedence:

- `CREATE EXTERNAL TABLE` syntax

- `COPY` option tuples

- Session-level config defaults

alamb · 2025-04-15T11:19:40Z

docs/source/user-guide/sql/format_options.md

+
+| Option     | Description                                                   | Default Value    |
+| ---------- | ------------------------------------------------------------- | ---------------- |
+| NULL_VALUE | Sets the string which should be used to indicate null values. | arrow-rs default |


I think this is a CSV specific option (not a generic option)

For example

> create external table my_table(a int) stored as JSON location '/tmp/foo' options('NULL_VALUE' 'NULL'); Invalid or Unsupported Configuration: Config value "null_value" not found on JsonOptions

docs/source/user-guide/sql/format_options.md

marvelshan · 2025-04-15T15:07:56Z

Thank you for the suggestions!
I realized there were several places I hadn’t tested thoroughly. After some testing, I’ve made updates and pushed a new version.
I also wanted to check with you: since there are quite a few available options on file config.rs, should we just list a few key ones in the documentation, or aim to include all of them? I’m wondering if listing everything might make the docs overly complex.

alamb

I also wanted to check with you: since there are quite a few available options on file config.rs, should we just list a few key ones in the documentation, or aim to include all of them? I’m wondering if listing everything might make the docs overly complex.

I think we should include all of the options in the documentation as a reference. While I agree there are a lot of options, I think it would be best for the documentation to reflects that (complex) reality

alamb · 2025-04-16T20:07:55Z

docs/source/user-guide/sql/write_options.md

-| DICTIONARY_PAGE_SIZE_LIMIT   | No                      | Sets best effort maximum dictionary page size in bytes                                                                              |
-| CREATED_BY                   | No                      | Sets the "created by" property in the parquet file                                                                                  |
-| COLUMN_INDEX_TRUNCATE_LENGTH | No                      | Sets the max length of min/max value fields in the column index.                                                                    |
-| DATA_PAGE_ROW_COUNT_LIMIT    | No                      | Sets best effort maximum number of rows in a data page.                                                                             |


We seem to have lost some of these options in the new doc

…clause

alamb

Thank you so much for working on this @marvelshan -- I think it is looking really nice now

alamb · 2025-04-17T17:16:34Z

docs/source/user-guide/sql/format_options.md

+OPTIONS('DELIMITER' '|', 'HAS_HEADER' 'true', 'NEWLINES_IN_VALUES' 'true');
+```
+
+## Parquet Format Options


👍 this looks great

alamb · 2025-04-17T17:16:52Z

docs/source/user-guide/sql/format_options.md

+
+The following options are available when executing a `COPY` query.
+
+| Option Key                                                  | Description                                                                                                  | Default Value |


This largely replicates what is already in this page: https://datafusion.apache.org/user-guide/configs.html

I think the idea is that the format_options page will describe options specific to formats -- I will push a small commit to fix

Thank you very much for your review! I truly appreciate you taking the time to make the final adjustments. I’ve learned a lot of valuable details from this process. If there’s anything else that needs further revision, please don’t hesitate to let me know—I’d be more than happy to make the necessary changes.

Thank you for helping push this along @marvelshan -- it takes non trivial effort to make these docs

* doc:Add documentation for OPTIONS clause syntax * doc:rename write_options.md to format_options.md and clarify its scope for both reading and writing * doc: change dml.md, cuz still have wrong write_options filename * doc: update doctest reference to renamed format_options.md * docs: update and correct format options documentation * doc: add more information of options content * remove execution settings, move note about insert * wordsmith example --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

marvelshan added 2 commits April 13, 2025 23:40

doc:Add documentation for OPTIONS clause syntax

3a7e441

doc:rename write_options.md to format_options.md and clarify its scop…

e47787d

…e for both reading and writing

github-actions bot added the documentation Improvements or additions to documentation label Apr 14, 2025

marvelshan added 2 commits April 14, 2025 22:15

doc: change dml.md, cuz still have wrong write_options filename

c3635db

doc: update doctest reference to renamed format_options.md

237023c

github-actions bot added the core Core DataFusion crate label Apr 14, 2025

alamb mentioned this pull request Apr 14, 2025

ci: fix workflow triggering extended tests from pr comments. #15704

Merged

alamb reviewed Apr 15, 2025

View reviewed changes

docs: update and correct format options documentation

197e14f

alamb changed the title ~~doc/document options clause~~ Improve documentation for format OPTIONS clause Apr 16, 2025

alamb reviewed Apr 16, 2025

View reviewed changes

marvelshan and others added 4 commits April 17, 2025 18:49

doc: add more information of options content

74b6c73

Merge remote-tracking branch 'apache/main' into doc/document-options-…

d582e93

…clause

remove execution settings, move note about insert

4818dae

wordsmith example

03c3b95

alamb approved these changes Apr 17, 2025

View reviewed changes

alamb merged commit f07fb10 into apache:main Apr 19, 2025
29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve documentation for format `OPTIONS` clause #15708

Improve documentation for format `OPTIONS` clause #15708

marvelshan commented Apr 14, 2025

alamb commented Apr 14, 2025

alamb commented Apr 14, 2025 •

edited

Loading

alamb left a comment

alamb Apr 15, 2025

alamb Apr 15, 2025

alamb Apr 15, 2025

marvelshan commented Apr 15, 2025

alamb left a comment

alamb Apr 16, 2025

alamb left a comment

alamb Apr 17, 2025

alamb Apr 17, 2025

marvelshan Apr 18, 2025

alamb Apr 19, 2025


		# Format Options

		DataFusion supports customizing how data is read from or written to disk as a result of a `COPY`, `INSERT INTO`, or `CREATE EXTERNAL TABLE` query. There are a few special options, file format (e.g., CSV or Parquet) specific options, and Parquet column-specific options. Options can also in some cases be specified in multiple ways with a set order of precedence.


		The following options are available when executing a `COPY` query.

		\| Option Key \| Description \| Default Value \|

Improve documentation for format OPTIONS clause #15708

Improve documentation for format OPTIONS clause #15708

Conversation

marvelshan commented Apr 14, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb commented Apr 14, 2025

alamb commented Apr 14, 2025 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marvelshan commented Apr 15, 2025

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Improve documentation for format `OPTIONS` clause #15708

Improve documentation for format `OPTIONS` clause #15708

alamb commented Apr 14, 2025 •

edited

Loading