[Epic] Better / Improved Documentation, Tutorials and Examples

[alamb]

This is a follow on to #3058 as we have made significant progress since @kmitchener originally posted that

“Write cool software and tell people about it” – Paul Dix @pauldix (Founder and CTO of InfluxData)

Call to action:

The DataFusion community has invested a lot in the cool software; This ticket tracks tasks to improve the "tell people about it"

User Guide

• Getting started guide for new users (who want to use DataFusion in their project)  #7014
• Automate production of SQL and DataFrame references for SQL functions / Expressions #3092
• Write a case study about using DataFusion #7015
• Document SQL operators (like @> and ~) #7153
• Document streaming usecase (like UNBOUNDED tables) #9016
• Make it easier to install / get started with DataFusion #7297
• Document DML statements #7319
• Hide boilerplate in documentation examples #14557
• Add a "Gentle Introduction to Arrow / Record Batches" #11336
• Improvements to user guide for DataFrames #11388
• Run / Test all examples in Documentation #14435

Rust Docs (docs.rs)

• Update SQL functions rustdocs with example usage
• Automate production of SQL and DataFrame references for SQL functions / Expressions #3092
• Include the latest rustdocs (e.g. cargo doc output on https://arrow.apache.org/datafusion/) #5981

Examples

• Review current examples and see how they can be improved
• Include examples reading from object stores (S3, Azure, GCS)
• Async User Defined Functions (UDF) #6518
• Add example for writing a FileFormat #11079
• Add a section to the documentation explaining that PGO can help up substantially (25%) and maybe offer some tips for users to use it? #9561
• Add example for writing an AnalyzerRule #10855
• Add example for writing an SQL analysis pass #10871
• [EPIC] Improve examples to make them easier to navigate #11172
• Provide an example of using a remote catalog #13714

Conference Papers

• Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782

Blog posts

• Write a blog post fast Vectorized grouping for high cardinality #6988
• Blog post with DataFusion Jun - Sep 2023 #6780
• Blog post about user defined window functions #6781

Other

• [Proposal] Ensure consistency between code and documentation. #9173
• Write documentation explaining how to enable metrics #1578
• Add documentation for support for skipping Parquet row groups #825
• Add content from design docs to datafusion site #1927
• Document "how to read an explain plan" #12088

[MrPowers]

@alamb - can you add a "devrel" tag to these issues? That would help me track them. Thanks!

[alamb]

@alamb - can you add a "devrel" tag to these issues? That would help me track them. Thanks!

Done @MrPowers : https://github.com/apache/arrow-datafusion/labels/devrel

[felipevalerio]

Hi, I would like to be responsible for this task "Review current examples and see how they can be improved"

But I have a question. Those examples are the ones in : https://github.com/apache/datafusion/tree/main/datafusion-examples

[alamb]

But I have a question. Those examples are the ones in : https://github.com/apache/datafusion/tree/main/datafusion-examples

Yes indeed.

One simple win I think would be to consolidate the examples of specific functions

• https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/to_char.rs
• https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/to_date.rs
• https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/to_timestamp.rs

Perhaps into https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/date_time_functions.rs or something to make it clearer they belonged togehter

Thank you!