From 001d8b0cddcec46a44e7c6e31612dc2baada05d5 Mon Sep 17 00:00:00 2001 From: Hyukjin Kwon Date: Mon, 6 Dec 2021 12:00:51 +0900 Subject: [PATCH] [SPARK-37554][BUILD] Add PyArrow, pandas and plotly to release Docker image dependencies ### What changes were proposed in this pull request? This PR proposes to add plotly, pyarrow and pandas dependencies for generating the API documentation for pandas API on Spark. The versions of `pandas==1.1.5 pyarrow==3.0.0 plotly==5.4.0` are matched with the current versions being used in branch-3.2 at Python 3.6. ### Why are the changes needed? Currently, the function references for pandas API on Spark are all missing: https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/series.html due to missing dependencies when building the docs. ### Does this PR introduce _any_ user-facing change? Yes, the broken links of documentation at https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/series.html will all be recovered. ### How was this patch tested? To be honest, it has not been tested. I don't have the nerve to run Docker releasing script for the sake of testing so I defer to the next release manager. The combinations of the dependency versions are being tested in GitHub Actions at `branch-3.2`. Closes #34813 from HyukjinKwon/SPARK-37554. Authored-by: Hyukjin Kwon Signed-off-by: Hyukjin Kwon (cherry picked from commit 03750c046b55f60b43646c8108e5f2e540782755) Signed-off-by: Dongjoon Hyun --- dev/create-release/spark-rm/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev/create-release/spark-rm/Dockerfile b/dev/create-release/spark-rm/Dockerfile index ecfb6762d9fc3..83752bd941d25 100644 --- a/dev/create-release/spark-rm/Dockerfile +++ b/dev/create-release/spark-rm/Dockerfile @@ -42,7 +42,7 @@ ARG APT_INSTALL="apt-get install --no-install-recommends -y" # We should use the latest Sphinx version once this is fixed. # TODO(SPARK-35375): Jinja2 3.0.0+ causes error when building with Sphinx. # See also https://issues.apache.org/jira/browse/SPARK-35375. -ARG PIP_PKGS="sphinx==3.0.4 mkdocs==1.1.2 numpy==1.19.4 pydata_sphinx_theme==0.4.1 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==2.11.3 twine==3.4.1 sphinx-plotly-directive==0.1.3" +ARG PIP_PKGS="sphinx==3.0.4 mkdocs==1.1.2 numpy==1.19.4 pydata_sphinx_theme==0.4.1 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==2.11.3 twine==3.4.1 sphinx-plotly-directive==0.1.3 pandas==1.1.5 pyarrow==3.0.0 plotly==5.4.0" ARG GEM_PKGS="bundler:2.2.9" # Install extra needed repos and refresh.