Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fix typo in spark-tests.sh [skip ci] #10328

Merged
merged 1 commit into from
Jan 30, 2024
Merged

Conversation

NvTimLiu
Copy link
Collaborator

Fix typo in spark-tests.sh for '--packages' test

Follow up: https://github.com/NVIDIA/spark-rapids/pull/10279/files#r1471168516

Signed-off-by: Tim Liu timl@nvidia.com

Fix typo in spark-tests.sh for '--packages' test

Follow up: https://github.com/NVIDIA/spark-rapids/pull/10279/files#r1471168516

Signed-off-by: Tim Liu <timl@nvidia.com>
@NvTimLiu NvTimLiu added the build Related to CI / CD or cleanly building label Jan 30, 2024
@NvTimLiu NvTimLiu self-assigned this Jan 30, 2024
@NvTimLiu
Copy link
Collaborator Author

NvTimLiu commented Jan 30, 2024

Sorry I had a typo error when filing the PR10279

Test script with below env, got PASS or SKIP

  if [[ "$CLASSIFIER" == "" || "$CLASSIFIER" == "cuda11" ]]; then
    SPARK_SHELL_SMOKE_TEST=1 \
    PYSP_TEST_spark_jars_packages=com.nvidia:rapids-4-spark_${SCALA_BINARY_VER}:${PROJECT_VER} \
    PYSP_TEST_spark_jars_repositories=${PROJECT_REPO} \
      ./run_pyspark_from_build.sh
  fi

CLASSIFIER='cuda11' or CLASSIFIER='', scala2.12

+ SPARK_SHELL_ARGS_ARR+=(--packages "${PYSP_TEST_spark_jars_packages}")
+ [[ -n $ART_URL ]]
+ SPARK_SHELL_ARGS_ARR+=(--repositories "${PYSP_TEST_spark_jars_repositories}")
+ spark-3.3.0-bin-hadoop3/bin/spark-shell --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.deploy.maxExecutorRetries=0 --packages com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT --repositories $ART_URL
+ grep -F 'res0: Array[org.apache.spark.sql.Row] = Array([4950])'
res0: Array[org.apache.spark.sql.Row] = Array([4950])
+ echo 'SUCCESS spark-shell smoke test'
SUCCESS spark-shell smoke test

CLASSIFIER='cuda12' or CLASSIFIER='cuda11-arm64', scala2.12

SKIP

CLASSIFIER='cuda11' or CLASSIFIER='', scala2.13

+ SPARK_SHELL_ARGS_ARR+=(--packages "${PYSP_TEST_spark_jars_packages}")
+ [[ -n $ART_URL ]]
+ SPARK_SHELL_ARGS_ARR+=(--repositories "${PYSP_TEST_spark_jars_repositories}")
+ spark-3.3.0-bin-hadoop3-scala2.13/bin/spark-shell --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.deploy.maxExecutorRetries=0 --packages com.nvidia:rapids-4-spark_2.13:24.02.0-SNAPSHOT --repositories $ART_URL
+ grep -F 'res0: Array[org.apache.spark.sql.Row] = Array([4950])'
scala> spark.range(100).agg(Map("id" -> "sum")).collect()val res0: Array[org.apache.spark.sql.Row] = Array([4950])
+ echo 'SUCCESS spark-shell smoke test'

CLASSIFIER='cuda12' or CLASSIFIER='cuda11-arm64', scala2.13

SKIP

@NvTimLiu
Copy link
Collaborator Author

Skip CI as this script will not be executed in pre-merge CI, it is only be called in integration test CI

@NvTimLiu
Copy link
Collaborator Author

build

@jlowe jlowe merged commit 252111a into NVIDIA:branch-24.02 Jan 30, 2024
41 checks passed
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants