Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[FEA] Support element_at with non-literal index #4814

Closed
viadea opened this issue Feb 17, 2022 · 2 comments · Fixed by #4858
Closed

[FEA] Support element_at with non-literal index #4814

viadea opened this issue Feb 17, 2022 · 2 comments · Fixed by #4858
Assignees
Labels
cudf_dependency An issue or PR with this label depends on a new feature in cudf feature request New feature or request P1 Nice to have for release

Comments

@viadea
Copy link
Collaborator

viadea commented Feb 17, 2022

I wish we can support element_at with non-literal index.

For example:

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row

val arrayData = Seq(
    Row("John",List("apple","orange","banana"),1),
    Row("David",List("apple","orange","banana"),2),
    Row("Harry",List("apple","other"),1)
)

val arraySchema = new StructType().add("name",StringType).add("fruits", ArrayType(StringType)).add("favorite",IntegerType)

val df = spark.createDataFrame(spark.sparkContext.parallelize(arrayData),arraySchema)
df.write.mode("overwrite").format("parquet").save("/tmp/testparquet")
val df2=spark.read.parquet("/tmp/testparquet")
df2.createOrReplaceTempView("df2")

spark.sql("select name,element_at(fruits,favorite) from df2").show()

The related unsupported messages:

!Expression <ElementAt> element_at(fruits#92, favorite#93, false) cannot run on GPU because ordinal only supports IntegerType if it is a literal value; index/key only supports IntegerType if it is a literal value
@viadea viadea added feature request New feature or request ? - Needs Triage Need team to review and classify labels Feb 17, 2022
@revans2
Copy link
Collaborator

revans2 commented Feb 18, 2022

We should update GpuGetArrayItem while we are at it, to make both consistent.

@firestarman
Copy link
Collaborator

Depends on rapidsai/cudf#10340

@firestarman firestarman added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Feb 22, 2022
@jlowe jlowe added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Feb 22, 2022
@sameerz sameerz added this to the Feb 28 - Mar 18 milestone Feb 26, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
cudf_dependency An issue or PR with this label depends on a new feature in cudf feature request New feature or request P1 Nice to have for release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants