Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Remove sparse tensor output type for list features #103

Merged

Conversation

oliverholworthy
Copy link
Member

@oliverholworthy oliverholworthy commented Feb 20, 2023

Remove sparse tensor output type for list features

Motivation

The value count attributes of columns in dataset.schema currently controls the output type of list columns.

With the addition of shape in the schema NVIDIA-Merlin/Merlin#813 we're going to start seeing value counts specified more. This will result in unexpected output type's if previously a value count was not specified.

We also currently have a possibility of output type sparse tensor which doesn't have a clear use-case and appears to be an implementation detail of padding ragged columns to dense.

Current

  • Dense Tensor
    • value_count.max specified and is_ragged=False
  • Sparse Tensor
    • value_count.max specified and is_ragged=True
  • Ragged Representation (values, offsets)
    • value_count.max not specified and is_ragged=True

After - With this PR

  • Ragged Representation
    Always returns this for all list columns. value count does not influence output type.

Planning a follow-up / independent change in #97 to return a dense tensor if the schema specifies that a column is of fixed size (is_ragged=False).

@oliverholworthy oliverholworthy marked this pull request as ready for review March 16, 2023 10:58
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
breaking enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants