Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Array loaderbase merge #121

Merged
merged 6 commits into from
Apr 4, 2023
Merged

Array loaderbase merge #121

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ed717af
fix array model tests imports
jperez999 Apr 3, 2023
ef319a9
remove arrayloaderbase entirely
jperez999 Apr 3, 2023
2bd47a3
remove array dataloader altogether all logic in loader base
jperez999 Apr 3, 2023
0198822
remove pack method no longer required
jperez999 Apr 3, 2023
2b4521c
fixing jax to accomodate new data loader logic
jperez999 Apr 3, 2023
a9468fe
merge in main
jperez999 Apr 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 0 additions & 282 deletions merlin/dataloader/array.py
View file
Open in desktop

This file was deleted.

29 changes: 0 additions & 29 deletions merlin/dataloader/jax.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@
#
import contextlib

import jax
import jax.dlpack
import jax.numpy as jnp
import numpy as np

Expand Down Expand Up @@ -72,33 +70,6 @@ def _split_fn(self, tensor, idx, axis=0):
def _sum(self, tensor):
return tensor.sum()

def _to_tensor(self, df_or_series):
if df_or_series.empty:
return

transpose = False

# checks necessary because of this bug
# https://github.com/tensorflow/tensorflow/issues/42660
# same logic as in TF dataloader
if len(df_or_series.shape) == 1 or df_or_series.shape[1] == 1:
dlpack = self._pack(df_or_series)
else:
transpose = True
dlpack = self._pack(df_or_series.values.T)

x = self._unpack(dlpack)

if transpose:
x = x.T

return x

def _unpack(self, gdf):
if hasattr(gdf, "shape"):
return jax.device_put(gdf)
return jax.dlpack.from_dlpack(gdf)

def _cast_to_numpy_dtype(self, dtype):
# jax uses numpy dtypes, so this is kinda easy
return dtype
Expand Down
Loading