Help with training an autoencoder on custom large data

[IanQS]

Hey all!

I've got a very large (130K x 130K), but sparse directed adjacency matrix that I'm trying to train an autoencoder on. The entries are either + indicating a positive connection and negative indicating a negative connection.

I'm basing my code off of Tutorial6.ipynb but updated to use things like RandomLinkSplit.

My code looks like the following:

def load_graph(npz_file_loc: Path) -> tuple[tuple[Tensor, Tensor], int]:
    data = np.load(npz_file_loc)
    rows, cols, vals, num_nodes = data["rows"], data["cols"], data["vals"], data["n"]

    sparse_cmat = csr_array((vals, (rows, cols)), shape=(num_nodes, num_nodes))

    edge_indices_edge_attrs = from_scipy_sparse_matrix(sparse_cmat)
    return edge_indices_edge_attrs, num_nodes

I was thinking that I could use it as such:

def run_modeling(edge_indices_edge_attrs, num_nodes)
    idx, attrs = edge_indices_edge_attrs
    data = Data(edge_index=idx, edge_attr=attrs, num_nodes=num_nodes)
    train_data, val_data, test_data = RandomLinkSplit(
        num_val=0.01, is_undirected=False
    )(data)

but I'm running into IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed.

---

I'm reading through the torch_geometric.data.Data documentation, and it explicitly mentions

x (torch.Tensor, optional) - Node feature matrix with shape [num_nodes, num_node_features].

My issue is that I'm not sure what to pass in... I have 130K "samples", and each sample can form a connection with the other 130K samples in the connectivity matrix. In a standard autoencoder we train the model to reconstruct its input. In my case, would my model be reconstructing the 130K connections? Which means that I'd need to pass in rows of size N x 130_000 for training?

I found another issue where the user is using from_scipy_sparse_matrix, Model Outputs The Same Value for All Graphs in Binary Classification , but that user's issue is different in that they have a separate dataset.