perf(cuBLAS): store device pointers in ggml_tensor

[jon-chuang]

We do not need to do (DTH/HTD) copy of tensor data. This is like how pytorch does it.

In self-attention, the kv cache could still be on host, but the host launches kernels on the device data based on the cache.

The ggml_tensor provides methods to sync to the operator device type to hide complexity.

Unfortunately, lazy sync is not the smartest way - knowing the full compute graph is much better to identify sync points; then one can overlap copy and compute wherever sync is required.

example:

if graph.sync_required(&tensor) {
  cudaAsyncCopy(...); // e.g. DTH
}

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.