Multiple simultaneous file moves to the same location when torrents complete.

[CyrusNajmabadi]

I've reported this over at qbitorrent:qbittorrent/qBittorrent#9407

However, i'm not sure if it's a qbitorrent issue or a libtorrent issue. If this doesn't belong here, my apologies.

I have qb set to download to one location (an ssd array), but then move torrents to a final location on large spinning-rust drives. Unfortunately, something changed in teh last few versions, possibly related to the "number of IO threads" option. The option for me is set at 4 (the default), and this seems to cause some extremely poor behavior.

Specifically, qb will be downloading several torrents simultaneously. The ssd array is easily able to keep up with the download rate (around 250mb/s). When one completes, it then starts getting moved to the HD array. This goes at around 640mb/s, so well above the rate that i can even download torrents. This all works fine as long as no other torrent finishes during the time when the torrent is being moved.

However, if another torrent does finish, it's a disaster. qb will then start moving that torrent as well to the destination (presumably because there are 4 IO threads available), but the simultaneous sequential writes just destroy disk performance (presumably because of all the jumping back and forth the HD head has to make). I'll often drop to something like 100mb/s, or worse. This has a compounded effect. Because there are now two torrents being copied, and the throughput is so low, it's even more likely that another torrent will finish, and start moving as well. With 3 torrents writing at the same time, disk performance is dismal, sometimes giving down to 20mb/s per torrent.

At this point, things don't get much worse. Mainly because 3 of the 4 async IO threads are occupied, and so qb doesn't seem to even want to move any more files until any of the other 3 are done. However, because it can't seem to do the writes it then wants to throttle torrenting, even though my ssd array would have no problem handling things in the interim period.

--

My request would be: do not perform moves in parallel. Or, at least, if you are going to do it in parallel, only do it if you can somehow guarantee you're not reading/writing to the same slow-spinny-HDs. These moves (which are often 10s of GB), can normally take just a few minutes to move if done sequentially. However, doing things in parallel actually ends up not only taking orders of magnitude longer, but it also ends up dramatically hurting torrenting performance due to consuming all the available IO resources just for highly contentious moves.

Thanks!

[arvidn]

It sounds like this is primarily a qBT issue. libtorrent could implement some queuing mechanism and detection of which volume is being copied to, but it seems like a stretch of the scope of the library.

libtorrent currently just does what it's told. If qBT asks it to move 3 torrents, they will be moved, and 3 I/O threads will be used doing it (if available). I think the simplest way to address this would be for qBT to have an option to not move torrents in parallel.

As an aside, while a torrent is being moved, it's blocking all other disk I/O to any of its files, so it won't be seeded to peers during that time. Or rather, if it takes a few minutes, peers requesting blocks from it may time-out.

[CyrusNajmabadi]

Ah gotcha. I can understand the desire to stay out of the queueing game. I thought it might be something easy to do in this end. One thought I had was whether or not the io threads could have capability bits saying what is ops they supported. That way, qb could mark just one of the threads as supporting moves. Then, even if they issued N move requests, only that thread would end up servicing them.

This is the approach I've used in my own projects to avoid the complexity of queuing, while also preventing problems due to overlapping operations that are contentious.

Would something like that be possible?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.