Recheck and move multiple torrents one-by-one

[a-raccoon]

Current Behavior: Performing a Force Recheck on multiple selected torrents results in concurrent scans of multiple files across multiple torrents all at the same time. On platter drives, this means lots of random seeking (slow). Same holds true when multiple torrents are being moved from temp to completion folder, or via category or destination folder change.

Desired Behavior: Sequential (one at a time) file rechecking or moving by implementing a queuing system.

[Syst3mSh0ck]

Not exactly ideal when you have a lot of torrents that require a recheck. I would also prefer the option to choose between serial/parallel processing for force rechecks.

[FranciscoPombal]

Related and relevant discussion: #9061
@arvidn @airium

[arvidn]

this is the behaviour for torrents that are "force-started" (i.e. not auto-managed). Is it possible that qBitTorrent clears the auto-managed flag whenever the user force-rechecks a torrent?
Otherwise, perhaps all your torrents were force-started before you force-rechecked them.

[a-raccoon]

@Syst3mSh0ck : Your statement is confusing. At first it sounds like you think my proposal is rubbish, but then it sounds like you mean the current behavior is rubbish. Please clarify.

@arvidn : I don't know about any of that. I don't use "Force Start", but I'll use "Force Recheck" on a selection of torrents that have completed, where I change Set Location to another harddrive, and then I use Force Recheck to confirm that the files still have integrity (as a file-copy verification).

In any event, it seems always-stupid, under any circumstance, for qBittorrent to recheck more than one torrent at a time, or more than one torrent from the same storage device. This just destroys any hope of performance.

[slrslr]

@sledgehammer999 In my case (qbt 4.1.1, WIn10) when i highlight 2 torrents and do Force recheck,
then one torrent is stalled at 0%, other is 99%. Then when i recheck "0%" one, it works and ends at 99%.

[dakusan]

Same here. Just imported from uTorrent and am having to recheck torrents 1 at a time due to it trying to recheck everything in parallel, causing thrashing from non-sequential reads.

[Edit] I realized that if I first force recheck them all, then pause them, and then unpause them, it does the rechecking serially. So while still a [bug/feature], not debilitating.

As a feature, it would also be nice if the rechecking could be done in parallel per mounted partition.

[a-raccoon]

Thanks for that find @dakusan ! I will have to try that trick next time... Select multiple torrents, choose Force Recheck, then Pause, then Resume.

[iheartcsharp]

Based on the libtorrent docs there is a separate queue for torrents that are in checking status. The default value for it is supposed to be 1, so it should check 1 torrent at a time. So not sure how qBittorent is checking them all. I checked the latest source code and it doesn't seem to be setting the active_checking value anywhere.

[arvidn]

torrents where the auto_managed flag is not set, are not subject to any of the queuing logic

[slrslr]

torrents where the auto_managed flag is not set, are not subject to any of the queuing logic

earlier in this thread i mentioned i am having this issue on Windows 10, 64b, qbt 4.1.1
now i tried on Windows 7, 64b, qbt 4.1.4 and no problem, i tried to rechceck multiple manual, multiple automatic management torrents and mixture. These torrents was all 100% completed and it always checked 1 torrent at a time, but when i did same for incomplete (like lets say 10% downloaded), it checked multiple torrents at same time, but apparently again without problem of setting torrent to 0% downloaded as described in my previous reply..

[a-raccoon]

Someone mentioned earlier in this thread that the problem is with FORCE Recheck. You have to turn off FORCE and then it queues the rechecks one-by-one. You cannot begin a recheck without first doing a FORCE recheck. I have verified this behavior myself.

[slrslr]

You cannot begin a recheck without first doing a FORCE recheck

I do not understand what you describe and how to "turn OFF FORCE", but in this thread when i spoken about "recheck" i always meant "Force recheck" option triggerred manually by the user from the torrent right click menu.

[a-raccoon]

You turn off FORCE by clicking it again. Perform the select, right-click menu routine twice. Then it will queue rechecks one-by-one.

Force -> Un-force -> Force -> Un-force -> Force -> Un-force ...

[thalieht]

From what i've seen, rechecking paused torrents clears the auto_managed flag for them. Here's a patch that fixes that although i have a feeling it's a workaround. ping @qbittorrent/demigods

--- a/src/base/bittorrent/torrenthandle.cpp
+++ b/src/base/bittorrent/torrenthandle.cpp
@@ -1270,13 +1270,21 @@ void TorrentHandle::forceRecheck()
 {
     if (!hasMetadata()) return;
 
+    if (hasError())
+        m_nativeHandle.clear_error();
+    if (hasMissingFiles())
+        m_hasMissingFiles = false;
+    if (isForced()) {
+        m_startupState = NotStarted;
+        m_needsToStartForced = true;
+    }
+
+    m_nativeHandle.auto_managed(true);
     m_nativeHandle.force_recheck();
     m_unchecked = false;
 
-    if (isPaused()) {
+    if (isPaused())
         m_nativeHandle.stop_when_ready(true);
-        resume_impl(true, true);
-    }
 }
 
 void TorrentHandle::setSequentialDownload(bool b)

[glassez]

@thalieht, I do not have time to see it in detail yet, I will do it later. At least touching m_startupState is totally incorrect and it breaks related logic. "startup state" is for handling initial startup of torrent. When initialization is completed the torrent is always in "started" state.

[thalieht]

That's why i said workaround. I just wanted to get to

qBittorrent/src/base/bittorrent/torrenthandle.cpp

Line 1558 in b87987e

resume(m_needsToStartForced);

[glassez]

From what i've seen, rechecking paused torrents clears the auto_managed flag for them.

It's because of resume_impl(true, true);. Can you try resume_impl(false, true); instead?

[thalieht]

Doh! I knew that but didn't think to try it. However i found this now: #8711

To get around this the torrent is placed in upload_mode and out of
automanagement.

[glassez]

Doh! I knew that but didn't think to try it. However i found this now: #8711

But since my last changes we no more rely on "checked" alert. Now it is driven by "stop_when_ready" flag and libtorrent pauses it synchronously. So we can try to revert it and use resume_impl(false, false); here.

[thalieht]

Here arvidn/libtorrent#2759 (comment) @sledgehammer999 says he tried various stuff including "stop_when_ready". Maybe wait for some feedback?

[iheartcsharp]

Ideally libtorrent should limit heavy disk operations such as moving/checking to 1 per disk. I don't understand any reason to check 2+ torrents on the same drive all at the same time. Even with SSDs performance suffers if you allow that. Also auto torrent management shouldn't really apply to these heavy disk operations. I think a better solution would be for qBittorrent to manage its own queue so that libtorrent only checks 1 torrent at a time.

For example, let's say you select 5 torrents and click Force Recheck. qBittorrent should have a separate queue where it puts those torrents in a list. It then takes the 1st torrent from that list and update the state in libtorrent. It waits until the torrent check is done, then it takes then next torrent and switches the state.

I wish I could help out but I haven't touched C/C++ in a very long time. Been a C# developer the last 15 years.

[arvidn]

@iheartcsharp I may have missed something earlier. Is there a reason the libtorrent queue is not sufficient? you can set active_checking to 1 (which is the default). Torrents that are not auto managed (i.e. have the torrent_flags::auto_managed flag set) aren't subject to any limits. Is it important that the checking limit be a special case to allow non-auto-managed torrents to be subject to it? if so, why?

The main benefit I can see with qbt managing checking torrents itself is that it can try to assess which drive a torrent belongs to, and have separate queues per drive. This is a surprisingly complicated thing to do though. Perhaps it can be made to work in the most common cases. But there's nothing that says that all files in a torrent are stored on the same drive. In fact, if you have a logical volume where the block device span multiple drives, even a single file can live on multiple drives.

[a-raccoon]

Given the present-day extreme of forced rechecking of every torrent at the same time, the ideal extreme would be to only check 1 torrent at a time, regardless of where the files live.

As I mentioned earlier, this latter behavior is possible, but you have to issue the Re-check action twice, because the first action is "Force Recheck" and then the second action is "Recheck"(without force).

So the solution, for now, is to remove that Force toggle from the first action. That's the fastest fix we can accomplish and release by next version.

[thalieht]

@sledgehammer999 in arvidn/libtorrent#2759 (comment) you said you tried various stuff including stop_when_ready() which is now used by the way. What didn't work with it? Was it the thing that patch fixed? If yes, and since we'll only support some some version after 1.1.7 as the minimum, can auto_managed be enabled again on rechecking paused torrents?
In #8711 you mention about the race condition that stop_when_ready() prevents.
From the docs:

The stop-when-ready flag fixes the inherent race condition of waiting
for the state_changed_alert and then call pause().

[glassez]

Can you check moving torrents (setting a location on a different drive) as well?

Maybe next time I can take a few hours off my personal life...

[mzso]

@glassez commented on 2019. jún. 30. 19:36 CEST:

Can you check moving torrents (setting a location on a different drive) as well?

Maybe next time I can take a few hours off my personal life...

Well, I was replying to @a-raccoon who seemed eager to test.

But since that's bugged as well (not enqueued at all) , if it's to be fixed a dev needs to look at it at some point.

[amloessb]

@glassez: I tried the binary provided, but it crashed immediately after asking if I wanted to make it the default application for torrents and magnet links. I have a minidump, so let me know if there's a way I can send it to you.

[Ryrynz]

Any chance of someone building a 4.1.7 release after this fix and whatever is left to backport? Do you need Sledgehammer for that? Seems he's gone off grid.

[thalieht]

This is finally fixed in 4.1.7 right?

[Ryrynz]

You did the testing didn't you? I think it's safe to close.

[a-raccoon]

Was Moving... fixed too, or do moves still happen in parallel instead of sequentially?

[Ryrynz]

They still happen in parallel.. checked with 4.20 Alpha as well. :(

[thalieht]

@Ryrynz one or more of those torrents were Forced right? I can reproduce in that case.

@a-raccoon still parallel.

[Ryrynz]

Nope, didn't force any. Simply added two torrents, downloaded and moved.

[thalieht]

Moved? This is about parallel rechecking.

[Ryrynz]

Add that to the list too then.

[arvidn]

@Ryrynz typically it's better to have tickets that are specific, and that can be closed when that specific issue has been resolved. You should probably create (or comment in an existing ticket, I'm sure there is one) about parallel moving of torrents. The topic of this ticket is checking of torrents in parallel.

[mzso]

@thalieht commented on 2019. szept. 8. 12:03 CEST:

Moved? This is about parallel rechecking.

Moving was mentioned as well, because it's a similar issue with just as painful consequences.

[mzso]

@arvidn commented on 2019. szept. 8. 13:22 CEST:

@Ryrynz typically it's better to have tickets that are specific, and that can be closed when that specific issue has been resolved. You should probably create (or comment in an existing ticket, I'm sure there is one) about parallel moving of torrents. The topic of this ticket is checking of torrents in parallel.

Because of the obscene amount of issues it's not too likely for a new one to get attention.

[mzso]

@arvidn commented on 2019. szept. 8. 13:22 CEST:

@Ryrynz typically it's better to have tickets that are specific, and that can be closed when that specific issue has been resolved. You should probably create (or comment in an existing ticket, I'm sure there is one) about parallel moving of torrents. The topic of this ticket is checking of torrents in parallel.

Anyway there already appears to be an issue (#9158) opened by him, then went unnoticed.

And avoiding needless moves would also help: #7916

[a-raccoon]

I think moving was also mentioned here because @glassez (et al) pull #10871 was perhaps intended to fix both issues of sequential rechecking and sequential moving. It only makes sense that these two file operations should even share the same code, especially given the qbt option to recheck a torrent after a torrent has completed (AND presumably moved from \temp to \complete).

@iheartcsharp commented on Jan 8

Ideally libtorrent should limit heavy disk operations such as moving/checking to 1 per disk. I don't understand any reason to check 2+ torrents on the same drive all at the same time. Even with SSDs performance suffers if you allow that. Also auto torrent management shouldn't really apply to these heavy disk operations. I think a better solution would be for qBittorrent to manage its own queue so that libtorrent only checks 1 torrent at a time.

File moving has been a part of this thread since the beginning.

[CyrusNajmabadi]

Is there any way for file moving to be addressed at all? I find it somewhat baffling that a library and tool written around the idea of extremely heavy IO would also be written in such a way as to hurt performance so heavily.

Does qbitorrent not have any sort of internal queues where async-io can be done? it seems like it would jsut need to enqueue these moves to that queue to be done sequentially.

It seems like this must be possible given that it was possible to "check" files one at a time. If we can "check" one a time... why can't hte same be done for moves?

[sgmihai]

Please enable some option to enqueue moves, I have 3GB/s ssd temporary download location and 1GB/s capable disk array, yet when I get 7+ moves happening all at once, everything goes to shit, takes forever to move, and even torrents transfers stop, probably because of the disk overload detection.

[Ryrynz]

I just changed a location for a label I have and rather than offer to move the torrents or just not do it (like I hoped it would) it just started moving all of them at once.

Looking at this closer though it appears as if it's actually moving them sequentially, all the torrents are marked as "Moving" but my disk performance is anywhere from 60MB/s to 90MB/s which it sure wouldn't be if it was moving them all at once. The disk contents also appear to be confirming this to be the case.

So it looks like ideally we'd just have the torrents marked "Moving" when they're being moved and when not, marked as Queued for Move.

[a-raccoon]

Ideally the progress indicator might do something too.

[ghost]

@FranciscoPombal this should be closed now right? There's a queueing system in place now for moving torrents.

[FranciscoPombal]

@An0n666 yes. The requested features are now available thanks to:
#10242
#10266
#12035
#12188