Busy hang

[jwr]

I hit another problem while inserting large amounts of data: the main process eventually hangs at close to 400% CPU usage, while RethinkDB stops receiving new data.

This happened after a loop inserted 14432000 records. The total number to be inserted was 24795802.

Records were inserted in batches (vectors) of 1024 at a time.

Unfortunately, I have little to go on, as the main process just continued spinning, there were no exceptions to be seen. I captured a CPU sample using YourKit, which might point someone to the problem.

This is using [com.apa512/rethinkdb "0.11.0"], so it's not the memory leak I reported before

[danielcompton]

Just to check, are you waiting for each result to return before running the next insert?

[jwr]

Hmm. The real answer is "I don't know" :-)

I just run a loop that builds the insert queries and calls r/run on them. The loop is a doseq, so I'm not holding onto the results of the inserts.

[danielcompton]

r/run is blocking, so that should be waiting. I'll take a look tomorrow.

[jwr]

Also, this is a fairly large table. Before that I was able to insert 1878859 records and 3699819 records into two other tables, so this is not something that happened immediately.

[danielcompton]

Hey sorry I haven't had a chance to look into this just yet. As a temporary workaround, have you considered using multiple connections? e.g. closing the old connection and creating a new one every 100,000 inserts? Obviously not a great long term solution, but it might help in the short term.

[jwr]

Thanks, I don't really need a workaround, because this isn't anything urgent — I'm just trying to use clj-rethinkdb for larger things. It works for my smaller application (https://partsbox.io/), but so far seems to break for larger amounts of data.

I'm increasingly worried about the complexity in the driver and the (inevitable) resulting problems.

[danielcompton]

I'm looking to move all of the connection and results handling to the official Java RethinkDB driver as soon as there is a release of it, which offloads the hard part to them, and lets us just focus on the query language.

[jwr]

FWIW, I think this is a very good idea. We are a small community for now, so we should keep our code as simple as possible.

[danielcompton]

Can you try this again? We ran into a similar sounding issue (nothing was running, but no exceptions being thrown). The root cause turned out to be that all of the go threads were blocked waiting on reads from the db. The connection code has been rewritten with manifold which shouldn't suffer from this.

[jwr]

Of course. I will have to dig up that project and reproduce the problem again, which might take a while, though.

[jwr]

I retried this with 0.15.23. I was able to insert 24795801 records successfully with no hangs or any other problems. I'd say the bug can be closed!

I find it slightly alarming that performance drops with time when inserting larger numbers of records — from about 8-10k inserts/s at the beginning down to around 2k inserts/s after 23M records have been inserted (it's for a table with no indexes, and the data is fairly homogeneous). But I strongly suspect it's a RethinkDB issue. It seems to be reading about twice as much data as is being written to disk.

As for clj-rethinkdb, I noticed that its performance improved since 0.11.0 — I happened to have notes on how long a certain task took and compared them to the current results:

• 0.11.0: 275s
• 0.15.23: 193s

This is with a smaller task that doesn't hit RethinkDB's limitations. That's a 30% improvement! Nice!

[danielcompton]

Great to hear! The degraded performance after lots of inserts is interesting, although there's not really enough info to say whether that's RethinkDB, the driver, or something else entirely. If you felt like it, taking a VisualVM profile over time would be helpful to check if there's a memory leak which is causing excessive GCing.

Thanks for following up!

[jwr]

I confirmed that the degraded performance is due to RethinkDB and is to be expected: rethinkdb/rethinkdb#5805

[danielcompton]

Thanks for that, I'll watch that issue with interest.