Bad performance for "hgetall"

[mkotsbak]

Trying to get a hashmap with 10 000 simple elements like this:

(tagkey8193,tagvalue8193)
(tagkey8748,tagvalue8748)
(tagkey6678,tagvalue6678)
(tagkey9674,tagvalue9674)

is extremely slow. It takes around 1 seconds to get all those values out, with a simple mapping. I suspect the algorithm to parse the results is too slow, and maybe is performing exponentially slow with increasing values.

[etaty]

This is were thing are happening:

rediscala/src/main/scala/redis/api/Hashes.scala

Line 26 in ecd2c7b

case class Hgetall[K, R](key: K)(implicit redisKey: ByteStringSerializer[K], deserializerR: ByteStringDeserializer[R]) extends RedisCommandMultiBulk[Map[String, R]] {

[etaty]

Oh I see, it is iterating twice on the reply

rediscala/src/main/scala/redis/api/Hashes.scala

Line 31 in ecd2c7b

val seq = r.map(_.toByteString)

Also I could give a sizeHint to the map builder

[etaty]

rediscala/src/main/scala/redis/protocol/RedisProtocolReply.scala

Line 171 in ecd2c7b

def bulks(bs: ByteString, i: Int, acc: mutable.Buffer[RedisReply]): Option[(MultiBulk, ByteString)] = {

Here we could also use a Seq builder and give a sizeHint

[mkotsbak]

Japp, just move the map inside the builder?

[mkotsbak]

Moved back to https://github.com/debasishg/scala-redis with connection pooling. The same test (with 20k values) performs around 15 req/s.

[etaty]

It should be resolved now

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
libraryDependencies += "com.github.etaty" %% "rediscala" % "1.6.0-SNAPSHOT"

Tell me if you don't see similar perf ( I can do 15 hgetall/s with 10k elements with one single connection)

[mkotsbak]

Hmm, much better after the modifications:
Requests per second: 3.72 #/sec

but it still seems far behind scala-redis with the same data in the DB:

Requests per second: 14.45 #/sec

[etaty]

Thanks,
I will try to do a benchmark diff with scala-redis
Can you post the code you are running in a gist?

[mkotsbak]

I'm sorry, the code cannot be disclosed, but it should be easy to reproduce using scala-redis future with connection pool example. Is this project using multiple connections, or just one?

[etaty]

Hum I see, the main difference is the number of connection to the redis server.
A RedisClient is just one connection while scala-redis use a pool in your bench.
But yeah I will publish some benchmarks.

[etaty]

Okay,
I did rewrite the redis reply decoder, so it can keep a much closer state of what has been decoded. Something like progressive decoding (or async decoding).

Before I was decoding a full redis response, and if I did not have the full response, I was starting again when receiving the next packet. It works with small response. But the response for HGETALL with 10000 hashes is around 1MB! So the decoding is deadly slow.

Conclusion is that now I am even faster than scala-redis. I will clean and post the result before the end of the year.

[mkotsbak]

Great! I can't see any code here yet?

[etaty]

I create a PR #119

[etaty]

snapshot available

resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
libraryDependencies += "com.github.etaty" %% "rediscala" % "1.6.0-SNAPSHOT"