start-buckets! creates more threads than necessary

[namenu]

start-buckets! is using swap! to update bucket information, and swap! must have no side effects. ¹

superlifter/src/superlifter/core.cljc

Lines 194 to 200 in eb89a40

	(defn- start-buckets! [{:keys [buckets] :as context}]
	(swap! buckets
	#(reduce-kv (fn [buckets id _opts]
	(update buckets id (partial start-bucket! context id)))
	%
	%))
	context)

And this situation can easily be happening where the server is under heavy load.

This results in creating many unnecessary threads, hogging the thread pool.

Footnotes

1. https://clojuredocs.org/clojure.core/swap! - Note that f may be called multiple times, and thus should be free of side effects. ↩

[oliyh]

Hi,

Thanks for reporting this, I will try to think of another way of doing this.

Idea: Have a lock for adding/mutating buckets that must be also acquired for starting them

[namenu]

@oliyh Great!

FYI, we've been workaround with this fix.
green-labs@4fe1e26

[oliyh]

Hi,

Thanks for sharing your workaround. Just reading through it, here's what I see:

• Calling start-trigger now assocs a :start-fn to each bucket, rather than actually starting it
• (:buckets context) is no longer an atom, but (-> context :buckets bucket-id) is

This means that adding a bucket dynamically is not really possible anymore - in fact the code still has swap-vals on (:buckets context) which won't work anymore, so I guess you are not using this feature: green-labs@4fe1e26#diff-53fa11e40c1e91d8226bb3a21d4ee41f1c4f14f02ea75e6c4e9e43b57301944aR228

So I don't think I can use your solution as it is, but it has given me some ideas thank you.

[oliyh]

Hi, I've pushed 0.1.5-SNAPSHOT to clojars with a fix - are you able to verify it works for your use case?

Cheers