Database as source instead of TCP sockets (baking)

[Zbysekz]

Rework of vis to use SQLite3 database.

[Zbysekz]

discussed in this thread
Separated here to keep topic clear.

[Zbysekz]

repliyng to @breznak from here

yes, but it's a bit smart. Stores only sparse synapses, also in binary format (possibly can even compress?).

So, I quick test: TM(2048cols, 32 cells per col), ran a couple of iterations of randomized SDR. tm.saveToFile("/tmp/tm.dump") results in ~1MB. Which would be much more reasonable. Could you try this on your workload?

I generated dumps for 1000 hotgym iterations and it has 1,54GB (compressed to zip leads to 525MB)
Amazing! I am surprised by the ease of use of such a serialization! 😮 👍

Since iterating and crunching of distal synapses in python can take more than 40sec per iteration i will store distal DUMP besides sqlite3 database and pandaVis will grab what it needs afterwards(by use of tm.connections.readFromFile(fileName).

Thanks for that advice :)

[breznak]

Since iterating and crunching of distal synapses in python can take more than 40sec per iteration

ok, you have a 2nd use-case for this, I've already started some work in htm-community/htm.core#668
but haven't gotten to it lately. I'll be able to give it some time next week.

So the querying might get a bit faster.

[Zbysekz]

Finished