lmdb_kv_store

• LMDB KV store offers the following functionality:

  • Python dictionary like interface
  • Persistent storage
  • Multi-process read safe without lock
  • Multi-process write safe with lock

• Methods

  • setitem
  • getitem
  • delitem
  • contains
  • iter
  • len
  • keys
  • values
  • pop
  • get_multi
  • set_multi
  • flush
  • close
  • cleanup

• Open Python terminal from the conda environment

# Activate conda env
conda activate pykvstores

# Open Python terminal
python

Examples

# Import the LMDBKVStore class
>>> from pykvstores.lmdb_kvstore import LMDBKVStore
# Create a LMDB backed key-value store
>>> store = LMDBKVStore("/tmp/pykvstore.lmdb")
# Set a key-value pair
>>> store["key"] = "value"
# Set multiple key-value pairs
>>> store.set_multi(keys=[1, 2, 3], values=[4, 5, 6])
# To write the data to the file system (persistent storage, must be called before closing the store otherwise data won't be written to the disk)
>>> store.flush()
# Re-open the store in read only mode and without lock (for concurrent access)
>>> store = LMDBKVStore("/tmp/pykvstore.lmdb", readonly=True, lock=False)
# Get a key-value pair
>>> store["key"]
"value"
# Delete a key-value pair
>>> store.pop("key")
"value"
# Contain a key
>>> "key" in store
False
# Iterate over the keys and values
>>> for key, value in store:
...     print(key, value)
key value
# Close the store
>>> store.close()
# Cleanup the store (remove the file and the lock file)
>>> store.cleanup()
# For more details
>>> help(LMDBKVStore)

• Multiprocessing safe (write and read example)

from multiprocessing import Process
from pykvstores.lmdb_kvstore import LMDBKVStore

#### Write example ####
# Create the LMDB store
store = LMDBKVStore("/tmp/pykvstore.lmdb")
processes = []

def square(num_list, store):
    """ Square the numbers in the list and store the result in the store"""
    for num in num_list:
        store[num] = num * num

# Let's square the numbers in the list and write the result to the store using multiple processes (in this example, we use 4 processes)
for i in range(0, 400, 100):
    p = Process(target=square, args=(range(i, i + 100), store))
    processes.append(p)
    p.start()

for proc in processes:
    proc.join()

# Flush the data to the file system
store.flush()

# Close the store
store.close()

#### Read example ####
# Reopen the store in read only mode without lock for concurrent read access
store = LMDBKVStore("/tmp/pykvstore.lmdb", readonly=True, lock=False)

# Check if the store contains all the squares
assert sorted(store.values()) == [i * i for i in range(0, 400)]

# We can also read the data from the store in a multi-process safe way
processes = []

def read_square(num_list, store):
    """ Read the square of the numbers in the list from the store"""
    for num in num_list:
        assert store[num] == num * num

# Let's read the squares from the store using multiple processes (in this example, we use 4 processes)
for i in range(0, 400, 100):
    p = Process(target=read_square, args=(range(i, i + 100), store))
    processes.append(p)
    p.start()

for proc in processes:
    proc.join()

# Close the store and cleanup (We don't have to call cleanup if we want to re-use the store in future/later)
store.close()
store.cleanup()