chore: cherry-pick changes from main to release (#1857)

Cherry picks: #1692 , #1790, #1701, #1724, #1702 , #1766 ,
part of #1808, #1853 , update ChangeLog

Author: joshua-goldstein

CHANGELOG.md

@@ -3,6 +3,18 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
+## [3.2103.5] - 2022-12-15
+
+We release Badger CLI tool binaries for amd64 and now arm64. This release does not involve any core code changes to Badger. We add a CD job for building Badger for arm64.
+
+## [3.2103.4] - 2022-11-04
+
+### Fixed
+  - fix(manifest): fix manifest corruption due to race condition in concurrent compactions (#1756)
+
+### Chores
+  - We bring the release branch to parity with main by updating the CI/CD jobs, Readme, Codeowners, PR and issue templates, etc.
+
 ## [3.2103.3] - 2022-10-14
 
 ### Remarks

Makefile

@@ -50,7 +50,6 @@ jemalloc:
 dependency:
 	@echo "Installing dependencies..."
 	@sudo apt-get update
-	@sudo apt-get -y upgrade
 	@sudo apt-get -y install \
     	ca-certificates \
     	curl \

README.md

@@ -42,19 +42,22 @@ For more details on our version naming schema please read [Choosing a version](#
 [Changelog]:https://github.com/dgraph-io/badger/blob/master/CHANGELOG.md
 
 ## Table of Contents
- * [Getting Started](#getting-started)
-    + [Installing](#installing)
+- [BadgerDB](#badgerdb)
+  - [Project Status](#project-status)
+  - [Table of Contents](#table-of-contents)
+  - [Getting Started](#getting-started)
+    - [Installing](#installing)
       - [Installing Badger Command Line Tool](#installing-badger-command-line-tool)
       - [Choosing a version](#choosing-a-version)
-  * [Badger Documentation](#badger-documentation)
-  * [Resources](#resources)
-    + [Blog Posts](#blog-posts)
-  * [Design](#design)
-    + [Comparisons](#comparisons)
-    + [Benchmarks](#benchmarks)
-  * [Projects Using Badger](#projects-using-badger)
-  * [Contributing](#contributing)
-  * [Contact](#contact)
+  - [Badger Documentation](#badger-documentation)
+  - [Resources](#resources)
+    - [Blog Posts](#blog-posts)
+  - [Design](#design)
+    - [Comparisons](#comparisons)
+    - [Benchmarks](#benchmarks)
+  - [Projects Using Badger](#projects-using-badger)
+  - [Contributing](#contributing)
+  - [Contact](#contact)
 
 ## Getting Started
 
@@ -176,7 +179,6 @@ Below is a list of known projects that use Badger:
 * [Sandglass](https://github.com/celrenheit/sandglass) - distributed, horizontally scalable, persistent, time sorted message queue.
 * [TalariaDB](https://github.com/grab/talaria) - Grab's Distributed, low latency time-series database.
 * [Sloop](https://github.com/salesforce/sloop) - Salesforce's Kubernetes History Visualization Project.
-* [Immudb](https://github.com/codenotary/immudb) - Lightweight, high-speed immutable database for systems and applications.
 * [Usenet Express](https://usenetexpress.com/) - Serving over 300TB of data with Badger.
 * [gorush](https://github.com/appleboy/gorush) - A push notification server written in Go.
 * [0-stor](https://github.com/zero-os/0-stor) - Single device object store.
@@ -211,6 +213,9 @@ Below is a list of known projects that use Badger:
 * [vxdb](https://github.com/vitalvas/vxdb) - Simple schema-less Key-Value NoSQL database with simplest API interface.
 * [Opacity](https://github.com/opacity/storage-node) - Backend implementation for the Opacity storage project
 * [Vephar](https://github.com/vaccovecrana/vephar) - A minimal key/value store using hashicorp-raft for cluster coordination and Badger for data storage.
+* [gowarcserver](https://github.com/nlnwa/gowarcserver) - Open-source server for warc files. Can be used in conjunction with pywb 
+* [flow-go](https://github.com/onflow/flow-go) - A fast, secure, and developer-friendly blockchain built to support the next generation of games, apps and the digital assets that power them.
+* [Wrgl](https://www.wrgl.co) - A data version control system that works like Git but specialized to store and diff CSV.
 
 If you are using Badger in a project please send a pull request to add it to the list.
 
@@ -219,6 +224,6 @@ If you are using Badger in a project please send a pull request to add it to the
 If you're interested in contributing to Badger see [CONTRIBUTING](./CONTRIBUTING.md).
 
 ## Contact
-- Please use [discuss.dgraph.io](https://discuss.dgraph.io) for questions, feature requests and discussions.
-- Please use [discuss.dgraph.io](https://discuss.dgraph.io) for filing bugs or feature requests.
+- Please use [Github issues](https://github.com/dgraph-io/badger/issues) for filing bugs.
+- Please use [discuss.dgraph.io](https://discuss.dgraph.io) for questions, discussions, and feature requests.
 - Follow us on Twitter [@dgraphlabs](https://twitter.com/dgraphlabs).

VERSIONING.md

@@ -43,5 +43,5 @@ For more background on our decision to adopt Serialization Versioning, read the
 [Semantic Versioning, Go Modules, and Databases][blog] and the original proposal on
 [this comment on Dgraph's Discuss forum][discuss].
 
-[blog]: https://blog.dgraph.io/post/serialization-versioning/
+[blog]: https://open.dgraph.io/post/serialization-versioning/
 [discuss]: https://discuss.dgraph.io/t/go-modules-on-badger-and-dgraph/4662/7

docs/content/faq/index.md

@@ -131,3 +131,12 @@ Badger does not directly use CGO but it relies on https://github.com/DataDog/zst
 zstd compression and the library requires `gcc/cgo`. You can build badger without cgo by running
 `CGO_ENABLED=0 go build`. This will build badger without the support for ZSTD compression algorithm.
 
+As of Badger versions [v2.2007.4](https://github.com/dgraph-io/badger/releases/tag/v2.2007.4) and
+[v3.2103.1](https://github.com/dgraph-io/badger/releases/tag/v3.2103.1) the DataDog ZSTD library
+was replaced by pure Golang version and CGO is no longer required. The new library is
+[backwards compatible in nearly all cases](https://discuss.dgraph.io/t/use-pure-go-zstd-implementation/8670/10):
+
+	> Yes they are compatible both ways. The only exception is 0 bytes of input which will give
+	> 0 bytes output with the Go zstd. But you already have the zstd.WithZeroFrames(true) which
+	> will wrap 0 bytes in a header so it can be fed to DD zstd. This will of course only be relevant
+	> when downgrading.

docs/content/get-started/index.md

@@ -54,7 +54,7 @@ To open your database, use the `badger.Open()` function, with the appropriate
 options. The `Dir` and `ValueDir` options are mandatory and must be
 specified by the client. They can be set to the same value to simplify things.
 
-```
+```go
 package main
 
 import (
@@ -84,16 +84,30 @@ in-memory mode. When Badger is running in in-memory mode, all the data is stored
 Reads and writes are much faster in in-memory mode, but all the data stored in Badger will be lost
 in case of a crash or close. To open badger in in-memory mode, set the `InMemory` option.
 
-```
+```go
 opt := badger.DefaultOptions("").WithInMemory(true)
 ```
 
+### Encryption Mode
+
+If you enable encryption on Badger, you also need to set the index cache size.
+
+{{% notice "tip" %}}
+Having a cache improves the performance. Otherwise, your reads would be very slow while encryption is enabled.
+{{% /notice %}}
+
+For example, to set a `100 Mb` cache:
+
+```go
+opts.IndexCache = 100 << 20 // 100 mb or some other size based on the amount of data
+```
+
 ## Transactions
 
 ### Read-only transactions
 To start a read-only transaction, you can use the `DB.View()` method:
 
-```
+```go
 err := db.View(func(txn *badger.Txn) error {
   // Your code here…
   return nil
@@ -108,7 +122,7 @@ seen by calls made within the closure.
 ### Read-write transactions
 To start a read-write transaction, you can use the `DB.Update()` method:
 
-```
+```go
 err := db.Update(func(txn *badger.Txn) error {
   // Your code here…
   return nil
@@ -129,7 +143,7 @@ the transaction exceeds a certain limit. In that case, it is best to commit the
 transaction and start a new transaction immediately. Here is an example (we are
 not checking for errors in some places for simplicity):
 
-```
+```go
 updates := make(map[string]string)
 txn := db.NewTransaction(true)
 for k,v := range updates {
@@ -161,7 +175,7 @@ sufficient for read-write transaction. However, if your code doesn’t call
 then please make sure you call `Txn.Discard()` in a `defer` block. Refer to the
 code below.
 
-```
+```go
 // Start a writable transaction.
 txn := db.NewTransaction(true)
 defer txn.Discard()
@@ -193,7 +207,7 @@ durable until the callback has been invoked with a `nil` error value.
 ## Using key/value pairs
 To save a key/value pair, use the `Txn.Set()` method:
 
-```
+```go
 err := db.Update(func(txn *badger.Txn) error {
   err := txn.Set([]byte("answer"), []byte("42"))
   return err
@@ -204,7 +218,7 @@ Key/Value pair can also be saved by first creating `Entry`, then setting this
 `Entry` using `Txn.SetEntry()`. `Entry` also exposes methods to set properties
 on it.
 
-```
+```go
 err := db.Update(func(txn *badger.Txn) error {
   e := badger.NewEntry([]byte("answer"), []byte("42"))
   err := txn.SetEntry(e)
@@ -215,7 +229,7 @@ err := db.Update(func(txn *badger.Txn) error {
 This will set the value of the `"answer"` key to `"42"`. To retrieve this
 value, we can use the `Txn.Get()` method:
 
-```
+```go
 err := db.View(func(txn *badger.Txn) error {
   item, err := txn.Get([]byte("answer"))
   handle(err)
@@ -272,7 +286,7 @@ invocations. Setting a bandwidth too low would do more disk writes, setting it
 too high would result in wasted integers if Badger is closed or crashes.
 To avoid wasted integers, call `Release` before closing Badger.
 
-```
+```go
 seq, err := db.GetSequence(key, 1000)
 defer seq.Release()
 for {
@@ -287,7 +301,7 @@ _merged_ with it. It returns a new value which is the result of the _merge_
 operation. All values are specified in byte arrays. For e.g., here is a merge
 function (`add`) which appends a  `[]byte` value to an existing `[]byte` value.
 
-```
+```go
 // Merge function to append one byte slice to another
 func add(originalValue, newValue []byte) []byte {
   return append(originalValue, newValue...)
@@ -302,7 +316,7 @@ method.
 `MergeOperator.Get()` method can be used to retrieve the cumulative value of the key
 associated with the merge operation.
 
-```
+```go
 key := []byte("merge")
 
 m := db.GetMergeOperator(key, add, 200*time.Millisecond)
@@ -317,7 +331,7 @@ res, _ := m.Get() // res should have value ABC encoded
 
 Example: Merge operator which increments a counter
 
-```
+```go
 func uint64ToBytes(i uint64) []byte {
   var buf [8]byte
   binary.BigEndian.PutUint64(buf[:], i)
@@ -353,7 +367,7 @@ elapsed, the key will no longer be retrievable and will be eligible for garbage
 collection. A TTL can be set as a `time.Duration` value using the `Entry.WithTTL()`
 and `Txn.SetEntry()` API methods.
 
-```
+```go
 err := db.Update(func(txn *badger.Txn) error {
   e := badger.NewEntry([]byte("answer"), []byte("42")).WithTTL(time.Hour)
   err := txn.SetEntry(e)
@@ -366,7 +380,7 @@ is represented by a single byte. It can be used to set certain bits along
 with the key to aid in interpreting or decoding the key-value pair. User
 metadata can be set using `Entry.WithMeta()` and `Txn.SetEntry()` API methods.
 
-```
+```go
 err := db.Update(func(txn *badger.Txn) error {
   e := badger.NewEntry([]byte("answer"), []byte("42")).WithMeta(byte(1))
   err := txn.SetEntry(e)
@@ -377,7 +391,7 @@ err := db.Update(func(txn *badger.Txn) error {
 `Entry` APIs can be used to add the user metadata and TTL for same key. This `Entry`
 then can be set using `Txn.SetEntry()`.
 
-```
+```go
 err := db.Update(func(txn *badger.Txn) error {
   e := badger.NewEntry([]byte("answer"), []byte("42")).WithMeta(byte(1)).WithTTL(time.Hour)
   err := txn.SetEntry(e)
@@ -391,7 +405,7 @@ To iterate over keys, we can use an `Iterator`, which can be obtained using the
 order.
 
 
-```
+```go
 err := db.View(func(txn *badger.Txn) error {
   opts := badger.DefaultIteratorOptions
   opts.PrefetchSize = 10
@@ -424,7 +438,7 @@ values altogether. See section below on key-only iteration.
 ### Prefix scans
 To iterate over a key prefix, you can combine `Seek()` and `ValidForPrefix()`:
 
-```
+```go
 db.View(func(txn *badger.Txn) error {
   it := txn.NewIterator(badger.DefaultIteratorOptions)
   defer it.Close()
@@ -452,7 +466,7 @@ enable key-only iteration, you need to set the `IteratorOptions.PrefetchValues`
 field to `false`. This can also be used to do sparse reads for selected keys
 during an iteration, by calling `item.Value()` only when required.
 
-```
+```go
 err := db.View(func(txn *badger.Txn) error {
   opts := badger.DefaultIteratorOptions
   opts.PrefetchValues = false
@@ -491,7 +505,7 @@ provide full and incremental backups quickly.  Dgraph is a heavy user of this
 framework.  In fact, this framework was developed and used within Dgraph, before
 getting ported over to Badger.
 
-```
+```go
 stream := db.NewStream()
 // db.NewStreamAt(readTs) for managed mode.
 
@@ -546,7 +560,7 @@ the following method, which can be invoked at an appropriate time:
   one log file. As an optimization, you could also immediately re-run it whenever
   it returns nil error (indicating a successful value log GC), as shown below.
 
-	```
+	```go
 	ticker := time.NewTicker(5 * time.Minute)
 	defer ticker.Stop()
 	for range ticker.C {
@@ -571,13 +585,13 @@ in your PATH to use this tool.
 The command below will create a version-agnostic backup of the database, to a
 file `badger.bak` in the current working directory
 
-```
+```sh
 badger backup --dir <path/to/badgerdb>
 ```
 
 To restore `badger.bak` in the current working directory to a new database:
 
-```
+```sh
 badger restore --dir <path/to/badgerdb>
 ```
 
@@ -587,7 +601,7 @@ If you have a Badger database that was created using v0.8 (or below), you can
 use the `badger_backup` tool provided in v0.8.1, and then restore it using the
 command above to upgrade your database to work with the latest version.
 
-```
+```sh
 badger_backup --dir <path/to/badgerdb> --backup-file badger.bak
 ```
 
@@ -597,7 +611,7 @@ latest value log which is append-only. So, rsync can be used as rudimentary way
 to perform a backup. In the following script, we repeat rsync to ensure that the
 LSM tree remains consistent with the MANIFEST file while doing a full backup.
 
-```
+```sh
 #!/bin/bash
 set -o history
 set -o histexpand
@@ -612,14 +626,10 @@ Badger's memory usage can be managed by tweaking several options available in
 the `Options` struct that is passed in when opening the database using
 `DB.Open`.
 
-- `Options.ValueLogLoadingMode` can be set to `options.FileIO` (instead of the
-  default `options.MemoryMap`) to avoid memory-mapping log files. This can be
-  useful in environments with low RAM.
 - Number of memtables (`Options.NumMemtables`)
   - If you modify `Options.NumMemtables`, also adjust `Options.NumLevelZeroTables` and
    `Options.NumLevelZeroTablesStall` accordingly.
 - Number of concurrent compactions (`Options.NumCompactors`)
-- Mode in which LSM tree is loaded (`Options.TableLoadingMode`)
 - Size of table (`Options.MaxTableSize`)
 - Size of value log file (`Options.ValueLogFileSize`)
 

docs/content/projects-using-badger/index.md

@@ -15,7 +15,6 @@ Below is a list of known projects that use Badger:
 * [Sandglass](https://github.com/celrenheit/sandglass) - distributed, horizontally scalable, persistent, time sorted message queue.
 * [TalariaDB](https://github.com/grab/talaria) - Grab's Distributed, low latency time-series database.
 * [Sloop](https://github.com/salesforce/sloop) - Salesforce's Kubernetes History Visualization Project.
-* [Immudb](https://github.com/codenotary/immudb) - Lightweight, high-speed immutable database for systems and applications.
 * [Usenet Express](https://usenetexpress.com/) - Serving over 300TB of data with Badger.
 * [gorush](https://github.com/appleboy/gorush) - A push notification server written in Go.
 * [0-stor](https://github.com/zero-os/0-stor) - Single device object store.
@@ -38,5 +37,20 @@ Below is a list of known projects that use Badger:
 * [Cete](https://github.com/mosuka/cete) - Simple and highly available distributed key-value store built on Badger. Makes it easy bringing up a cluster of Badger with Raft consensus algorithm by hashicorp/raft. 
 * [Volument](https://volument.com/) - A new take on website analytics backed by Badger.
 * [KVdb](https://kvdb.io/) - Hosted key-value store and serverless platform built on top of Badger.
+* [Opacity](https://github.com/opacity/storage-node) - Backend implementation for the Opacity storage project
+* [Vephar](https://github.com/vaccovecrana/vephar) - A minimal key/value store using hashicorp-raft for cluster coordination and Badger for data storage.
+* [gowarcserver](https://github.com/nlnwa/gowarcserver) - Open-source server for warc files. Can be used in conjunction with pywb 
+* [flow-go](https://github.com/onflow/flow-go) - A fast, secure, and developer-friendly blockchain built to support the next generation of games, apps and the digital assets that power them.
+* [Wrgl](https://www.wrgl.co) - A data version control system that works like Git but specialized to store and diff CSV.
+* [chaindb](https://github.com/ChainSafe/chaindb) - A blockchain storage layer used by [Gossamer](https://chainsafe.github.io/gossamer/), a Go client for the [Polkadot Network](https://polkadot.network/).
+* [m2](https://github.com/qichengzx/m2) - A simple http key/value store based on the raft protocol.
+* [Xuperchain](https://github.com/xuperchain/xupercore) - A highly flexible blockchain architecture with great transaction performance.
+* [Pyroscope](https://github.com/pyroscope-io/pyroscope) - Open source confinuous profiling platform built with BadgerDB
+* [Souin](https://github.com/darkweak/Souin) - A RFC compliant HTTP cache with lot of other features based on Badger for the storage. Compatible with all existing reverse-proxies.
+* [Veri](https://github.com/bgokden/veri) - A distributed feature store optimized for Search and Recommendation tasks.
+* [bIter](https://github.com/MikkelHJuul/bIter) - A library and Iterator interface for working with the `badger.Iterator`, simplifying from-to, and prefix mechanics.
+* [ld](https://github.com/MikkelHJuul/ld) - (Lean Database) A very simple gRPC-only key-value database, exposing BadgerDB with key-range scanning semantics.
+* [smallstep/certificates](https://github.com/smallstep/certificates) - Step-ca is an online certificate authority for secure, automated certificate management.
+* [vxdb](https://github.com/vitalvas/vxdb) - Simple schema-less Key-Value NoSQL database with simplest API interface.
 
 If you are using Badger in a project please send a pull request to add it to the list.