v0.8 Breaking changes

[KwilLuke]

I am keeping a list here of things that we need to note as breaking for the next release. Please add to this as you see fit:

• Old parsers, JSON schemas are broken. v0.8 added the procedures field and changed the properties on the Extension struct.

[jchappelow]

Let's also state the scope of the breaking change. In particular, is it just tooling that needs to update, or would kwild handle a transaction differently (consensus failure), on account of any changes inc. serialization, execution, etc.

[jchappelow]

Here's a start on some material for release notes:

Breaking changes in v0.8

Consensus and Upgrading

This version of kwild is not compatible with data or networks from previous releases. There are a number of changes that prevent:

• a v0.8 node from running in a network with v0.7 nodes
• a v0.8 node from being installed with existing data (cometbft or postgresql databases) from a v0.7 installation

@charithabandi perhaps something can be said here about possible migration paths?

The core Go module

The core module, which provides the SDK and client for Kwil, will be tagged as core/v0.2.0 for the v0.8 release of Kwil DB.

core/crypto/auth

The Signature structure has the same Go field names and types, but the JSON tags have changed:

• signature_bytes => sig
• signature_type => type

These fields are less redundant now when considering how a signature would be in a larger JSON object. For example in a transaction, the JSON is less repetitive: {"signature": {"sig": "1234...", "type": "secp256k1_ep"}.

core/types and core/types/transactions

Many of the types in the core/types/transactions package that defined the structure of a Kuneiform schema are now mirrored in core/types. The types in core/types are used on the method signatures of core/client and are also JSON-tagged. The types in core/types/transactions are primarily intended for serialization that is handled by the core/client functions to communicate with a Kwil node. To convert from a type in core/types/transactions to a corresponding type in core/types, use the ToTypes method of the given type.

@brennanjl could probably provide more clarity on the above

core/types

The Schema structure and all of it's composing types are now defined in core/types/schema.go with JSON tags and methods to clean and canonicalize their field values.. The types are mirrored in core/types/transactions, primarily for RLP serialization purposes.

There are many new types and fields to support procedures and strongly typed values.

core/types/transactions

NOTE: Most users and developers do not need to be concerned with the changes to this packages as the core/client and core/rpc/... packages handle their use.

The CallMessage and CallMessageBody types are now JSON-tagged for consistent snake case marshaling. The CallMessage.Sender field is changed from []byte to types.HexBytes so that hexadecimal JSON (un)marshalling is used instead of base64. This decision was made to reflect that a "sender" is typically identified by an hexadecimal Ethereum address.

The Transaction type is also now JSON-tagged, and the Sender field is also now type.HexBytes for the same reasons as for CallMessage. TransactionBody is also JSON-tagged, but custom MarshalJSON and UnmarshalJSON methods are defined so that the Fee *big.Int field is represented as a string in JSON instead of a number.

The PayloadTypeExecuteAction payload type is renamed to PayloadTypeExecute to reflect its use with both actions and procedures.

The helper function UnmarshalPayload is added to assist in unmarshalling a payload from a []byte given it's PayloadType.

The following new TxCode values are now defined:

	// engine-related error codes
	CodeInvalidSchema  TxCode = 100
	CodeDatasetMissing TxCode = 110
	CodeDatasetExists  TxCode = 120

The TcTxQueryResponse.Tx field is now a *Transaction instead of a Transaction. The change to a pointer type is to facilitate returning null as the JSON value for unconfirmed transactions. (See RPC server changes.)

core/types/transactions.Schema Structure

The Schema structure and most of the composing field types have been redefined to support the addition of procedures, foreign procedures, and strongly typed arguments and results.

There are new types and fields for Procedures and ForeignProcedures.

The DataType struct is added:

// DataType is a data type.
// It includes both built-in types and user-defined types.
type DataType struct {
	// Name is the name of the type.
	Name string
	// IsArray is true if the type is an array.
	IsArray bool
	// Metadata is the metadata of the type.
	Metadata *[2]uint16 `rlp:"optional"`
}

The Column.Type field is changed from string to *DataType.

The Column.Attributes []*Attribute field and the Attributes type are added:

// Attribute is a column attribute.
// These are constraints and default values.
type Attribute struct {
	Type  string
	Value string
}

The Actions structure is entirely redefined, with the entire body of the action encapsulated in a Body string field instead of Statements []string.

See core/types/transactions/payload_schema.go for all of the new type definitions.

core/types/validation

The new package core/types/validation exposes the ValidateIdentifier(name string) error helper.

A number of global limits are defined as const values in restrictions.go.

core/types/decimal

This new package defines the Decimal type, which provides arbitrary precision numbers.

core/types/client

The Client interface has the following changes:

• The Call and Execute methods are added.
• The CallAction method is deprecated in favor of Call.
• The ExecuteAction action is deprecated in favor of Execute.
• The DeployDatabase method's payload parameter is now a *types.Schema. It was previously a *transactions.Schema.
• The GetSchema method returns a *types.Schema instead of a *transactions.Schema.

core/client

Detailed documentation was added in a new README.md file.

An example app using core/client.Client was added in core/client/example.

The Client type now uses the new JSON-RPC server instead of the HTTP gateway to the gRPC server. This means it will only work with v0.8 servers.

The client methods satisfy the updated core/types/client.Client interface (see the previous section). In particular, parameters and returns are now from the core/types package, and the ExecuteAction and CallAction methods are deprecated in favor of Execute and Call, respectively.

core/rpc/client/...

NOTE: The packages under core/rpc/... are relatively low level and most users only need to be concerned with the core/client and core/types packages.

Support for the new JSON-RPC server is added with the new core/rpc/client.JSONRPCClient type. This type provides the infrastructure for the "user", "admin", and "kgw" RPC services implemented in the core/rpc/client/user/jsonrpc, core/rpc/client/admin/jsonrpc, and core/rpc/client/gateway/jsonrpc packages.

The new ErrInvalidRequest error kind is returned when an RPC error is the result of a bad request, such as invalid arguments.

core/rpc/json

This package defines the common JSON-RPC types as well as the method names and types that are specific to the "user", "admin", and "function" RPC services.

core/rpc/http

This entire package is DEPRECATED with the switch to JSON-RPC.

The main kwil-db Go module

The main module (github.com/kwilteam/kwil-db) pertains to the command line applications:

• kwild, the Kwil DB node
• kwil-cli, the user client
• kwil-admin, the node administration tool

It will be tagged as v0.8.0.

JSON-RPC server

A JSON-RPC server is added. The gRPC server is no longer exposed, and it's HTTP gateway is DEPRECATED:

• jsonrpc_listen_addr is the new setting for the JSON-RPC server, which by default listens on port 8484.
• http_listen_addr is deprecated.
• grpc_listen_addr is no longer recognized.

The admin service now listens on "/tmp/kwild.socket" by default, where it was previously "unix:///tmp/kwil_admin.sock".,

The updated clients in the core module use the new JSON-RPC server.

Timeout Settings

The app.rpc_timeout setting imposes a timeout on RPC requests, after which the request is canceled and the connection closed. The default is 45 sec.

The app.db_read_timeout setting imposes a timeout on read-only DB transactions, such as those used when calling a view action or performing an ad-hoc SQL query. The default is 5 sec.

The rpc.broadcast_tx_timeout setting is added to allow a node to change the timeout for a broadcast request using the --sync flag with kwil-cli, which makes the request wait for the transaction to be included in a block before returning. The default is 10 sec.

Snapshot Settings

The app.snapshots setting is added to support snapshot creation. See the StateSyncConfig struct for details.

The app.snapshot_file setting is added to support starting a network with a state at genesis. This setting is used when app.app_hash is non-null.

Unconfirmed Transaction Queries

When handling a query for transaction status, if the transction is unconfirmed (not yet in a block) kwild will now return null for the "tx" field. kwil-cli is updated for this new behavior. Older versions of kwil-cli may have unexpected behavior with v0.8 servers when querying unconfirmed transactions.

...lots more server stuff

...

The parse Go module

Lotsa change to this module. Absorbed and merged the kuneiform, action, and sql parsers into a single unified parser.

This will be tagged as parse/v0.2.0 for the v0.8 release of Kwil DB.

[jchappelow]

Will update with this too: #688

The flag will remain, but deprecated. The change in config file path makes recognizing the grpc_url setting less useful, particularly since grpc was already deprecated.

[KwilLuke]

We probably need to delineate where we call out these breaking changes for the release. Changes in the "kwil-db Go Module"
should probably be called out when presenting v0.8 as a whole, while core and kwil-cli changes should be presented with their new versions... I'll put up a PR in a bit to demonstrate this idea.