diff --git a/src/understand-tidb/cbo.md b/src/understand-tidb/cbo.md index af6a049d..2a5afc1e 100644 --- a/src/understand-tidb/cbo.md +++ b/src/understand-tidb/cbo.md @@ -162,4 +162,4 @@ But we still have some problems that the current framework can not handle. For e * For optimization rules that may not always get better, we currently do not support the selection of rules through cost. The search space of the optimizer is restricted. * ... -We want to solve these problems, so we are planning to develop a new optimizer framework the `Cascades Planner`. But it still works in progress. You can learn more by reading [`planner/cascades`](https://github.com/pingcap/tidb/tree/master/planner/cascades), [`planner/memo`](https://github.com/pingcap/tidb/tree/master/planner/memo) and [`planner/implementation`](https://github.com/pingcap/tidb/tree/master/planner/implementation) under the [`planner`](https://github.com/pingcap/tidb/tree/master/planner) module. \ No newline at end of file +We want to solve these problems, so we are planning to develop a new optimizer framework the `Cascades Planner`. But it still works in progress. You can learn more by reading [`planner/cascades`](https://github.com/pingcap/tidb/tree/master/pkg/planner/cascades), [`planner/memo`](https://github.com/pingcap/tidb/tree/master/pkg/planner/memo) and [`planner/implementation`](https://github.com/pingcap/tidb/tree/master/pkg/planner/implementation) under the [`planner`](https://github.com/pingcap/tidb/tree/master/pkg/planner) module. diff --git a/src/understand-tidb/introduction.md b/src/understand-tidb/introduction.md index f7bb0d76..95733c04 100644 --- a/src/understand-tidb/introduction.md +++ b/src/understand-tidb/introduction.md @@ -36,42 +36,42 @@ In addition to the code, there are many test cases, which can be found with suff TiDB has a number of modules. Table below is an overview that shows what each module does, and if you want to see the code for the relevant function, you can find the corresponding module directly. -| Package | Description | -| :------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [pingcap/tidb/bindinfo](https://github.com/pingcap/tidb/tree/master/bindinfo) | Handles all global sql bind operations, and caches the sql bind info from storage. | -| [pingcap/tidb/config](https://github.com/pingcap/tidb/tree/master/config) | The configuration definition. | -| [pingcap/tidb/ddl](https://github.com/pingcap/tidb/tree/master/ddl) | The execution logic of data definition language (DDL). | -| [pingcap/tidb/distsql](https://github.com/pingcap/tidb/tree/master/distsql) | The abstraction of the distributed computing interfaces to isolate the logic between the executor and the TiKV client | -| [pingcap/tidb/domain](https://github.com/pingcap/tidb/tree/master/domain) | The abstraction of a storage space in which databases and tables can be created. Like namespace, databases with the same name can exist in different domains. In most cases, a single TiDB instance only creates one Domain instance with details about the information schema and statistics. | -| [pingcap/tidb/errno](https://github.com/pingcap/tidb/tree/master/errno) | The definition of MySQL error code, error message, and error summary. | -| [pingcap/tidb/executor](https://github.com/pingcap/tidb/tree/master/executor) | The operator related code that contains the execution logic of most statements. | -| [pingcap/tidb/expression](https://github.com/pingcap/tidb/tree/master/expression) | The expression-related code that contains various operators and built-in functions. | -| [pingcap/tidb/infoschema](https://github.com/pingcap/tidb/tree/master/infoschema) | The metadata management module for SQL statements; accessed when all the operations on the information schema are executed. | -| [pingcap/tidb/kv](https://github.com/pingcap/tidb/tree/master/kv) | The Key-Value engine interface and some public methods; the interfaces defined in this package need to be implemented by the storage engine which is going to adapt TiDB SQL layer. | -| [pingcap/tidb/lock](https://github.com/pingcap/tidb/tree/master/lock) | The implementation of LOCK/UNLOCK TABLES. | -| [pingcap/tidb/meta](https://github.com/pingcap/tidb/tree/master/meta) | Manages the SQL metadata in the storage engine through the features of the structure package; infoschema and DDL use this module to access or modify the SQL metadata . | -| [pingcap/tidb/meta/autoid](https://github.com/pingcap/tidb/tree/master/meta/autoid) | A module to generate the globally unique monotonically incremental IDs for each table, as well as the database ID and table ID. | -| [pingcap/tidb/metrics](https://github.com/pingcap/tidb/tree/master/metrics) | Store the metrics information of all modules. | -| [pingcap/tidb/owner](https://github.com/pingcap/tidb/tree/master/owner) | Some tasks in the TiDB cluster can be executed by only one instance, such as the asynchronous schema change. This owner module is used to coordinate and generate a task executor among multiple TiDB servers. Each task has its own executor. | -| [pingcap/tidb/parser](https://github.com/pingcap/tidb/tree/master/parser) | A MySQL compatible SQL parser used by TiDB, also contains the data structure definition of abstract syntax tree (AST) and other metadata. | -| [pingcap/tidb/planner](https://github.com/pingcap/tidb/tree/master/planner) | Queries optimization related code. | -| [pingcap/tidb/plugin](https://github.com/pingcap/tidb/tree/master/plugin) | The plugin framework of TiDB. | -| [pingcap/tidb/privilege](https://github.com/pingcap/tidb/tree/master/privilege) | The management interface of user privileges. | -| [pingcap/tidb/server](https://github.com/pingcap/tidb/tree/master/server) | Code of the MySQL protocol and connection management. | -| [pingcap/tidb/session](https://github.com/pingcap/tidb/tree/master/session) | Code of session management. | -| [pingcap/tidb/sessionctx/binloginfo](https://github.com/pingcap/tidb/tree/master/sessionctx/binloginfo) | Output binlog information. | -| [pingcap/tidb/sessionctx/stmtctx](https://github.com/pingcap/tidb/tree/master/sessionctx/stmtctx) | Necessary information for the statement of a session during runtime. | -| [pingcap/tidb/sessionctx/variable](https://github.com/pingcap/tidb/tree/master/sessionctx/variable) | System variable related code. | -| [pingcap/tidb/statistics](https://github.com/pingcap/tidb/tree/master/statistics) | Code of table statistics. | -| [pingcap/tidb/store](https://github.com/pingcap/tidb/tree/master/store) | Storage engine drivers, wrapping Key-Value client to meet the requirements of TiDB. | -| [tikv/client-go](https://github.com/tikv/client-go) | The Go client of TiKV. | -| [pingcap/tidb/structure](https://github.com/pingcap/tidb/tree/master/structure) | The structured API defined on the Transactional Key-Value API, providing structures like List, Queue, and HashMap. | -| [pingcap/tidb/table](https://github.com/pingcap/tidb/tree/master/table) | The abstraction of Table in SQL. | -| [pingcap/tidb/tablecodec](https://github.com/pingcap/tidb/tree/master/tablecodec) | Encode and decode data from SQL to Key-Value. See the codec package for the specific encoding and decoding solution for each data type. | -| [pingcap/tidb/telemetry](https://github.com/pingcap/tidb/tree/master/telemetry) | Code of telemetry collect and report. | -| [pingcap/tidb/tidb-server](https://github.com/pingcap/tidb/tree/master/tidb-server) | The main method of the TiDB service. | -| [pingcap/tidb/types](https://github.com/pingcap/tidb/tree/master/types) | All the type related code, including the definition of and operation on types. | -| [pingcap/tidb/util](https://github.com/pingcap/tidb/tree/master/util) | Utilities. | +| Package | Description | +| :-------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [pingcap/tidb/pkg/bindinfo](https://github.com/pingcap/tidb/tree/master/pkg/bindinfo) | Handles all global sql bind operations, and caches the sql bind info from storage. | +| [pingcap/tidb/pkg/config](https://github.com/pingcap/tidb/tree/master/pkg/config) | The configuration definition. | +| [pingcap/tidb/pkg/ddl](https://github.com/pingcap/tidb/tree/master/pkg/ddl) | The execution logic of data definition language (DDL). | +| [pingcap/tidb/pkg/distsql](https://github.com/pingcap/tidb/tree/master/pkg/distsql) | The abstraction of the distributed computing interfaces to isolate the logic between the executor and the TiKV client | +| [pingcap/tidb/pkg/domain](https://github.com/pingcap/tidb/tree/master/pkg/domain) | The abstraction of a storage space in which databases and tables can be created. Like namespace, databases with the same name can exist in different domains. In most cases, a single TiDB instance only creates one Domain instance with details about the information schema and statistics. | +| [pingcap/tidb/pkg/errno](https://github.com/pingcap/tidb/tree/master/pkg/errno) | The definition of MySQL error code, error message, and error summary. | +| [pingcap/tidb/pkg/executor](https://github.com/pingcap/tidb/tree/master/pkg/executor) | The operator related code that contains the execution logic of most statements. | +| [pingcap/tidb/pkg/expression](https://github.com/pingcap/tidb/tree/master/pkg/expression) | The expression-related code that contains various operators and built-in functions. | +| [pingcap/tidb/pkg/infoschema](https://github.com/pingcap/tidb/tree/master/pkg/infoschema) | The metadata management module for SQL statements; accessed when all the operations on the information schema are executed. | +| [pingcap/tidb/pkg/kv](https://github.com/pingcap/tidb/tree/master/pkg/kv) | The Key-Value engine interface and some public methods; the interfaces defined in this package need to be implemented by the storage engine which is going to adapt TiDB SQL layer. | +| [pingcap/tidb/pkg/lock](https://github.com/pingcap/tidb/tree/master/pkg/lock) | The implementation of LOCK/UNLOCK TABLES. | +| [pingcap/tidb/pkg/meta](https://github.com/pingcap/tidb/tree/master/pkg/meta) | Manages the SQL metadata in the storage engine through the features of the structure package; infoschema and DDL use this module to access or modify the SQL metadata . | +| [pingcap/tidb/pkg/meta/autoid](https://github.com/pingcap/tidb/tree/master/pkg/meta/autoid) | A module to generate the globally unique monotonically incremental IDs for each table, as well as the database ID and table ID. | +| [pingcap/tidb/pkg/metrics](https://github.com/pingcap/tidb/tree/master/pkg/metrics) | Store the metrics information of all modules. | +| [pingcap/tidb/pkg/owner](https://github.com/pingcap/tidb/tree/master/pkg/owner) | Some tasks in the TiDB cluster can be executed by only one instance, such as the asynchronous schema change. This owner module is used to coordinate and generate a task executor among multiple TiDB servers. Each task has its own executor. | +| [pingcap/tidb/pkg/parser](https://github.com/pingcap/tidb/tree/master/pkg/parser) | A MySQL compatible SQL parser used by TiDB, also contains the data structure definition of abstract syntax tree (AST) and other metadata. | +| [pingcap/tidb/pkg/planner](https://github.com/pingcap/tidb/tree/master/pkg/planner) | Queries optimization related code. | +| [pingcap/tidb/pkg/plugin](https://github.com/pingcap/tidb/tree/master/pkg/plugin) | The plugin framework of TiDB. | +| [pingcap/tidb/pkg/privilege](https://github.com/pingcap/tidb/tree/master/pkg/privilege) | The management interface of user privileges. | +| [pingcap/tidb/pkg/server](https://github.com/pingcap/tidb/tree/master/pkg/server) | Code of the MySQL protocol and connection management. | +| [pingcap/tidb/pkg/session](https://github.com/pingcap/tidb/tree/master/pkg/session) | Code of session management. | +| [pingcap/tidb/pkg/sessionctx/binloginfo](https://github.com/pingcap/tidb/tree/master/pkg/sessionctx/binloginfo) | Output binlog information. | +| [pingcap/tidb/pkg/sessionctx/stmtctx](https://github.com/pingcap/tidb/tree/master/pkg/sessionctx/stmtctx) | Necessary information for the statement of a session during runtime. | +| [pingcap/tidb/pkg/sessionctx/variable](https://github.com/pingcap/tidb/tree/master/pkg/sessionctx/variable) | System variable related code. | +| [pingcap/tidb/pkg/statistics](https://github.com/pingcap/tidb/tree/master/pkg/statistics) | Code of table statistics. | +| [pingcap/tidb/pkg/store](https://github.com/pingcap/tidb/tree/master/pkg/store) | Storage engine drivers, wrapping Key-Value client to meet the requirements of TiDB. | +| [tikv/client-go](https://github.com/tikv/client-go) | The Go client of TiKV. | +| [pingcap/tidb/pkg/structure](https://github.com/pingcap/tidb/tree/master/pkg/structure) | The structured API defined on the Transactional Key-Value API, providing structures like List, Queue, and HashMap. | +| [pingcap/tidb/pkg/table](https://github.com/pingcap/tidb/tree/master/pkg/table) | The abstraction of Table in SQL. | +| [pingcap/tidb/pkg/tablecodec](https://github.com/pingcap/tidb/tree/master/pkg/tablecodec) | Encode and decode data from SQL to Key-Value. See the codec package for the specific encoding and decoding solution for each data type. | +| [pingcap/tidb/pkg/telemetry](https://github.com/pingcap/tidb/tree/master/pkg/telemetry) | Code of telemetry collect and report. | +| [pingcap/tidb/cmd/tidb-server](https://github.com/pingcap/tidb/tree/master/cmd/tidb-server) | The main method of the TiDB service. | +| [pingcap/tidb/pkg/types](https://github.com/pingcap/tidb/tree/master/pkg/types) | All the type related code, including the definition of and operation on types. | +| [pingcap/tidb/pkg/util](https://github.com/pingcap/tidb/tree/master/pkg/util) | Utilities. | At a glance, TiDB has 80 packages, which might let you feel overwhelmed, but not all of them are important, and some features only involve a small number of packages, so where to start to look at the source code depends on the purpose of looking at the source code. diff --git a/src/understand-tidb/parser.md b/src/understand-tidb/parser.md index a195ef31..417f6354 100644 --- a/src/understand-tidb/parser.md +++ b/src/understand-tidb/parser.md @@ -2,7 +2,7 @@ Parser is responsible for interpreting a SQL string into an abstract syntax tree (AST), which is more structural and easier to process. AST can be used for preprocessing, syntactic analysis, and so on. -The code lives in the [pingcap/tidb repo, parser directory](https://github.com/pingcap/tidb/tree/master/parser). +The code lives in the [pingcap/tidb repo, parser directory](https://github.com/pingcap/tidb/tree/master/pkg/parser). ## Understand Parser diff --git a/src/understand-tidb/planner.md b/src/understand-tidb/planner.md index 58f850a4..72e42853 100644 --- a/src/understand-tidb/planner.md +++ b/src/understand-tidb/planner.md @@ -4,14 +4,14 @@ The `planner` package contains most of the codes related to SQL optimization. Th ## Package Structure -| Package | Description | -| :--------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------| -| [tidb/planner/cascades](https://github.com/pingcap/tidb/tree/master/planner/cascades) | The next generation Cascades model planner, which is under development and disabled by default | -| [tidb/planner/core](https://github.com/pingcap/tidb/tree/master/planner/core) | The core logic of the currently used System R model planner. The Cascades model planner also calls utility functions in this package | | -| [tidb/planner/implementation](https://github.com/pingcap/tidb/tree/master/planner/implementation) | Physical implementations for the operators in Cascades planner | -| [tidb/planner/memo](https://github.com/pingcap/tidb/tree/master/planner/memo) | Intermediate results for the searching procedure of Cascades planner | -| [tidb/planner/property](https://github.com/pingcap/tidb/tree/master/planner/property) | Properties about the output of operators, including schema, stats, order property, partition property, etc | -| [tidb/planner/util](https://github.com/pingcap/tidb/tree/master/planner/util) | Common utility functions / structures shared by the two planners | +| Package | Description | +| :----------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------| +| [tidb/pkg/planner/cascades](https://github.com/pingcap/tidb/tree/master/pkg/planner/cascades) | The next generation Cascades model planner, which is under development and disabled by default | +| [tidb/pkg/planner/core](https://github.com/pingcap/tidb/tree/master/pkg/planner/core) | The core logic of the currently used System R model planner. The Cascades model planner also calls utility functions in this package | | +| [tidb/pkg/planner/implementation](https://github.com/pingcap/tidb/tree/master/pkg/planner/implementation) | Physical implementations for the operators in Cascades planner | +| [tidb/pkg/planner/memo](https://github.com/pingcap/tidb/tree/master/pkg/planner/memo) | Intermediate results for the searching procedure of Cascades planner | +| [tidb/pkg/planner/property](https://github.com/pingcap/tidb/tree/master/pkg/planner/property) | Properties about the output of operators, including schema, stats, order property, partition property, etc | +| [tidb/pkg/planner/util](https://github.com/pingcap/tidb/tree/master/pkg/planner/util) | Common utility functions / structures shared by the two planners | We can see that, TiDB has two planners, one is of System R model, which is defaultly used, and the other is of Cascades model, which is still under development. The unified entry function of planner module is `Optimize()`, before diving into either of the two planners, it would firstly check if there is any intervention for the planner from the "SQL Plan Management" module, if yes, the AST of the query would be modified before going through the optimization procedures. "SQL Plan Management" module is beyond the scope of this article, and it would be introduced in the [SQL Plan Management](sql-plan-management.md) section. diff --git a/src/understand-tidb/plugin.md b/src/understand-tidb/plugin.md index 65035529..8f8c7ad6 100644 --- a/src/understand-tidb/plugin.md +++ b/src/understand-tidb/plugin.md @@ -2,7 +2,7 @@ The plugin API allows TiDB to be extended with new features such as audit logging or IP allow/deny listing. -Sample code is provided for a basic audit logging plugin at [`plugin/conn_ip_example/`](https://github.com/pingcap/tidb/tree/master/plugin/conn_ip_example). For an example on compiling TiDB and this plugin: +Sample code is provided for a basic audit logging plugin at [`plugin/conn_ip_example/`](https://github.com/pingcap/tidb/tree/master/pkg/plugin/conn_ip_example). For an example on compiling TiDB and this plugin: ```bash plugin="conn_ip_example" diff --git a/src/understand-tidb/the-lifecycle-of-a-statement.md b/src/understand-tidb/the-lifecycle-of-a-statement.md index 8a590518..f3697a48 100644 --- a/src/understand-tidb/the-lifecycle-of-a-statement.md +++ b/src/understand-tidb/the-lifecycle-of-a-statement.md @@ -118,7 +118,7 @@ func (cc *clientConn) writeChunks(ctx context.Context, rs ResultSet ...) (bool, ## Statement string to Abstract Syntax Tree -In [handleQuery](https://github.com/pingcap/tidb/blob/30cf15a59db11c34ffe05fc926152a43327eaa61/server/conn.go#L1633) the statement string is [parsed](https://github.com/pingcap/tidb/blob/9f68c8e92a994e4790bfd9e567e5ad86c8daa861/parser/yy_parser.go#L172) by [the parser](https://github.com/pingcap/tidb/tree/master/parser), that is a MySQL compatible parser parsing statements and returns an Abstract Syntax Tree (AST) representing the statement. See more in the [parser section](parser.md) +In [handleQuery](https://github.com/pingcap/tidb/blob/30cf15a59db11c34ffe05fc926152a43327eaa61/server/conn.go#L1633) the statement string is [parsed](https://github.com/pingcap/tidb/blob/9f68c8e92a994e4790bfd9e567e5ad86c8daa861/parser/yy_parser.go#L172) by [the parser](https://github.com/pingcap/tidb/tree/master/pkg/parser), that is a MySQL compatible parser parsing statements and returns an Abstract Syntax Tree (AST) representing the statement. See more in the [parser section](parser.md) Example of Abstract Syntax tree, the fragment of a `WHERE` clause `` `id` > 1 AND `value` = 'Second' `` looks like: ```golang