Optimization of table metadata initialization

[tristaZero]

To make start up of ShardingSphere faster, we will do the following improvements:

• Only load one sharding table meta data into ShardingSphere.
• Instead of table structures, only table names from default data source will be loaded.

This issue is related with #1933.

[zhang98722]

i meet the same issue
its cost too much time to load too many default tables like 5000+
i just make the initializer skipped load the default table meta data by only create new
dont know what would happen
and the switch check.table.metadata.enabled cant solve this issue

    public TableMetaData load(final String logicTableName, final ShardingRule shardingRule) throws SQLException {
        if(!isCheckingMetaData){
            return new TableMetaData(new ArrayList<ColumnMetaData>());
        }
        List<TableMetaData> actualTableMetaDataList = load(getDataNodeGroups(logicTableName, shardingRule), shardingRule.getShardingDataSourceNames());
        checkUniformed(logicTableName, actualTableMetaDataList);
        return actualTableMetaDataList.iterator().next();
    }

@tristaZero

[wgy8283335]

@tristaZero @terrymanu. Does this issue still need a volunteer?

[cbbaaa1989]

can lazyload sharding table meta data?not load all table meta data on start

[cbbaaa1989]

@wgy8283335 did you have fixed this issue already?

[daiduan]

@tristaZero Hello, I am using shardingsphere3.1.0. In #1944, it has solved the problem of not loading the schema when loading all the tables, but it also does not consider the schema when loading all the columns of the table in TableMetaDataLoader. When using the oracle database, if the oracle user has multiple schema view permissions, it will get all the columns of the same name table, it will also make the system start slower.

private List<ColumnMetaData> getColumnMetaDataList(final Connection connection, final String catalog, final String actualTableName) throws SQLException {
        List<ColumnMetaData> result = new LinkedList<>();
        Collection<String> primaryKeys = getPrimaryKeys(connection, catalog, actualTableName);
        try (ResultSet resultSet = connection.getMetaData().getColumns(catalog, null, actualTableName, null)) {
            while (resultSet.next()) {
                String columnName = resultSet.getString("COLUMN_NAME");
                String columnType = resultSet.getString("TYPE_NAME");
                result.add(new ColumnMetaData(columnName, columnType, primaryKeys.contains(columnName)));
            }
        }
        return result;
    }

Is it possible to add connection.getSchema() as the second parameter in getColumns as well?

[terrymanu]

@daiduan

Is it possible to add connection.getSchema() as the second parameter in getColumns as well?

you may have a try

[daiduan]

@terrymanu Yes, I tried. And the system startup speed is getting faster. Now, in our project, we skip some code about init table metadata. I have not found any effect on the system running.