Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Issue] Remove limit in IdListBuilder #39621

Open
3 of 5 tasks
m2-assistant bot opened this issue Feb 11, 2025 · 0 comments · May be fixed by #39603
Open
3 of 5 tasks

[Issue] Remove limit in IdListBuilder #39621

m2-assistant bot opened this issue Feb 11, 2025 · 0 comments · May be fixed by #39603
Labels
Reported on 2.4.x Indicates original Magento version for the Issue report.

Comments

@m2-assistant
Copy link

m2-assistant bot commented Feb 11, 2025

This issue is automatically created based on existing pull request: #39603: Remove limit in IdListBuilder

There should be no limit on the IdListBuilder since it is chunked in: vendor/magento/module-sales/Model/ResourceModel/Grid.php:

    public function refreshBySchedule()
    {
        $lastUpdatedAt = null;
        $notSyncedIds = $this->notSyncedDataProvider->getIds($this->mainTableName, $this->gridTableName);
        foreach (array_chunk($notSyncedIds, self::BATCH_SIZE) as $bunch) {
            $select = $this->getGridOriginSelect()->where($this->mainTableName . '.entity_id IN (?)', $bunch);

Description (*)

The limit in the select is removed so it will fetch all results and later it will be chuncked.

Related Pull Requests

Manual testing scenarios (*)

  1. Make sure you have more than 100 orders
  2. Make sure your sales_order_grid is also filled with this orders
  3. Truncate sales_order_grid
  4. Make sure async indexing of grids is true
  5. run magerun2 sys:cron:run sales_grid_order_async_insert

vendor/magento/module-sales/Model/ResourceModel/Grid.php

    public function refreshBySchedule()
    {
        $lastUpdatedAt = null;
        $notSyncedIds = $this->notSyncedDataProvider->getIds($this->mainTableName, $this->gridTableName);
        foreach (array_chunk($notSyncedIds, self::BATCH_SIZE) as $bunch) {
            $select = $this->getGridOriginSelect()->where($this->mainTableName . '.entity_id IN (?)', $bunch);

The code above will be run and fetch $notSyncedIds and chunck it in batches. The provider used is: vendor/magento/module-sales/Model/ResourceModel/Provider/UpdatedIdListProvider.php and does this:

    public function getIds($mainTableName, $gridTableName)
    {
        $mainTableName = $this->resourceConnection->getTableName($mainTableName);
        $gridTableName = $this->resourceConnection->getTableName($gridTableName);
        $select = $this->idListQueryBuilder->build($mainTableName, $gridTableName);
        return $this->getConnection()->fetchAll($select, [], \Zend_Db::FETCH_COLUMN);

and builds a select with the idListQueryBuilder:

public function build(string $mainTableName, string $gridTableName): Select
    {
        $select = $this->getConnection()->select()
            ->from(['main_table' => $mainTableName], ['main_table.entity_id'])
            ->joinLeft(
                ['grid_table' => $this->resourceConnection->getTableName($gridTableName)],
                'main_table.entity_id = grid_table.entity_id',
                []
            );

        $select->where('grid_table.entity_id IS NULL');
        $select->limit(Grid::BATCH_SIZE); // should be removed
        foreach ($this->additionalGridTables as $table) {
            $select->joinLeft(
                [$table => $table],
                sprintf(
                    '%s.%s = %s.%s',
                    'main_table',
                    'entity_id',
                    $table,
                    'entity_id'
                ),
                []
            )
                ->where($table . '.entity_id IS NULL');
        }
        return $select;
    }

In here there is limit: $select->limit(Grid::BATCH_SIZE); the same as the chunk batch size in refreshBySchedule. This looks like a bug to me, because you will never get more items than 100. Now this will probably only occur when the sales_order_grid is truncated or more than 100 orders are placed between the cron execution.

Questions or comments

Eventually the grid will be filled, because the cron is run every minute. But it takes a lot of time if you have a lot of orders.

Contribution checklist (*)

  • Pull request has a meaningful description of its purpose
  • All commits are accompanied by meaningful commit messages
  • All new or changed code is covered with unit/integration tests (if applicable)
  • README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
  • All automated tests passed successfully (all builds are green)
@m2-assistant m2-assistant bot linked a pull request Feb 11, 2025 that will close this issue
Open
6 tasks
@engcom-Bravo engcom-Bravo added the Reported on 2.4.x Indicates original Magento version for the Issue report. label Feb 11, 2025
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
Reported on 2.4.x Indicates original Magento version for the Issue report.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove limit in IdListBuilder sanderjongsma/magento2
1 participant
@engcom-Bravo