Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Implement sort index hints for postgres eventstore #2204

Merged
merged 1 commit into from
Jan 18, 2022

Conversation

IhostVlad
Copy link
Contributor

~12 million events eventstore, RDS with 32 ACU
Speed up ~40 times within specific eventstore

Before changes: execution time ~34000 ms, execution plan:

Limit (cost=2529306.86..2529309.36 rows=1000 width=237) (actual time=34616.751..34617.251 rows=1000 loops=1)
-> Sort (cost=2529306.86..2529946.86 rows=256000 width=237) (actual time=34616.750..34617.182 rows=1000 loops=1)
Sort Key: events."timestamp", events."threadCounter", events."threadId"
Sort Method: top-N heapsort Memory: 1926kB
-> Append (cost=556.04..2515270.65 rows=256000 width=237) (actual time=18.115..34592.014 rows=197739 loops=1)
-> Limit (cost=556.04..10045.03 rows=1000 width=237) (actual time=18.114..45.263 rows=1000 loops=1)
-> Bitmap Heap Scan on events (cost=556.04..48703.20 rows=5074 width=237) (actual time=18.113..45.180 rows=1000 loops=1)
Recheck Cond: (("threadId" = 0) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 1150
Heap Blocks: exact=1158
-> Bitmap Index Scan on events_pkey (cost=0.00..554.77 rows=14221 width=0) (actual time=15.769..15.769 rows=8402 loops=1)
Index Cond: (("threadId" = 0) AND ("threadCounter" >= '0'::bigint))
-> Limit (cost=556.04..10045.03 rows=1000 width=237) (actual time=16.213..44.080 rows=1000 loops=1)
-> Bitmap Heap Scan on events events_1 (cost=556.04..48703.20 rows=5074 width=237) (actual time=16.211..43.999 rows=1000 loops=1)
Recheck Cond: (("threadId" = 1) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 893
Heap Blocks: exact=1232
-> Bitmap Index Scan on events_pkey (cost=0.00..554.77 rows=14221 width=0) (actual time=14.652..14.652 rows=2072 loops=1)
.......

After changes: execution time ~700 ms, execution plan:

Sort (cost=2788858.58..2788861.08 rows=1000 width=237) (actual time=293.167..293.437 rows=1000 loops=1)
Sort Key: events."threadCounter", events."threadId"
Sort Method: quicksort Memory: 1613kB
-> Limit (cost=2788796.25..2788798.75 rows=1000 width=237) (actual time=292.061..292.418 rows=1000 loops=1)
-> Sort (cost=2788796.25..2789436.25 rows=256000 width=237) (actual time=292.060..292.349 rows=1000 loops=1)
Sort Key: events."timestamp"
Sort Method: top-N heapsort Memory: 1926kB
-> Append (cost=0.56..2774760.04 rows=256000 width=237) (actual time=0.041..268.203 rows=197739 loops=1)
-> Limit (cost=0.56..11136.97 rows=1000 width=237) (actual time=0.040..1.421 rows=1000 loops=1)
-> Index Scan using events_pkey on events (cost=0.56..56506.68 rows=5074 width=237) (actual time=0.040..1.350 rows=1000 loops=1)
Index Cond: (("threadId" = 0) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 1150
-> Limit (cost=0.56..11136.97 rows=1000 width=237) (actual time=0.027..1.291 rows=1000 loops=1)
-> Index Scan using events_pkey on events events_1 (cost=0.56..56506.68 rows=5074 width=237) (actual time=0.026..1.220 rows=1000 loops=1)
Index Cond: (("threadId" = 1) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 893
-> Limit (cost=0.56..11136.97 rows=1000 width=237) (actual time=0.025..0.458 rows=388 loops=1)
-> Index Scan using events_pkey on events events_2 (cost=0.56..56506.68 rows=5074 width=237) (actual time=0.024..0.431 rows=388 loops=1)
.......

@IhostVlad IhostVlad force-pushed the feature/es-pg-sort-hints branch from bcfde5c to 0b14f87 Compare January 18, 2022 10:27
@FreeSlave FreeSlave merged commit 5a1aea9 into dev Jan 18, 2022
@FreeSlave FreeSlave added the enhancement Issues related to potential improvements and new features label Jan 18, 2022
@FreeSlave FreeSlave deleted the feature/es-pg-sort-hints branch January 18, 2022 11:14
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement Issues related to potential improvements and new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants