Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

/rest/v1/project_directory performance #4725

Closed
MichaelAkvo opened this issue Dec 8, 2021 · 1 comment
Closed

/rest/v1/project_directory performance #4725

MichaelAkvo opened this issue Dec 8, 2021 · 1 comment
Assignees
@MichaelAkvo
Labels
Priority: Low python Pull requests that update Python code

Comments

@MichaelAkvo
Copy link
Contributor

MichaelAkvo commented Dec 8, 2021 •
edited
Loading

The endpoint is unbelievably slow and incredibly large.

$ time curl 'http://localhost/rest/v1/project-directory' > dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 11.6M    0 52749    0     0    113      0 29:56:54  0:07:46 29:49:08 13131

real    7m46.442s
user    0m0.021s

Basically 9k projects are selected in production and for each of them a bunch more requests are made.
@MichaelAkvo MichaelAkvo changed the title /rest/v1/project_directory /rest/v1/project_directory performance Dec 8, 2021
@MichaelAkvo MichaelAkvo self-assigned this Dec 8, 2021
@MichaelAkvo MichaelAkvo added Priority: High python Pull requests that update Python code labels Dec 8, 2021
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
fix(quick): cache entire project directory request
82e63ce 
The projects don't change often, so we should be able to cache them for a while.
Additionally, every hour we will regenerate their cached values (take ~7m right now).

So the quick fix is two-fold:

 - cache the entire result and keep for an hour
 - regenerate per project cache every hour

Once the page cache expires, queries should take ~10-20 seconds, but only once until the next hour.

#4725: /rest/v1/project_directory performance
@MichaelAkvo MichaelAkvo mentioned this issue Dec 9, 2021
Closed
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
fix(quick): cache entire project directory request
ef6b81e 
The projects don't change often, so we should be able to cache them for a while.
Additionally, every hour we will regenerate their cached values (take ~7m right now).

So the quick fix is two-fold:

 - cache the entire result and keep for an hour
 - regenerate per project cache every hour

Once the page cache expires, queries should take ~10-20 seconds, but only once until the next hour.

#4725: /rest/v1/project_directory performance
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
fix(quick): cache entire project directory request
6a315de 
The projects don't change often, so we should be able to cache them for a while.
Additionally, every hour we will regenerate their cached values (take ~7m right now).

So the quick fix is two-fold:

 - cache the entire result and keep for an hour
 - regenerate per project cache every hour

Once the page cache expires, queries should take ~10-20 seconds, but only once until the next hour.

#4725: /rest/v1/project_directory performance
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
fix: Convert absolute_url into a cached property
7ade8e0 
For some reason it takes a whole lot of time to reverse an URL.
Even the example for `cached_property` uses `absolute_url` as an example.

#4725: /rest/v1/project_directory performance
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
workaround: Build cache from required tables for project_dir
132cc92 
Instead of relying on django to make deferred queries, we make all queries ourselves,
 build the cache, and use it.
It's per project ID for faster access.

Additionally, `url` is not required anymore as the UI builds the url to the project now

#4725: /rest/v1/project_directory performance
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
workaround: Ultra hacky limitation to speed up project_dir
047785b 
The request takes way too long when the cache is broken.
On production, it never actually is able to fill the cache.

To give us time to work on a solution, this limits the number of projects in the directory from

#4725: /rest/v1/project_directory performance
@MichaelAkvo MichaelAkvo mentioned this issue Dec 9, 2021
Merged
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
workaround: Build cache from required tables for project_dir
7221ff4 
Instead of relying on django to make deferred queries, we make all queries ourselves,
 build the cache, and use it.
It's per project ID for faster access.

Additionally, `url` is not required anymore as the UI builds the url to the project now

#4725: /rest/v1/project_directory performance
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
workaround: Ultra hacky limitation to speed up project_dir
e9aab7f 
The request takes way too long when the cache is broken.
On production, it never actually is able to fill the cache.

To give us time to work on a solution, this limits the number of projects in the directory from

#4725: /rest/v1/project_directory performance
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
Merge pull request #4727 from akvo/issue/4725-ultrahack-limit-project…
6732a5d 
…_dir-to-10

[#4725] workaround: Ultra hacky limitation to speed up project_dir
MichaelAkvo added a commit that referenced this issue Dec 9, 2021
@MichaelAkvo
workaround: Build cache from required tables for project_dir
9b0f6a3 
Instead of relying on django to make deferred queries, we make all queries ourselves,
 build the cache, and use it.
It's per project ID for faster access.

Additionally, `url` is not required anymore as the UI builds the url to the project now

#4725: /rest/v1/project_directory performance
zuhdil added a commit that referenced this issue Dec 10, 2021
@zuhdil
[#4725] Increase serialized_project performance
6abcf43 
- Add fields that should not be deffered
zuhdil added a commit that referenced this issue Dec 10, 2021
@zuhdil
[#4725] Implement get_thumbnail with timeout
84a0851
zuhdil added a commit that referenced this issue Dec 10, 2021
@zuhdil
Merge pull request #4728 from akvo/4725-project-dir-perf
c0b7e09 
[#4725] Improve project directory performance
zuhdil added a commit that referenced this issue Dec 11, 2021
@zuhdil
[#4725] Cache the project list of project directory
1e0dc42
zuhdil added a commit that referenced this issue Dec 11, 2021
@zuhdil
[#4725] Use SWR in frontend
eaa48b6
zuhdil added a commit that referenced this issue Dec 11, 2021
@zuhdil
[#4725] Use SWR in frontend
951a097
zuhdil added a commit that referenced this issue Dec 11, 2021
@zuhdil
Merge pull request #4729 from akvo/4725-project-dir-perf-2
4444446 
[#4725] Project directory performance improvement part 2
@MichaelAkvo
Copy link
Contributor Author

The problem in production seems to have come from get_thumbnail never ending and blocking the entire process.

It might still be possible to further improve performance by using cache_page on the endpoint and running the populate_project_directory_cache django command in a cron.

A rework of the landing page is coming, so this might not be relevant anymore.

@MichaelAkvo MichaelAkvo moved this to Done in RSR Dec 7, 2022
@MichaelAkvo MichaelAkvo added this to RSR Dec 7, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@MichaelAkvo MichaelAkvo

Labels
Priority: Low python Pull requests that update Python code
Projects
RSR
Archived in project
Milestone
No milestone
Development

No branches or pull requests
1 participant
@MichaelAkvo