fix-slow-family-endpoint-in-the-admin-service

[odrakes-cpr]

Description

The search endpoint api/v1/families/?q= for families was running quite slow in the admin service, taking over 6 seconds to return a response. Notably on the cclw user admin account which has the most amount of documents and families, we suspected that this had something to do with the multiple geographies changes that we made.

To resolve this we have gone with using a raw sql query and instead executing that.

• The get endpoint works fine and does not seem to be an intensive process so we will leave that be for now.
• This update relates to the search endpoint even though the title says all endpoint, the endpoint of concern is the search
• We have tried to replicate the query as best as we can, retrieving the family slugs, family documents, events etc in the main query rather than running a separate query in the serialization process (_family_to_dto......)
• The main query should replicate the same joins etc and where values are not present in the join, such as document ids which would be easier accessed through the orm, running subqueries to get those values, i.e slugs/geos
• Where some of the properties required for the front are actually hybrid properties and therefore not accessible in a sql query as they sit on the orm, we have tried to match the calculation of those values see family_status|last_updated_date|published_date in the family models.

These are represented in the sql query as such

        CASE
            WHEN EXISTS (
                SELECT 1
                FROM family_document fd
                WHERE fd.family_import_id = f.import_id
                AND fd.document_status = 'PUBLISHED'
            ) THEN 'PUBLISHED'
            WHEN EXISTS (
                SELECT 1
                FROM family_document fd
                WHERE fd.family_import_id = f.import_id
                AND fd.document_status = 'CREATED'
            ) THEN 'CREATED'
            ELSE 'DELETED'
        END AS family_status,
        (
            SELECT MAX(fe.date)
            FROM family_event fe
            WHERE fe.family_import_id = f.import_id
        ) AS last_updated_date,
        (
            SELECT MIN(fe.date)
            FROM family_event fe
            WHERE fe.family_import_id = f.import_id
            AND EXISTS (
                    SELECT 1
                    FROM jsonb_array_elements_text(fe.valid_metadata::jsonb->'datetime_event_name') AS datetime_event_name
                    WHERE datetime_event_name = fe.event_type_name
                )
        ) AS published_date

• Finally for the query, we have passed in variables/params into the main query rather than the actual values to prevent against sql injection or anything going awry with user inputs.
• Running the raw sql query and the dto mapping now takes and average of 1.5seconds as illustrated in the docker logs below, which solves the problem of a slow endpoint as the query is coming back quicker.

Proposed version

Please select the option below that is most relevant from the list below. This
will be used to generate the next tag version name during auto-tagging.

• Skip auto-tagging
• Patch
• Minor version
• Major version

Visit the Semver website to understand the
difference between MAJOR, MINOR, and PATCH versions.

Notes:

• If none of these options are selected, auto-tagging will fail
• Where multiple options are selected, the most senior option ticked will be
  used -- e.g. Major > Minor > Patch
• If you are selecting the version in the list above using the textbox, make
  sure your selected option is marked [x] with no spaces in between the
  brackets and the x

Type of change

Please select the option(s) below that are most relevant:

• Bug fix
• New feature
• Breaking change
• GitHub workflow update
• Documentation update
• Remove legacy code
• Dependency update

How Has This Been Tested?

Please describe the tests that you added to verify your changes.

Reviewer Checklist

• DB_CLIENT DEPENDENCY IS ON THE LATEST VERSION
• The PR represents a single feature (small drive-by fixes are also ok)
• The PR includes tests that are sufficient for the level of risk
• The code is sufficiently commented, particularly in hard-to-understand areas
• Any required documentation updates have been made
• Any TODOs added are captured in future tickets
• No FIXMEs remain

[odrakes-cpr]

https://linear.app/climate-policy-radar/issue/APP-183/fix-slow-family-endpoint-in-the-admin-service

[annaCPR]

suggestion: Would you mind documenting in the PR description why and how this solution fixes the problem we had with the endpoint? Just so that I don't have to try to figure it out from the SQL :-P

[odrakes-cpr]

Resulting query -

[odrakes-cpr]

suggestion: Would you mind documenting in the PR description why and how this solution fixes the problem we had with the endpoint? Just so that I don't have to try to figure it out from the SQL :-P

Sure hopefully that has helped.

[jamesgorrie]

Really nice investigative work - I wish we had some metrics to show the improvement in real time.

Might be worth getting current local metrics and compare them so we can see the improvement.