feat: implement pagination for transaction log queries #5281

dhlidongming · 2024-12-16T12:47:16Z

This PR introduces pagination to the transactions log using fastapi-pagination on the backend and PaginatorComponent on the frontend.
These changes aim to enhance retrieval speed, reduce memory consumption, and prevent potential lags during frequent debugging sessions.

codspeed-hq · 2024-12-16T13:02:05Z

CodSpeed Performance Report

Merging #5281 will degrade performances by 45.18%

_{Comparing dhlidongming:add-transaction-pagination (7bdd67d) with main (4736aae)}

Summary

⚡ 2 improvements
❌ 1 regressions
✅ 12 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`main`	`dhlidongming:add-transaction-pagination`	Change
❌	`test_setup_llm_caching`	1.2 ms	2.2 ms	-45.18%
⚡	`test_successful_run_with_input_type_any`	331.1 ms	273.5 ms	+21.05%
⚡	`test_successful_run_with_output_type_any`	279.5 ms	239.1 ms	+16.86%

ogabrielluiz

Hey @dhlidongming

Are you using the Transactions table in your developments?

ogabrielluiz · 2024-12-16T15:01:08Z

src/frontend/src/modals/flowLogsModal/index.tsx

+  const {
+    data: TransactionData,
+    isLoading,
+    refetch,
+  } = useGetTransactionsQuery({
    id: currentFlowId,
+    params: {
+      page: pageIndex,
+      size: pageSize,
+    },
    mode: "union",
  });

+  const data = {
+    rows: TransactionData?.rows ?? [],
+    columns: TransactionData?.columns ?? [],
+    pagination: {
+      page: TransactionData?.pagination?.page ?? 1,
+      size: TransactionData?.pagination?.size ?? 10,
+      total: TransactionData?.pagination?.total ?? 0,
+      pages: TransactionData?.pagination?.pages ?? 0,
+    },
+  };
+
  useEffect(() => {
-    if (data) {
+    if (TransactionData) {
      const { columns, rows } = data;
      setColumns(columns.map((col) => ({ ...col, editable: true })));
      setRows(rows);
    }
-    if (open) refetch();
-  }, [data, open, isLoading]);
+  }, [TransactionData]);


Shouldn't the variables named TransactionData be data instead?

@ogabrielluiz Thank you for pointing that out.I have removed the redundant TransactionData variable and updated the code accordingly.

dhlidongming · 2024-12-16T15:56:13Z

@ogabrielluiz Yes, we need to use the Transactions table to view the execution logs of various components to determine if the Agent is running correctly.
Recently, we found that when a certain flow is executed frequently, the Logs page opens very slowly and consumes a lot of browser memory because pagination was not implemented for the queries.

Hey @dhlidongming

Are you using the Transactions table in your developments?

ogabrielluiz · 2024-12-16T18:17:24Z

@ogabrielluiz Yes, we need to use the Transactions table to view the execution logs of various components to determine if the Agent is running correctly. Recently, we found that when a certain flow is executed frequently, the Logs page opens very slowly and consumes a lot of browser memory because pagination was not implemented for the queries.

Hey @dhlidongming
Are you using the Transactions table in your developments?

That's good to know. We are planning on improving logs and stuff like that. We'll use this PR as an example that people are still using the transactions.

Thanks @dhlidongming !

…nation with Page[TransactionTable].

… check for 'items' in JSON response instead of direct equality to an empty list.

- Updated TransactionReadResponse to use 'id' as an alias for 'transaction_id'. - Introduced a new function, transform_transaction_table, to convert TransactionTable instances to TransactionReadResponse, supporting both single and list inputs. - Improved import structure for better readability in crud.py.

…ion retrieval - Updated the get_transactions function to utilize the new transform_transaction_table function, enhancing the transformation of TransactionTable instances during pagination. - Improved the import structure by adding the necessary import for transform_transaction_table.

…eval - Added 'params' to VertexTuple for better clarity in transaction logging. - Modified assertions in test_delete_folder_with_flows_with_transaction_and_build to check for 'items' in the JSON response instead of direct equality to an empty list, ensuring more accurate test validation.

- Integrated fastapi_pagination into the main application by adding the add_pagination function to the FastAPI instance. - Removed the redundant add_pagination call from the flows module to streamline the pagination setup.

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Dec 16, 2024

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Dec 16, 2024

ogabrielluiz requested changes Dec 16, 2024

View reviewed changes

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Dec 16, 2024

dhlidongming requested a review from ogabrielluiz December 16, 2024 16:31

ogabrielluiz approved these changes Dec 16, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 16, 2024

ogabrielluiz force-pushed the add-transaction-pagination branch from 0611411 to 8c32ef1 Compare December 16, 2024 18:16

ogabrielluiz enabled auto-merge (squash) December 16, 2024 18:16

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Dec 16, 2024

dhlidongming and others added 12 commits December 16, 2024 17:51

Add pagination support for transaction logs.

9995ee6

[autofix.ci] apply automated fixes

f74edf8

[autofix.ci] apply automated fixes (attempt 2/3)

4468e16

Replace 'TransactionData' with 'data' variable.

11f1bfa

[autofix.ci] apply automated fixes

236ba12

Enhance transaction retrieval by updating return type to support pagi…

d597473

…nation with Page[TransactionTable].

Fix test assertion in test_delete_flows_with_transaction_and_build to…

6e33493

… check for 'items' in JSON response instead of direct equality to an empty list.

chore: increase flow logs modal page size to 20

32105ce

feat: Add pagination support to FastAPI application

7bdd67d

- Integrated fastapi_pagination into the main application by adding the add_pagination function to the FastAPI instance. - Removed the redundant add_pagination call from the flows module to streamline the pagination setup.

ogabrielluiz force-pushed the add-transaction-pagination branch from ff280e1 to 7bdd67d Compare December 16, 2024 20:51

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Dec 16, 2024

ogabrielluiz merged commit 1ec6380 into langflow-ai:main Dec 16, 2024
36 checks passed

dhlidongming deleted the add-transaction-pagination branch December 17, 2024 02:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement pagination for transaction log queries #5281

feat: implement pagination for transaction log queries #5281

dhlidongming commented Dec 16, 2024

codspeed-hq bot commented Dec 16, 2024 •

edited

Loading

ogabrielluiz left a comment

ogabrielluiz Dec 16, 2024

dhlidongming Dec 16, 2024

dhlidongming commented Dec 16, 2024

ogabrielluiz commented Dec 16, 2024

feat: implement pagination for transaction log queries #5281

feat: implement pagination for transaction log queries #5281

Conversation

dhlidongming commented Dec 16, 2024

codspeed-hq bot commented Dec 16, 2024 • edited Loading

CodSpeed Performance Report

Merging #5281 will degrade performances by 45.18%

Summary

Benchmarks breakdown

ogabrielluiz left a comment

Choose a reason for hiding this comment

ogabrielluiz Dec 16, 2024

Choose a reason for hiding this comment

dhlidongming Dec 16, 2024

Choose a reason for hiding this comment

dhlidongming commented Dec 16, 2024

ogabrielluiz commented Dec 16, 2024

codspeed-hq bot commented Dec 16, 2024 •

edited

Loading