Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Calls to the get_online_features API are slower in v0.42.0 than in v0.40.1 #4887

Open
jparkzz opened this issue Jan 3, 2025 · 1 comment · Fixed by robhowley/feast#10 · May be fixed by #4892
Open

Calls to the get_online_features API are slower in v0.42.0 than in v0.40.1 #4887

jparkzz opened this issue Jan 3, 2025 · 1 comment · Fixed by robhowley/feast#10 · May be fixed by #4892

Comments

@jparkzz
Copy link
Contributor

jparkzz commented Jan 3, 2025

Expected Behavior

The call to the get_online_features API must be faster or equal to v0.42.0 than v0.40.1.

Current Behavior

v0.42.0 is slower than v0.40.1

Steps to reproduce

Specifications

  • Version: v0.42.0, v0.41.*
  • Platform: Kubernetes + Redis (Online) + Bigquery (Offline) + PostgreSQL(SQLRegistry)
  • Subsystem:

Possible Solution

Need to change the default behavior to sync (based fastapi - anyio threadpool), or change the part where blocking (or delay) occurs in async to async.

I was running 0.40.1, and upgraded to 0.41.2 / 0.42.0, but I actually found that it got slower.

Looking at the structure of async def get_online_features
(1) Get the information of the requested feature_view.
(2) Check the permission (when using RBAC).
(3) Get the value from the online store using asynchronous.
(4) Deserialize the protocol buffer message from the online store into a dict.

It flows in this order, but only (3) is actually protected by the threadpool or eventloop.
However, (1), (2), and (4) also take a lot of time for protobuf serialization-deserialization, and as a result, (1), (2), and (4) lose time when performed with a single thread (async) compared to multi-threading.

@jparkzz
Copy link
Contributor Author

jparkzz commented Jan 3, 2025

Screenshot 2025-01-03 at 5 46 36 PM

This is a graph of the latency changes (v0.42.0 -> v0.40.1) in my system.

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
1 participant