Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat: Audio Filter plugin #559

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Conversation

typester
Copy link
Member

This pull request introduces a feature that allows users to load their own audio filter library and use it as an inbound audio filter. It can be used for purposes such as noise cancellation, VAD, and more.

The plugins must be dynamic libraries (cdylib) and are required to export the following functions:

type CreateFn = unsafe extern "C" fn(sampling_rate: u32, options: *const c_char) -> *mut c_void;
type DestroyFn = unsafe extern "C" fn(*const c_void);
type ProcessI16Fn = unsafe extern "C" fn(*const c_void, usize, *const i16, *mut i16);
type ProcessF32Fn = unsafe extern "C" fn(*const c_void, usize, *const f32, *mut f32);

Copy link
Contributor

ilo-nanpa bot commented Jan 27, 2025

it seems like you haven't added any nanpa changeset files to this PR.

if this pull request includes changes to code, make sure to add a changeset, by writing a file to .nanpa/<unique-name>.kdl:

minor type="added" "Introduce frobnication algorithm"

refer to the manpage for more information.

@davidzhao davidzhao requested a review from theomonnom February 4, 2025 08:05
Copy link
Contributor

@bcherry bcherry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm but i wouldn't give my review too much weight, needs @theomonnom

let options = CString::new(options.as_ref()).unwrap_or(CString::new("").unwrap());
unsafe {
let on_load = lib.get::<Symbol<OnLoadFn>>(b"on_load")?;
on_load(options.as_ptr());
Copy link
Member

@theomonnom theomonnom Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the on_load function do a blocking request to our livekit server (on the krisp-plugin repo).
I think we should add 3 things:

  • making sure we don't block the tokio loop when loaded.
  • add retries to this request (or expose failures?)
  • check how slow it is to load the weights into memory

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Moved this initialization to Room::connect, now connect returns error if on_load fails.
  • Put plugin loading codes in spawn_blocking so that we can avoid tokio runtime are blocked.

let create_fn: CreateFn = unsafe { std::mem::transmute(self.create_fn_ptr) };

let options = CString::new(options.as_ref()).unwrap_or(CString::new("").unwrap());
let ptr = unsafe { create_fn(sampling_rate, options.as_ptr()) };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we check if the ptr is valid?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just added for null check for this in 1bdb932

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants