Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat: add ai-prompt-guard plugin #12008

Open
wants to merge 18 commits into
base: master
Choose a base branch
from

Conversation

Revolyssup
Copy link
Contributor

@Revolyssup Revolyssup commented Feb 28, 2025

Description

The ai-prompt-guard plugin safeguards your AI endpoints by inspecting and validating incoming prompt messages. It checks the content of requests against user-defined allowed and denied patterns to ensure that only approved inputs are processed. Based on its configuration, the plugin can either examine just the latest message or the entire conversation history, and it can be set to check prompts from all roles or only from end users.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. plugin labels Feb 28, 2025
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Feb 28, 2025
}

function _M.check_schema(conf)
return core.schema.check(schema, conf)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to confirm the items in allow_patterns[] and deny_patterns[], they should be a valid regex

you can call this function: https://github.com/openresty/lua-resty-core/blob/master/lib/resty/core/regex.lua#L639

if #conf.allow_patterns > 0 then
local any_allowed = false
for _, pattern in ipairs(conf.allow_patterns) do
if ngx.re.find(content_to_check, pattern, "jou") then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the pattern is not a plain text, it is a regex

the current way is wrong

@Revolyssup
Copy link
Contributor Author

@membphis Made changes

@Revolyssup Revolyssup requested a review from membphis February 28, 2025 10:41
membphis
membphis previously approved these changes Mar 2, 2025
Copy link
Contributor

@bzp2010 bzp2010 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a description of this feature to the PR, otherwise I don't know why it should exist. The project is in the public domain and I think there will be others who would like to know the details as well.

https://www.open.edu/openlearn/money-business/hybrid-working-change-management/content-section-2.1

@Revolyssup
Copy link
Contributor Author

Please add a description of this feature to the PR, otherwise I don't know why it should exist. The project is in the public domain and I think there will be others who would like to know the details as well.

https://www.open.edu/openlearn/money-business/hybrid-working-change-management/content-section-2.1

Added description

Comment on lines 114 to 116
if not conf.match_all_roles and #messages > 0 and messages[#messages].role ~= "user" then
return
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't understand this check? should we check all messages that's role is user when match_all_roles is false? why only check last message in array?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combined with the match_all_conversation_history configuration, we should first obtain the list of messages to be judged based on match_all_conversation_history, and then decide whether to perform pattern judgment according to each message's role and match_all_roles.

@Revolyssup Revolyssup dismissed stale reviews from shreemaan-abhishek and membphis via ac03b08 March 5, 2025 10:33
nic-6443
nic-6443 previously approved these changes Mar 5, 2025
membphis
membphis previously approved these changes Mar 6, 2025
Copy link
Member

@membphis membphis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rerun CI, some of them failed

@Revolyssup Revolyssup dismissed stale reviews from membphis and nic-6443 via 3396c06 March 6, 2025 06:17
Copy link
Member

@membphis membphis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

"plugins": {
"ai-prompt-guard": {
"match_all_roles": true,
"allow_patterns": [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bad indentation too

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
plugin size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants