🔴 Anomaly detection

[hejny]

This is a similar concept to 🧪 Expectations and 👮 Agent Adversary Expectations, but more in depth how language models and vector spaces work.

On each result or even parameter (despite its generated inside pipeline or its input parameter) based on samples or previous result detect anomalies and different texts as usual.

Use combination of several methods:

• Using agents to detect anomalies 👮 Agent adversary expectations #39
• Distance in the embedded vector space
• Keyword search
• Compression Search
• Strange character search

When an anomaly is detected, do one of the following

• Just retry
• Pause and check manually
• Escalate to a better model

---

See also #33

[hejny]

🧪 What we tried

(Describe our experience here)

[hejny]

🔎 Existing solutions

(Do some research / discussion)

• Has anyone else had the same problem?
• Has a project come up with the solution
  • https://www.evidentlyai.com/
  • https://langtail.com/
• Is there a research paper about it?
• Is there an article, video, podcast about it?
• Is this even a good idea?