u/Humanbound_AI
- 1points
- 1points
- 1points
- 1points
- 1points
- 1points
- 1points
- 0points
- 1points
- 2points
- 6points
- 1points
- 1points
- 1points
- 1points
- 1points
- 1points
- 1points
- 3points
- 1points
- 10points
Humanbound_AI
Humanbound_AI
Ha, fair! Quick honesty check though: $0.14 was one scan of our hello-world arena on a cheap model with thinking off, and we only tested plumbing, not attack quality, so don't read it as a deep scan. BTW the cheapest host was also the slowest (~36 min). Curious what you'd run it on.
Humanbound_AI
Ha, "got lost in thought and forgot to speak" is exactly it. Same pain on the routing side: with default routing, 68% of our spend went on empty replies. Which host did you end up pinning finally?
Humanbound_AI
No pattern really, sometimes even the benign requests were denied.
We're obviously biased: we build open source tooling for testing AI agents. Two things that do most of the work.
One credential per agent, scoped to the task, never the shared service account. The common failure isn't a clever jailbreak, it's an agent handed the token the app already had, so a read-only job is one confused tool call away from sending mail as you.
Then gate on reversibility, not sensitivity. Reading and drafting are cheap to get wrong. Sending, writing, deleting, paying are not. Approval only on that second list. Gate everything and people click approve without reading, which is worse than no gate.
For production, log every tool call with the input that triggered it and the identity it ran as. Otherwise your incident writeup is "the model did something weird."
Neither of those needs our tooling, which is why we'd lead with them.