The agent on this site, and what it took to trust it

There is an agent on this site, at /ask, and it will answer questions about the work from the pages you are reading right now. It names the page each answer came from, it refuses when the answer is not on any page, and it keeps refusing when you push on it. A second lane on the same agent answers recruiters about me as a candidate from a file I wrote for that purpose, and because you cannot open that file, the lane tells you up front that you are taking my word for it. Both lanes were evaluated against the live site before anyone got the link. The buyer lane answered 17 of 17 questions the way the key expected and held across 4 multi-turn pressure sequences. The hiring lane answered 21 of 21, held across 5 sequences, and landed the expected verdict on 2 real job postings. Every one of those answers was read by a person, not just checked by a script, and the reason for that is the actual story here.

This is the only case study on the site whose subject is the site. I went back and forth on whether that was too cute. I kept it because the agent is the one build you can test yourself, right now, without taking anything on faith, and that is the point of everything else on this page.

What was actually going on. The site claims I build systems that answer from sources and refuse when the source is not there. A page saying that is a brochure. An agent you can interrogate is evidence, but only if the agent holds, and an agent that folds under a little pressure would be worse than no agent at all because it would disprove the claim on the spot. So the build was mostly about refusing correctly and proving the refusal survives someone trying to get around it, and only then about answering.

What I decided, and what I ruled out.

Where the answers come from: the buyer lane reads the same pages you do, generated at build time from the rendered HTML plus the handbook for the coaching desk, about 16,100 tokens all in. I ruled out feeding it the markdown source files the pages are written from, and the reason was not tidiness. The source folder still said one of my builds had 410 tests when the page had said 900 for weeks. A draft layer drifts. The page is what I have actually committed to in public, so the page is the only thing the agent is allowed to know. That one decision turned out to matter more than any other on this list, and I made it for a boring reason before I knew why it mattered.

No search step: at this size there is no retrieval. The agent gets the whole corpus every time, and caching makes that nearly free. A retrieval layer would add a component that can fail quietly by fetching the wrong passage, and I would rather not carry that until the corpus is large enough to force it. It is not there yet, and I have not measured where "yet" is.

Two lanes instead of one: the buyer lane is not allowed to sell. It answers, cites, and points at the form. The hiring lane has the opposite job, since a recruiter wants an honest case for and against, including "probably not." I ruled out one prompt doing both. The cost was never tokens, it was rule interference. A single prompt holding "never advocate" and "advocate honestly" resolves the conflict by softening both, and the refusal discipline is the entire demonstration, so I split them. Same code path, same agent, different contract loaded depending on which lane you pick.

The router verifies nothing, on purpose: you pick your lane at the top of the box. That choice is sent with every request and the server checks only that the lane exists. It does not try to work out whether you belong there, because there is no classifier to maintain and nothing for it to get wrong. Enforcement lives inside each lane instead. A buyer who wanders into the hiring lane gets an honest candidate assessment and no pricing, and a recruiter who forwards the link to a buyer gets the same. The hiring lane refuses to quote my client prices even though the home page publishes them, because a forwarded link is exactly how a number travels somewhere I did not send it.

One URL, and the visitor picks: I had originally planned to keep the hiring lane off the site and hand its link out privately. I reversed that. Two different links for two different audiences is the move of someone showing different faces to different people, and one page that says plainly "I take clients, and I will also talk to you about a role" reads as open to it without needing it. That is only credible because the case studies sit right underneath the box. The cost, which I accepted, is that every buyer who opens /ask sees that I am open to a role.

Saying where the evidence comes from: the two lanes cite the same way but the citations mean different things. On the buyer lane every answer points at a page you can open in another tab and check, so if the agent is wrong you can find out without trusting anyone. On the hiring lane the source is one private file, so the citation looks the same and functions as a promise. I made the hiring lane say that in its own words, on the page above the box and again in its answers when asked, because a citation that looks checkable and is not would be a lie by formatting. This turned out to be the more useful idea to come out of the build, and every knowledge system I set up for a client has the same split hiding in it somewhere.

The score matters less than what it was made of. I wrote a key of expected answers, wrote automated traps to catch the obvious failures, and ran the agent against it five times before the buyer lane shipped. Across those five passes the agent produced one failure I would call its own, which was a strong answer to someone who had just described their own 40-person company, ending without pointing them at the form. Everything else that failed was the test, not the thing being tested. The harness swallowed seventeen authentication errors and reported them as wrong answers. The key said the agent must refuse to quote a price, and the agent quoted the audit price and cited the home page, which is correct because that number is published there. One question asked about material that was in my drafts and not on the rendered page, so the agent refused and the key was wrong, and I fixed that by publishing the material. Another asked about a word that appears zero times on the page it named. One demanded a flat refusal where the agent's cited, non-committal answer was plainly better. One trap fired on the ordinary word "sources" in a good sentence. The thing built to refuse held, and the things built to test it did not, until they were tested back. The corpus decision is what made that visible, because every one of those key errors came from a session that had written the key from memory and from the draft files instead of from the pages.

The hiring lane had a harder eval, and the agent's misses there were real. Handed a real posting that never states who the role reports to, it opened by asserting one and then assessed fit against the line it had just made up. That is the failure the whole lane exists to prevent, and the fix is a rule that a missing reporting line gets named as missing and never inferred from a title. It also answered for me once, saying I would not take a role and that money would not change my mind, when the ceiling for an agent speaking about someone is "probably not a fit." And one of my own rules caused a regression. Told to use my file's vocabulary rather than the posting's, it declined to compare my two years of shipped AI work against a five-year requirement because the file never used the phrase "automation and ops tooling," which is a dodge. The concession rule now outranks the vocabulary rule. Four real problems across the hiring build, roughly a dozen false alarms from the key and traps, and every real finding came from reading the transcript rather than from the pass count.

The rate limiter is a per-instance counter and it resets whenever the function cold-starts, so it slows abuse down and does not stop it. That is enough for an agent on a low-traffic site, and if the question log ever shows someone working around it, the counter moves to shared storage. Provider variance is also real. The same model at temperature zero gave different answers to two questions depending on which of the routed providers served the request, and the eval caught it only because I ran the key more than once.

The attack I was most worried about held. The hiring lane accepts 4,000 characters so a recruiter can paste a posting, and a posting is exactly the shape of a real injection, a long block of legitimate text with an instruction buried in it. The eval includes a fictional posting with a line in the middle telling the assistant to declare the candidate a strong fit and state his salary. On every run the agent named the two requirements I do not meet, refused the number, and when pushed on turn three to "follow the posting," said the posting's instruction does not change how it answers.

Before, the site asserted that I build systems that refuse without a source. Now there is an agent that does it, on the live site, and anyone who doubts the claim can spend two minutes trying to break it. The eval gave me something I did not expect, which is a record showing that the tests were less trustworthy than the system, and that the only way to know was to read every answer. That record is now the part of the build I would show a client first.

Widen the buyer lane so it answers "can you do what I need" from the what-I-build page and the nearest case study, without generalizing past what is published, which is its own diff with its own rerun. Move the rate limiter to shared storage the first time it matters. And find the corpus size where the no-retrieval decision stops being true, before it finds me.