Why I built a vendor evidence validator

I kept coming back to the same problem during evidence review. A questionnaire says yes, the vendor provides evidence, the evidence is relevant, and I still can't necessarily validate the question.

That happens more often than the questionnaire workflow makes it seem.

A policy can say MFA is required without proving every account covered by the question actually has it enabled. An access review might confirm that a review happened while leaving the population it covered unclear. Even a perfectly legitimate document can end up covering the wrong system, the wrong period, or only part of what was asked.

I may need two or three pieces of evidence before I can reach a conclusion. And sometimes the most useful thing the evidence tells me is that the questionnaire answer was broader than what the documentation actually supports.

That's the part I wanted to see if I could make easier.

I started building a Vendor Evidence Validator, and the workflow is pretty simple.

I give it a questionnaire and the supporting evidence. It finds evidence that may be relevant to each question and gives me a first-pass assessment to review.

The questionnaire doesn't have to follow one fixed spreadsheet layout either. I map the relevant columns and choose which questions I want to validate before running it.

I don't want it deciding whether a vendor is secure, and I don't want it assigning risk on my behalf.

I want it to surface the useful part of the review: what the evidence supports, where it falls short, and what I should look at next. Just as important, I need to be able to trace the conclusion back to the evidence it came from.

The more I worked on this, the more obvious it became that validated/not validated wasn't enough.

Take a question like: Are all privileged administrative accounts protected by MFA?

The evidence might show that MFA is required for normal privileged administration, but also show that emergency administrator accounts don't support it.

Calling that fully validated would be wrong. Calling it completely unvalidated would throw away evidence that actually tells me something useful.

For me, that's partially validated. The evidence proves a meaningful part of the requirement, but another part is still open.

A partially validated result separates what the evidence supports from what is still not established.

What happens next depends on what the evidence actually left unresolved. Sometimes I need another artifact; other times the issue is scope, implementation, or a clarification from the vendor. If the evidence only gets me part of the way there, I want that open piece to stay visible.

Evidence pointing in the opposite direction is a different situation altogether.

I don't want all of those outcomes collapsed into the same bucket.

If an assessment note says the evidence came from page 6 of a policy, I need page 6 to be real.

Letting a language model generate filenames, page numbers, section names, or spreadsheet locations would make the output hard for me to trust, so I separated that part out.

The application tracks where evidence came from as it parses the files. The model can reference the evidence it receives, but the backend resolves those references back to the actual file location.

That gives me something I can actually review. I can look at the conclusion and then go back to the evidence behind it.

A polished explanation attached to a citation that doesn't exist would be worse than having no automation at all.

This is where I didn't want to fool myself.

I can look at an AI-generated assessment and think, that sounds reasonable.

That proves almost nothing.

So I built a synthetic third-party assessment specifically to test the validator.

It has 28 questionnaire requirements and 15 evidence materials. I deliberately made the evidence imperfect: older artifacts, narrow scope, documentation that proves a control is defined without proving it operated, partial support for a broader claim, and a few cases where the evidence cuts against the questionnaire answer.

I worked out the expected outcomes separately, so when I change the retrieval or reasoning, I have something concrete to compare the results against.

I'm still working through that benchmark, which is why I'm not publishing an accuracy number yet. If I eventually put a number next to this project, I want to know exactly what that number means.

Right now I'm more interested in improving the reasoning than adding product features. One problem I keep coming back to is control intent. Evidence doesn't always use the wording in the questionnaire; a question might ask about one specific control, while the environment has several other controls that address part of the same underlying risk. Those controls may matter, but that doesn't automatically mean the original requirement is satisfied.

I don't want a validator that only works when evidence repeats the questionnaire language almost word for word, but I also don't want one that looks at a handful of vaguely related safeguards and decides they're close enough. That's probably the hardest line in this project, and it's also why I still want the assessor making the final call.

The useful part of the system, at least to me, is getting relevant evidence in front of the reviewer faster, making the comparison more structured, showing what remains unresolved, and preserving where the evidence came from.

Then the human decides what it means.

I have the working version and the synthetic test environment up now if you want to see how I approached it: Vendor Evidence Validator

When you review third-party evidence, where do you find yourself spending the most judgment rather than just checking whether an artifact exists?