Here is a question that arrives in every inbox eventually: a vendor wants twenty minutes to demo their AIOps for Security platform, and you suspect — correctly — that twenty minutes is exactly enough time to be impressed and not nearly enough to be safe. The good news is that a respectable trial does not need a budget line, a lab, or a quarter of engineering time. It needs five evenings, one alert category, a spreadsheet, and the discipline to write your numbers down before the vendor writes theirs. This is that plan, step by step, friendly to teams of two.
Before evening one: assemble your AIOps for Security test kit
Gather the boring parts first, because every later step leans on them:
- An export of your team’s past alert dispositions — every close decision is already a click in the logs, so this is usually a support ticket to whoever owns the tooling.
- One alert category, chosen deliberately. Phishing is the classic pick because volume is high and verdicts are checkable.
- Two people willing to label: one to judge, one to keep the labels honest.
- A spreadsheet you control, with a column you will fill in blind.
- Written confirmation that the product has a genuine observe-only mode. Ask in writing; a surprising number of agents begin acting the day they are installed, and one large vendor’s phishing agent does exactly that — its documentation shows no observe mode as of October 2026.
Evening one: count what is real
Open the disposition export for your chosen category and count the genuine threats among the total. If your mailbox triage closed 4,000 phishing reports last quarter and 90 were real, your base rate is 90 in 4,000 — write it down, because that humble fraction governs everything that follows. Rarity is the silent tax on every detector: the scarcer the truth, the more each false alarm erodes the value of a positive. Teams skip this step constantly, then wonder why a tool that looked brilliant on the vendor’s data produces a slush pile on theirs. Your evening-one number is the antidote: it is the one figure in the whole exercise that no salesperson can reinterpret for you. One category now, a second next month — that sequencing matters, because pooled averages are how weak tools hide, and a team that counts category by category can never be pooled at.
Evening two: label a blind batch
Now the labelling. Pull a sample of recent cases — the floor most evaluation ladders set for recommendation-level claims is 200 labelled alerts carrying at least 30 genuine threats — and have your judge mark each one true or false without seeing what the software says. Blind is the operative word: if the judge peeks at the machine’s verdict, the labels learn to agree with it, and every number downstream becomes flattery. Expect the session to be slower than anticipated and duller than hoped; that is the price of evidence, and it is cheap at double the rate. By the end of the evening you own something most buyers never possess — a small private truth against which any vendor claim can be checked. When the two of you disagree on a borderline case, resolve it with a third reader rather than a coin flip, and log the disagreements; a high disagreement rate is itself a finding, because it marks the category as one where even humans argue and a machine deserves extra scrutiny.
Evening three: let the AIOps for Security candidate shadow the queue
With observe mode confirmed, point the tool at live traffic and let it rank and score while your team keeps working as before. Nothing it says may trigger an action this week; the tool is a passenger, not a driver. Capture its score on every case your evening-two labels already cover — timestamp, case number, score, nothing fancy — and resist the urge to intervene when it disagrees with a colleague, because disagreements are data. A side benefit shows up here for free: the score distribution tells you how chatty the model is about borderline cases, which is exactly where auto-close policies go to fail. Keep the terminal running quietly in the background all week if you can; a longer shadow window costs nothing and catches the mid-week behaviour the demo never shows. Teams that already work with clinical-style decision support will recognise the shape of this evening; the logic mirrors how AI earns a role in cardiology, where a model must prove itself against expert-labelled ground truth before anyone lets it near a patient pathway.
Evening four: turn your counts into a bound
This is the evening that separates a trial from a demo. Take the agreement table you have built and compute two things. First, precision on the closes the tool would have made — of everything it wanted to shut, how much was genuinely harmless? Second, and more important, the miss rate: of the genuine threats, how many did it want to wave through? Then convert the miss count into a bound, because small samples lie in both directions. The rule of three is your friend: with zero errors over n cases, it caps the real error rate near 3/n, so a spotless fortnight proves less than it feels like. When errors appear, sample needs inflate quickly — a single miss means roughly 236 reviewed cases before a 2% claim can stand, and a second miss lifts the requirement to 313.
A worked example makes it concrete. One published triage evaluation blind-labelled 1,240 emails, found 71 genuine threats, and watched the agent miss two of them. The headline miss rate reads 2.8%, and the 95% upper bound lands near 8.6% — which fails a conventional 2% auto-close gate outright, despite the same agent closing with 99.8% precision. Almost nothing it shut was real; that was never the question. The question was what it let past, and the honest bound said “not proven”.
Evening five: decide, and write the gates down
Decision time, and the scoring approach keeps it honest. Rate the candidate across the criteria that actually predict trouble — evidence quality, per-category visibility, observe-mode support, data-handling defaults, reopen ergonomics — scoring each zero to three and weighting by what your team can tolerate. The published scheme tops out at 27 points: a total above 14 earns a continued shadow run, 9 to 14 justifies a pilot with a fixed end date, and anything below 9 means declining without drama. Whatever you decide, write the gates into a short document anyone can read: which action class is permitted, what miss bound must hold in which categories, and who signs. Add a review date too, because the arithmetic you did this week ages — the mail mix shifts, the attackers adjust, and the bound you proved in spring says little about winter. Teams who skip the document rediscover its contents during their first incident, at considerably worse rates.
What the five evenings buy you
Two outcomes, both wins. Either you hold a small set of numbers you computed yourself — base rate, blind labels, a bound, a gate — and a vendor conversation conducted entirely in evidence, or you hold a documented reason to wait, which is a perfectly respectable purchase decision. Both beat the alternative, which is a pilot that begins acting before anyone measured anything.
It also leaves your team sharper for the next round. The labelling instinct transfers to adjacent tooling, the bound arithmetic transfers to any rare-event problem, and the shadow-mode habit generalises well beyond security — the same staged trust appears in digital cardiology workflows and in how a cardiac telemetry unit escalates a reading it does not fully trust. When you want the full framework behind the evenings — autonomy rungs, gate values, adversarial drills — the AIOps for Security atlas page at xOps School collects it in one place, and it pairs well with this plan as the reference half of your trial kit.
Five evenings, one category, no risk to the queue, and a paper trail your auditors will actually enjoy reading. The vendors will still be demoing in PowerPoint next quarter; you will be the one holding a bound.
