Catch Advisors
cyber

Cybersecurity Vendor Evaluation: 20 Questions to Ask Before You Sign

Buying cybersecurity is one of the hardest procurement decisions an IT leader makes.

The technical complexity is real. The sales process is uniquely aggressive. Vendors have strong financial incentives to overstate threats and understate their limitations. And unlike most technology purchases, a bad cybersecurity vendor doesn’t just waste your money — it leaves you thinking you’re protected when you’re not.

Every year, we see mid-market organizations that paid for managed security services, got breached anyway, and discovered post-incident that their vendor’s coverage had significant gaps. The pattern is consistent: the buyer asked surface-level questions, the vendor gave polished answers, and nobody asked the questions that would have revealed the truth.

This guide gives you the 20 questions that separate credible security vendors from vendors selling confidence without capability. Use them in vendor briefings, RFP processes, and renewal conversations.

Before You Ask Anything: Know What You’re Buying

Cybersecurity vendors use terminology inconsistently. “SOC,” “MDR,” “XDR,” “MSSP,” and “managed detection and response” can mean very different things from vendor to vendor.

Before evaluating vendors, get clear on what capability gap you’re trying to fill:

  • Detection and response: Do you need someone monitoring your environment 24/7 and responding to threats? That’s MDR.
  • Firewall and network security management: Do you need someone managing and monitoring your perimeter? That’s closer to traditional MSSP.
  • Endpoint protection: Do you need EDR deployed and managed? That may be EDR-as-a-service.
  • Vulnerability management: Do you need regular scanning and remediation guidance? That’s a distinct service.
  • SIEM/log management: Do you need centralized log collection and alerting? That may or may not include response capability.

Many vendors sell “all of the above” under a single label. The questions below will help you understand what you’re actually getting.

The 20 Questions

Detection and Response Capability

1. What is your mean time to detect (MTTD) and mean time to respond (MTTR)?

Ask for specific numbers, not ranges, and ask for them segmented by threat type (ransomware, phishing, credential compromise). Industry benchmarks for MDR services are typically MTTD under 24 hours and MTTR under 4 hours for high-priority incidents. Be skeptical of vendors who can’t provide these metrics or who cite only their best-case numbers.

2. What telemetry do you actually ingest?

Effective detection requires broad telemetry: endpoint logs, network flow data, identity logs (Active Directory, Entra ID), cloud workload logs, email security events. Ask vendors to enumerate exactly what they ingest from your environment. Some “MDR” services ingest only EDR telemetry and call it comprehensive coverage. That’s not comprehensive.

3. How do you handle detections that don’t match known threat signatures?

This question separates behavioral detection capability from signature-based tools with a human helpdesk attached. Look for answers that describe machine learning models trained on environmental baselines, anomaly detection, and analyst workflows for investigating novel behavior. If the answer is essentially “we check against our threat intelligence database,” that’s signature-based detection with limited coverage of unknown threats.

4. Walk me through what happens when you detect a ransomware precursor in my environment at 2 AM.

Ask for the specific workflow, step by step. Who gets paged? What authority do analysts have to take automated action? How do they contact you? What decisions require your approval vs. autonomous action? Vague or generalized answers here are a red flag. A vendor with real operational maturity will be able to describe this workflow in detail.

5. How many analysts are in your SOC? What is the ratio of analysts to client environments?

SOC analyst-to-client ratios vary widely. High-quality MDR providers typically maintain ratios under 1:50 (one analyst per 50 client environments or fewer). Discount MSSPs sometimes run ratios of 1:200 or higher, which means your environment isn’t getting meaningful human attention. Ask specifically whether your account will have dedicated analysts or shared pool coverage.

Coverage and Architecture

6. What is explicitly out of scope for your service?

Ask vendors to describe what they don’t cover. This is more revealing than asking what they do cover. Common exclusions: OT/ICS environments, specific cloud platforms, on-premises infrastructure of certain types, custom applications, or specific endpoint operating systems. If you have Linux servers, industrial control systems, or a significant AWS footprint, verify those are in scope.

7. How do you handle cloud environments? Which cloud platforms do you have native integrations with?

Cloud coverage varies dramatically. Ask specifically about AWS, Azure, and GCP. Ask whether they ingest CloudTrail, Azure Activity Logs, and GCP Audit Logs by default. Ask about coverage for serverless functions, container workloads, and Kubernetes. Many vendors have mature on-premises coverage and weaker cloud coverage — this matters if you’re cloud-heavy.

8. What is your deployment timeline and what’s required from my team?

Understand the onboarding investment before you commit. Some MDR services require 2-4 weeks of professional services engagement, agent deployment across all endpoints, firewall integration, and SIEM tuning. Others can be up in days. Neither is inherently better — but you need to know what you’re committing your team to.

9. Do you provide active response (taking action in my environment) or passive monitoring (alerting me to take action)?

This is the most important architectural question. Some services monitor and alert; others have the ability to isolate endpoints, block network connections, and take containment actions autonomously. In a ransomware scenario, the difference between active and passive response can be the difference between a contained incident and a full encryption event. Know what you’re buying.

Threat Intelligence and Research

10. Do you produce original threat intelligence, or do you consume third-party feeds?

Vendors with original threat research capability will typically have named threat hunting teams, threat intelligence publications, or CVE disclosures to their credit. Those consuming only commercial feeds will have vaguer answers. Original threat intelligence capability doesn’t guarantee better detection — but it’s a signal of organizational investment in security research vs. resale of services.

11. How is threat intelligence operationalized into your detection rules?

This question gets at operational maturity. The answer should describe a process: how new TTPs (tactics, techniques, and procedures) from threat intelligence get translated into detection logic, how often detection content is updated, and how that update process is quality-controlled. If the answer is “our team updates rules as needed,” push for specifics on frequency and process.

Compliance and Certification

12. What certifications does your SOC hold, and can I see the underlying audit reports?

Relevant certifications include SOC 2 Type II (ask for the full report, not just the attestation letter), ISO 27001, and for specialized sectors, FedRAMP, HITRUST, or StateRAMP. Certifications matter less than what’s in the underlying reports — scope exclusions, control exceptions, and auditor observations tell you more than the certificate itself.

13. How do you support my compliance requirements specifically?

If you’re subject to HIPAA, PCI DSS, CMMC, or state privacy regulations, ask specifically how the vendor’s service supports your obligations. Ask for examples of documentation they provide to support compliance audits (log reports, incident reports, coverage attestations). Some vendors have compliance-specific service tiers; others treat compliance support as an afterthought.

Incident Response

14. What is your incident response retainer policy?

Clarify whether IR is included in the base service or billed separately. Some MDR vendors include IR hours up to a specified threshold; others charge hourly rates for anything beyond containment. Understand the financial exposure before you’re in an incident.

15. Have you responded to ransomware incidents? Can I speak to a reference customer who experienced an incident?

This is the most revealing reference question you can ask. Vendors who have been through real incidents with real customers — and retained those customers — have demonstrated operational performance under pressure. References from uneventful deployments tell you much less. Ask specifically for customers who experienced and recovered from incidents.

16. How do you coordinate with law enforcement and legal counsel during an incident?

Sophisticated incident response often involves law enforcement notification, legal hold obligations, and forensic preservation requirements. Ask whether the vendor has established relationships with the FBI, CISA, or sector-specific ISACs. Ask how they handle evidence preservation for potential litigation. This matters more than most buyers realize until they’re in the middle of an incident.

Commercial and Relationship Terms

17. How are pricing changes handled at renewal? Is there a cap on year-over-year increases?

Multi-year cybersecurity contracts without price escalation caps can become expensive surprises. Ask what drove pricing changes for existing customers at renewal over the past two years. Push for contractual caps if the market permits.

18. What are the contract exit terms if I’m not satisfied with the service?

Ask specifically about cure periods (how long they have to fix a service deficiency before you can exit), termination for convenience terms, and data return procedures. A vendor confident in their service quality will agree to reasonable exit terms. Aggressive lock-in provisions are worth scrutinizing.

19. Who is my named point of contact, and what is their role?

Understand whether you’ll have a dedicated Customer Success Manager or Technical Account Manager, what their responsibilities are, and how often you’ll have structured reviews. Security services without regular operational review meetings tend to drift out of alignment with your environment.

20. What does success look like after 12 months with your service?

This question reveals how the vendor thinks about outcomes vs. inputs. Strong answers describe measurable improvements: reduction in phishing click rates, MTTD/MTTR metrics, coverage expansion, compliance posture improvements. Weak answers focus on process commitments (“we’ll run quarterly reviews”) without outcome commitments. Hold vendors to outcomes, not activities.

How to Use These Questions

Not every question is equally important for every vendor evaluation. Prioritize based on what you’re buying:

For MDR/Managed Detection and Response: Questions 1-9 are critical. Pay particular attention to Questions 4 (the 2 AM scenario), 5 (analyst ratios), and 9 (active vs. passive response).

For MSSP/Managed Security Services: Questions 6, 8, 12, 17, and 18 should get the most attention. MSSP contracts have notoriously complex scope exclusions and commercial terms.

For Cyber Insurance Preparation: Questions 12, 13, and 16 are most relevant if your primary goal is demonstrating adequate security controls to your insurer.

For Incident Response Retainer: Focus on Questions 14, 15, and 16 above all others.

The Difference a Technology Advisor Makes

Running a structured cybersecurity vendor evaluation is time-consuming, technically demanding, and requires market knowledge that most IT teams don’t maintain in-house. Vendor sales teams know this — and they optimize their pitches accordingly.

Working with Catch Advisors gives you a material advantage in this process. As a vendor-neutral technology advisor, we:

  • Know how vendors perform operationally, not just how they present in sales cycles
  • Have benchmarked contract terms across dozens of MDR, MSSP, and security services contracts
  • Run the evaluation process so your team can focus on current operations
  • Advocate for your interests in contract negotiations — with no vendor relationship conflicts

We work across 580+ provider relationships in cybersecurity, networking, cloud, and communications. Our advisory services are typically vendor-funded, meaning there’s no direct cost to you.

Schedule a free consultation with Catch Advisors →

Summary

Buying cybersecurity is hard. The market is crowded, the terminology is inconsistent, and the sales process is designed to create urgency and obscure limitations.

These 20 questions won’t make the process easy, but they will make it honest. Push for specifics. Verify claims. Call references who experienced actual incidents. Read the contract before you sign it.

The best cybersecurity vendors welcome hard questions — because they know they can answer them.