Why Point-in-Time Assessments Fail and What Must Replace Them

Most organizations continue to rely on point-in-time assessments to manage security and third-party risk. This includes annual vendor reviews, security questionnaires, SOC 2 reports and one-time technical evaluations. These assessments are familiar, auditable and deeply embedded in how procurement and risk teams operate. They are also increasingly misaligned with how modern software actually behaves.

A point-in-time assessment is essentially a picture. It’s useful at the moment it’s taken, but it cannot identify risks later. That’s like photographing your car in the driveway twice a week. The photos show the car at specific points in time, but they cannot tell you what occurred in between. At a time when code ships daily, infrastructure evolves continuously and data-sharing relationships expand quietly, a SOC 2 assessment report from January says very little about what is running in August.

Why these methods persist despite their flaws

Businesses are not relying on point-in-time assessments because they work. They continue using them because they are easy. It’s easy to collect, easy to file and easy to defend in an audit. The regulatory frameworks organizations rely on, such as SOC 2, ISO 27001, HIPAA, GLBA and NYDFS, operate on set review periods, freeing teams from reporting changes throughout the year.

Point-in-time assessments provide a point-in-time snapshot. They cannot provide the around-the-clock verification needed today. 

Consider the following:

• SaaS Vendors: A typical SaaS vendor may ship code multiple times per day, so by the time a SOC 2 Type II report reaches a customer, the audit window has usually been closed for three to six months, and the environment being described no longer exists.
  
  
• Sub-Processors or Third-Party Companies. Businesses today are increasingly using sub-processors or third-party companies to help deliver products or services. This is especially common with AI features, which introduce silent fourth-party data flows that never appear in the most recent questionnaire. Think OpenAI, Anthropic, Pinecone and Snowflake.
  
  
• Configuration Drift: At one point in time, a business can accurately report that it is enforcing MFA on all of its admin accounts, only to see that control silently lapses 90 days later. And despite this change, there is no triggering event for the customer.

And let’s not forget that point-in-time assessments describe how systems are supposed to work, not how they behave in runtime reality. They cannot show where data flows, which APIs are called, or which third parties have access. They also cannot report what new features, dependencies, analytics tools and AI services have been introduced. It’s within this gray area that risk lives and thrives.

Attestation vs. evidence

Most third-party risk management programs today collect attestations (claims) and treat them as evidence (observations). That is a category error.

Real evidence occurs in runtime. Think network traffic patterns, API call graphs, data egress destinations, authentication logs, dependency manifests and the actual set of subprocessors receiving customer data. What matters is not whether a vendor has a policy, but whether what’s happening in a real-time environment matches it. That gap is where breaches live.

Rapid AI adoption has exacerbated matters. A product team can integrate a new LLM provider in a single afternoon, launching new data flows that no questionnaire captured. Third-party and fourth-party breaches happen on a regular basis, and in nearly every case, the victim organization had current, signed attestations from the compromised vendor. Now consider that the average enterprise manages hundreds to thousands of vendors. Conducting annual deep-dive assessments across a footprint of this size is economically infeasible. That explains why most programs quietly tier 90% of vendors into what’s called a “light-touch bucket” that receives no meaningful scrutiny.

Two cases that define the problem

In March 2026, threat actor TeamPCP published backdoored versions of the widely used LiteLLM Python package after compromising its CI/CD pipeline. With three million daily downloads and direct access to multiple AI service providers, the malicious payload executed automatically on Python interpreter startup, simply running pip install was enough. No questionnaire, SOC 2 report, or attestation would have flagged it. The compromise lived and died inside the gap between assessment cycles.

The Snowflake breaches of 2024 illustrate the same failure at the other end of the sophistication spectrum. Attackers used stolen credentials to access accounts across 160 organizations that had not enforced MFA. Every victim had a current, signed attestation from Snowflake. What no attestation captured was the runtime reality that MFA was not being enforced on the accounts that mattered.

Together, these examples show that point-in-time failures occur at both ends of the spectrum, from sophisticated AI supply chain attacks to the most mundane configuration gaps imaginable.

What must replace it

The tools to observe runtime reality now exist. Advances in AI and analytics make it possible to continuously monitor technical behavior, detect meaningful drift and spot risk in real time. 

These four elements are key.

The first is continuous observation of vendor technical behavior that directly ingests telemetry rather than asking vendors to self-report. This includes Single Sign-On (SSO) and Cloud Access Security Broker (CASB) data, egress monitoring, Domain Name System (DNS) telemetry, SaaS management platforms and increasingly agentic inspection of vendor environments.

The second is event-driven risk triggers. Rather than relying on calendar-driven review cycles, these triggers fire on real changes, such as a new subprocessor detected, a new data flow observed, a certificate change, an ownership change, or a breach disclosure.

The third is structured, composable controls that replace the massive annual questionnaire. The idea is that the unit of assessment should be a specific control tested against observable evidence, not a 400-question spreadsheet that teams fill out once a year.

The fourth is a shift in the vendor relationship model itself, where vendors provide living profiles with standing evidence and customers subscribe to monitoring rather than repeatedly re-collecting the same artifacts. This is both more accurate and radically cheaper per vendor.

When these shifts take hold, risks begin surfacing in near-real time, not during an annual review. A vendor that quietly adds a new AI subprocessor on a Tuesday generates a signal that same week, not eleven months later. Security and procurement teams stop being document-chasers and become exception-handlers. And audit and regulatory posture actually improve because the organization can demonstrate continuous control validation rather than a once-a-year snapshot.

The path forward

The industry has spent twenty years perfecting a system for validating vendors at a single moment in time. But that system was built for a world where software changed quarterly and data stayed in one place. Neither condition holds anymore.

The organizations that adopt continuous, evidence-based oversight will spend the next decade managing risk as it actually behaves rather than as it was described on a form in January. The ones that continue using outdated practices will keep discovering, after the fact, that the breach lived exactly where it always does, in the gap between the last assessment and the moment something went wrong.