Evaluating Large Language Models for Enterprise Access Review Decisions

In this paper, we present a real-world benchmark evaluating the ability of modern large language models (LLMs) to perform access review decisions using production-grade enterprise data. Rather than treating access reviews as a simple binary classification task, we evaluate models based on the quality, coherence, and evidentiary strength of their reasoning.

Our results show that frontier LLMs can meet or exceed human-level performance, while also revealing that increased reasoning capacity does not necessarily lead to better outcomes. Finally, we describe the system-level challenges that must be addressed to make these models reliable in practice.