Skip to main content

Enterprise Agent workloads

An Enterprise Agent workload measures agents doing business work instead of coding. Each task set holds one realistic case. At each concurrency level, every agent works its own copy of that case at the same time, and a grader scores each finished copy for correctness and quality.

Enterprise Agent selected in Agent Config

Choose an Enterprise Agent workload​

  1. In Step 1, Agent Config, select Enterprise Agent.
  2. In Task set, choose one of the three enterprise task sets.
  3. Task codes fills in with the task set's one case. You cannot add or type another case.
  4. Complete Steps 2 to 5 as for any run. See Configure an Agentic Bench run.

Enterprise task sets

Each enterprise task set is its own workload. Results from different task sets are never averaged or compared with each other, and never compared with Coding Agent results.

The three task sets​

Task setThe caseWhat it stressesShared services
Enterprise: Contract ReviewA bundle of 10 commercial contracts, about 122,000 tokens in all. The agent reviews it against the CUAD clause categories and quotes every finding from the text.Prefill and the KV cache: every agent sends a very long prompt.None
Enterprise: Document RedactionTwo generated business documents of about 512,000 characters each, an HR letter and a support ticket, with personal data to find and mask.Agent fan-out and host CPU.None
Enterprise: Email Threat AnalysisFive emails, four phishing and one legitimate, each triaged with about 290 retrieved prior campaigns in context.Retrieval and the CPU reranker.A vector search index, a text embedder, and a reranker. All three run on CPU.

Email Threat Analysis selected

Case time limits​

Task setCaseAgent limitGrading limitSetup limit
Contract Reviewcontract-review-128k-case-00211 h 40 min20 min10 min
Document Redactioncase-00291 h 30 min20 min10 min
Email Threat Analysisemail-threat-analysis-case-00511 h 40 min3 min10 min

Enterprise SLA and validation

How an Enterprise run differs from a Coding Agent run​

BehaviorCoding AgentEnterprise Agent
TasksOne or more task codes you choose.The one case of the task set.
Trials at a levelTasks × trials per task, raised to at least the level.The larger of Trials per task and the level. For example, with 3 trials per task: 3 copies at 1 agent, 8 at 8 agents, 32 at 32 agents.
Model server between levelsStays up.Restarted before every level, outside the measured time, so each level starts with an empty prompt cache. Copies share the prompt cache only within their own level.
Shared servicesNone.Email Threat Analysis starts its three services, and restarts them before every level.
Default minimum pass rate85%75%
Default minimum GPU busy ratio60%40%
QualificationPass rate and GPU busy ratio.Pass rate, GPU busy ratio, and the enterprise gates below.

Enterprise SLA defaults

Enterprise gates​

A level of an Enterprise run counts as a valid operating point only when every gate below passes and the GPU busy ratio meets its minimum. The report lists each gate with its measured value and its threshold. A gate that cannot be computed is shown as not evaluable, never as passed.

Integrity gates​

These check that the measurement itself is sound.

GatePasses when
Fleet validityEvery expected agent was ready at the same time, and the level was not marked invalid.
Worst node coverageTelemetry coverage on the worst machine and source is at least 95%. The worst value is used, never the average.
Samples per seriesThe thinnest telemetry series holds at least 2 samples.

Completion gates​

GatePasses when
Case pass ratePassed cases divided by graded cases is at least your minimum pass rate (75% by default). At least 2 cases must be graded.
Observation completionAt least 98% of requested cases reported a result.
Infrastructure successAt least 95% of requested cases were not lost by the platform.
Grader timeout rateAt most 2% of requested cases could not be graded in time.

Quality gates per task set​

The averages are over the graded cases of the level.

Task setGatePasses when
Contract ReviewCategory precisionAt least 0.917 of the clause categories called present are present.
Contract ReviewCategory recallAt least 0.433 of the present clause categories were found.
Contract ReviewCitation precisionAt least 0.4 of the quotes match a true clause span.
Contract ReviewCitation recallAt least 0.037 of the true clause spans are matched by a quote.
Document RedactionLeaked valuesNo high-severity value is left unmasked.
Document RedactionIntegrity failuresNo document has text altered outside its masks.
Document RedactionMissing documentsEvery document has a redaction returned.
Document RedactionNon-personal text keptMore than 0.5 of the non-personal text is kept in the worst document.
Document RedactionCategory accuracy over baselineCategory accuracy is above the share of the most common category.
Email Threat AnalysisUnjudged emailsEvery email has a verdict.
Email Threat AnalysisFabricated citationsNo cited evidence is missing from the email.
Email Threat AnalysisUncited threatsEvery threat verdict cites evidence.
Email Threat AnalysisBalanced accuracyThe mean of threat recall and clean-email retention is above 0.5, better than chance.

Sizing advice​

  • Contract Review needs a model with a long context. Contract Review sends about 122,000 tokens per agent. Of the verified models, Qwen3.5 9B, Qwen 3.8 27B and Qwen 3.8 27B FP8 hold it, at 262,144 tokens. Qwen3-4B, Qwen3-14B and Qwen3-32B hold 40,960 tokens as Agentic Bench serves them, which is too short. For an unverified model, check that its context length is at least 131,072 tokens.
  • Contract Review is limited by KV cache memory. Each agent holds a very long context, so the number of agents one GPU serves is far lower than for coding tasks. Start with levels such as 1, 2, 4, 8, and add higher levels only after those pass.
  • Email Threat Analysis uses host CPU for its services. The embedder and the reranker run on the machines' CPUs alongside the sandboxes. Leave CPU headroom, or use Pinned sandbox CPUs to keep agents and services apart.
  • Document Redaction uses host CPU for agent fan-out. Watch the CPU charts in the report before you add levels.
warning

An Enterprise model call that receives no output for 1,400 seconds fails. When a level holds more agents than the model server can serve, requests wait in its queue, and calls that wait that long fail. In a staging validation on two NVIDIA L40S GPUs, Contract Review passed every case at 1 and 8 agents, and at 32 agents more than half the cases failed this way. Choose levels the model server can serve, or add GPUs.

Example: Email Threat Analysis on two machines​

This example configures an Enterprise run with an unverified model and a prefill and decode split.

StepSettingValue
1. Agent ConfigWorkloadEnterprise Agent
Task setEnterprise: Email Threat Analysis
2. Model ConfigModelmeta-llama/Llama-3.1-8B-Instruct, pasted as a Hugging Face ID
VersionvLLM 0.32.0
Engine argstool_call_parser = llama3_json
3. HardwareCluster groupTwo machines with 2 GPUs each
GPUs per node, tensor parallel2 and 2
Split prefill and decodeOn, 1 prefill machine
4. LoadConcurrency levels1, 8, 32
5. SLAThresholdsDefaults: 75% pass rate, 40% GPU busy

Two machines selected for an Enterprise run

This example shows the configuration screens. It is not a tested result. For tested settings, see Tested configurations.

What's next​