Artificial Intelligence
How Octave built a secure multi-tenant AI foundation on Amazon Bedrock
In quality management for regulated industries, the institutional memory that resolves a problem is almost always buried somewhere in a customer’s history. A similar defect investigated last year, a Corrective and Preventive Action (CAPA) that fixed the root cause, an audit finding that flagged the same gap. That knowledge sits in billions of documents no human can search by semantic similarity across languages and formats, so every investigation starts from scratch. Octave Reliance has helped each organization collect documents across multiple systems to help them search more easily and keep track of everything they have. The Octave Reliance team saw unlocking this history as its single largest opportunity to help customers work faster and stay compliant. The obstacle was equally clear: complaint records, audit findings, and corrective actions carry the platform’s strictest isolation requirements, and those were exactly the records the AI would need to read. In a platform serving over 14,000 customers across medical devices, pharmaceuticals, and food manufacturing, a single inadvertent disclosure from one customer to another would carry regulatory consequences far outweighing any AI benefit.
Octave operates across 180 countries, serving more than 60% of the Global Fortune 500. Its cloud-native quality management system, Reliance, supports over 650 enterprise customers with over 40 applications, from complaint management and CAPA tracking to audit management and supplier quality. Everyone generates regulated data that AI could reason over. Each customer is a separate tenant and no data can cross a tenant boundary.
Faced with that, the Reliance team made an unusual call: rather than shipping one AI feature, it spent its first cycle building a foundation every future feature would sit on. That foundation has since cut deployment time per use case by 50 percent. If you are bringing AI to a software as a service (SaaS) platform where the data is valuable, but isolation is non-negotiable, the reasoning travels beyond quality management. In this post, we walk through the four tensions that shaped the design and the three-layer foundation Octave built with Cohere Embed v4 on Amazon Bedrock. We then follow the pipeline that embedded 2 billion documents and the Complaint Triage Assistant that proved it in production.
Why a foundation, not a feature
Envisioning workshops with AWS surfaced four high-impact use cases: complaint summarization and triage, audit readiness monitoring, incident narrative generation, and predictive quality risk scoring. They also surfaced the insight that shaped everything after dozens of applications, each with different document types and workflows. Building AI for one in a silo meant rebuilding the infrastructure for the next.
So, the team inverted the plan: build the horizontal platform first, then cut vertical feature slices into it. Complaint summarization and triage went first, to prove the architecture before the rest followed.
The four tensions
Before selecting a single model, the team spent its time on four problems that pulled in different directions. None could be traded away.
Making data available without risking inadvertent disclosure. Complaint records, CAPAs, standard operating procedures (SOPs), and audit findings belong to one tenant and must be unreachable by another. Not even an errant AI query could cross. Vector store, retrieval logic, and generation context all had to be scoped to a single tenant.
Executing agents in isolation. The permissions governing an agent’s tools, data stores, and model endpoints had to be scoped to the requesting user rather than a shared service role, available only during execution, and respectful of that user’s access rights. A shared agent pool with static credentials failed all three.
Reading content that cannot be trusted. Agents here ingest complaints, email attachments, scanned PDFs, and images from outside the organization. The OWASP Top 10 for large language model (LLM) Applications (2025) names the threats that apply directly: prompt injection through ingested content (LLM01), sensitive information disclosure (LLM02), improper output handling (LLM05), and excessive agency (LLM06). Hidden instructions in an attached PDF are not hypothetical at this volume.
Embedding 2 billion documents without a quarter-long wait. Reliance hosts more than 2 billion documents, with individual tenants storing millions of Word files, PDFs, scanned forms, spreadsheets, images, and email. The team committed to onboarding even the largest tenant within a single working day. Because these documents inform regulatory decisions, quality could not be traded for throughput. A missed complaint erodes trust in the whole system.
Where the architecture landed
Holding four tensions at once narrowed the options considerably. Amazon Bedrock addressed them as one platform: multi-model access through a single API, the AWS Region residency and AWS Identity and Access Management (IAM) governance the AWS account already enforces, and Guardrails as production content controls rather than something to build.
The foundation resolves into three layers, shown in Figure 1. Data preparation embeds quality documents with Cohere Embed v4 on Amazon Bedrock and writes them to a per-tenant vector store on Amazon OpenSearch Service. Secure execution passes a user request through AgentCore Gateway, a capability of Amazon Bedrock AgentCore, which applies authentication and policy before handing it to an isolated microVM session that calls an Anthropic Claude model on Amazon Bedrock. Wrapping both, content protection applies Amazon Bedrock Guardrails twice: input filtering on the way in, output filtering before anything reaches the handler. Retrieval, execution, and screening stay in separate stages, so each can be enforced and evidenced on its own.
Choosing how to represent the data
Nothing else worked until the documents were retrievable. That meant accurately, across languages and formats, and not across a tenant boundary. The team evaluated embedding models against its own corpus rather than public leaderboards. The deciding questions were specific to this data: what a photo of a cracked housing has to do with the paragraph describing it, and a German complaint with an English CAPA.
Cohere Embed v4 on Amazon Bedrock won on five counts and they weren’t independent of each other. What was decided first was how much of the pipeline was made unnecessary. Quality records carry photos of defective products, scanned inspection forms, and annotated diagrams alongside text. Embed v4 places both modalities in one vector space. A photo of a cracked housing and the paragraph describing that defect are retrievable from a single query, with no separate pipeline per modality. Native multilingual support does the same work along the other axis: documents arrive in multiple languages, often inside one tenant’s corpus, and need no translation preprocessing. Two pipelines the team never had to build, and two fewer places for a document to fall out of retrieval.
The remaining three counts settled where the vectors could live and what they would cost. Customer data can’t leave its AWS Region, even for inference, so in-Region model availability on Amazon Bedrock was a precondition rather than a convenience. Embed v4 supports in-Region inference in the US East (N. Virginia) Region (us-east-1), Europe (Ireland) Region (eu-west-1), and Asia Pacific (Tokyo) Region (ap-northeast-1), covering three of Octave’s regional compliance requirements. Model and feature availability varies by Region and changes over time. For current coverage, see Regional availability by models in the Amazon Bedrock User Guide. Within each Region, Amazon OpenSearch Service costs scale linearly with dimension count. That made Embed v4’s configurable output dimensions of 256–1,536 worth testing rather than accepting by default. Testing against representative production queries showed 512 dimensions holding target accuracy while cutting storage roughly 66 percent against the 1,536 default. The 128K token context window paid off elsewhere entirely: chunks of roughly 4K tokens pack many to a request, which is where the onboarding throughput came from.
Keeping tenants apart at runtime
The harder question was where to enforce tenant separation. Application code would have worked, but it would have meant every future feature team getting it right on every code path, where one missed check risks inadvertent cross-tenant disclosure. The team pushed isolation into the runtime instead. AgentCore runtime, a capability of Amazon Bedrock AgentCore, enforces separation below the application layer, and because it stays framework- and model-independent, the agent framework and models chosen for the Complaint Triage Assistant were not constrained by the isolation decision.
Each user session runs in a dedicated microVM with its own CPU, memory, and filesystem, stopped and sanitized when the session ends. That is hardware-level separation designed to hold regardless of how the agent behaves. AgentCore carries the authenticated user’s identity through the whole execution chain: per-user OAuth token vaults verify the agent acts for that specific user rather than a shared service identity, and credentials are scoped per-agent and live only during the session. AgentCore Gateway authorizes at the tool, operation, and parameter level. Automated reasoning provides formal verification against the defined rules. It’s the same discipline behind IAM and Amazon Simple Storage Service (Amazon S3) policy validation. Figure 2 traces that separation across two tenants.
A complaint submitted by a quality handler at Tenant A therefore reaches only Tenant A’s token vault, tools, and sandbox in AgentCore Code Interpreter, a capability of Amazon Bedrock AgentCore. Tenant B’s session has no path to any of them.
Treating ingested content as untrusted
This tension had no clean answer. A complaint attachment is untrusted input, but it’s also the thing the agent exists to read, so suspicious content couldn’t be rejected. The team settled on defense in depth.
Octave applied Amazon Bedrock Guardrails over the agent logic rather than inside it, at three points in the complaint workflow. The ApplyGuardrail API evaluates text extracted from attachments before embedding, retrieved documents and tool outputs before they enter the agent’s context, and the synthesized brief before the handler sees it. Attachment-derived content is passed with input tagging, marking it as data rather than instruction. That is what makes an injection buried mid-document detectable. Personally Identifiable Information (PII) is masked on the way in.
Detection alone wasn’t sufficient, so bounded agency sits underneath. Whatever an agent is asked to do, AgentCore Gateway still governs which tools it can invoke, and document extraction runs in the AgentCore Code Interpreter sandbox with no network egress and no persistence beyond the session. A payload hidden in a PDF has to defeat every layer, not one.
The effect: detection and masking rates are tracked per tenant, so what the screening caught is something Octave can evidence rather than assert.
Getting 2 billion documents through the pipeline
Ahead of the embedding model sat the harder problem: extracting usable content from Word files, spreadsheets, digital PDFs, scanned pages, photographs, and email threads. Several languages appear, sometimes inside one record. And it had to run fast enough that onboarding took a day rather than a quarter. Figure 3 traces the resulting pipeline end to end.
Document events fan out through Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) into an AWS Step Functions state machine. That state machine orchestrates parsing, chunking, and embedding across concurrent AWS Lambda executions inside a dedicated Amazon Virtual Private Cloud (Amazon VPC). Because parallelism comes from fanning batches across executions rather than scaling one resource, throughput grows without rearchitecting.
Parsing handles the heterogeneity: semantic hierarchy preserved from Word files and spreadsheets, optical character recognition (OCR) applied to scanned PDFs lacking a text layer, embedded images treated as first-class content. Chunks carry metadata: tenant, document type, source application, language, and a content hash. That metadata drives both retrieval filtering and per-tenant namespace enforcement.
The throughput came from packing. Interleaving many 4K chunks into single requests against Embed v4’s 128K context window cut the API calls for a large corpus by about an order of magnitude. In Octave’s internal testing on Amazon Bedrock, Embed v4 delivered 28 million tokens per minute. After tuning batch sizes with Cohere’s solutions architects until memory bottlenecks cleared, the pipeline sustained 1.4 million documents per 24 hours per tenant workload. That is what made the single-working-day commitment real.
The Complaint Triage Assistant
A hospital reports defective syringes to a medical device manufacturer, and the record lands with email, photos of damaged units, and supporting documentation attached. A quality handler has to piece the context together across those formats, work out whether anything like it has been reported before, and find what resolved it. Only then can a priority be assigned, often inside a reporting window measured in days.
The assistant is built with the Strands Agents SDK, open-sourced by AWS, which composes agentic workflows as graphs, with nodes as tool invocations or model calls and edges as the data dependencies between them. AgentCore runtime runs that graph and isolates it per session, and five stages carry a complaint from raw attachments to a finished brief.
Document extraction comes first. PDFs, images, email attachments, and videos all enter AgentCore Code Interpreter, which parses each artifact into a structured intermediate representation. A Claude model on Amazon Bedrock then summarizes the record, and that summary, rather than the raw multi-thousand-token source, becomes the retrieval query. Retrieval runs it against the tenant’s OpenSearch index of Embed v4 embeddings, surfacing similar past complaints and their CAPAs, with AgentCore Gateway enforcing that only authorized namespaces are reachable.
What comes back is not trusted on similarity alone. A separate model call scores results by confidence and discards matches that share vocabulary but differ in root cause. What survives goes to synthesis, where Anthropic Claude Sonnet on Amazon Bedrock assembles the output: key facts, sentiment, urgency indicators, the closest historical records with their outcomes, and a recommended triage priority.
Completed briefs are cached in Amazon DynamoDB with a per-environment time to live (TTL), so repeated views cost nothing extra. The handler no longer starts from zero, but from institutional memory.
Figure 4 places that engine behind a private API boundary. A portal request reaches AgentCore runtime through a private Amazon API Gateway and a Lambda function, and retrieval reads the document index through role-based access with pre-signed URLs while attachments stay in Amazon S3. Cached summaries and OpenTelemetry traces exit to the summarization table and the evaluation dashboard.
Figure 5 reduces the same path to a single line, with Guardrails filtering at both ends.
Proving what the agent did
Observability here is not only how the engineering team debugs but also how they integrate with system auditing capabilities to meet compliance requirements. An auditor can ask why a particular complaint was triaged the way it was, and “the model decided” will not survive that conversation.
AgentCore captures OpenTelemetry-compatible traces across every node of the graph: each model call, retrieval, and AgentCore Code Interpreter run. Months later, Octave can show which retrieved documents and prompts produced a given brief. The model invocation logging of Amazon Bedrock adds a per-tenant record of every request and response, scoped through the IAM identity AgentCore propagates, with costs attributing to the initiating tenant in Amazon CloudWatch.
Offline, AgentCore Evals reruns the workflow against expert-annotated ground truth after any change to prompts, models, or retrieval parameters, reporting summarization faithfulness, retrieval precision at K, and output completeness. A prompt that reads better but retrieves worse gets caught before users see it.
Results
The Complaint Triage Assistant has been running in production for three months, and the following numbers reflect that period.
The handler who used to open an investigation with nothing now opens it with the relevant history already assembled. The 4–6 hours Octave measures per investigation is the work the pipeline absorbed: extraction across formats, the hunt for precedent, and the reading it takes to tell a genuinely similar defect from a coincidence of wording.
The foundation also changed how quickly Octave ships. By its own accounting, development cycles for new AI capabilities went from months to weeks. Shared data preparation, secure execution, and retrieval are the three layers of Figure 1. They cut deployment time per use case by 50 percent, because each new capability inherits them rather than rebuilding them. That landed semantic search in Q2, two quarters ahead of the original Q4 target.
Looking back
The decision to build a foundation first was the one the team second-guessed most. Months of infrastructure work is hard to defend when stakeholders want an AI feature they can see. The next round of use cases has borne it out. Two choices carried it: isolation pushed into microVM sessions and identity propagation rather than application code, and embedding models evaluated against Octave’s own corpus, which surfaced the 512-dimension trade-off no benchmark would have shown.
The team also drew a line it chose not to cross. The assistant surfaces context and precedent. The quality handler still makes the call. In an industry where that call carries regulatory weight, that boundary shaped what got automated and what did not.
What’s next
Octave is now extending the foundation across the Reliance suite. Audit readiness monitoring, incident narrative generation, and predictive quality risk scoring are next, each building on the same infrastructure.
If your platform faces a similar tension: valuable data behind hard isolation requirements. Octave’s experience shows the trade-offs are workable. To learn more about the services behind this story, see Amazon Bedrock, the Amazon Bedrock AgentCore documentation, and Cohere on Amazon Bedrock. For related architectural guidance, see AWS vector solutions: Build agentic AI where your data lives and Scaling agentic AI: Enterprise patterns without vendor lock-in.