A reference architecture for enterprise GenAI on Google Cloud: Vertex AI, Gemini, vector search, security and LLMOps

Most enterprise GenAI projects don't fail on model quality. They stall in security review, break when a second team tries to reuse them, or become too expensive to scale. A clear reference architecture prevents all three. This post sets out a layer-by-layer design on Google Cloud that we use as a starting point, with the official documentation for each building block so you can verify the details yourself.

The architecture at a glance

Google also publishes its own reference designs, including a RAG-capable generative AI application using Vertex AI and broader AI and ML architecture guidance. Use them alongside the Google Cloud Architecture Framework when you write your design document.

Layer 1: Experience

Keep the user-facing layer thin. It should handle authentication (your corporate identity provider), session state, streaming responses, citations and a feedback control (thumbs up/down plus a reason). Feedback is cheap to collect and becomes the fuel for evaluation later.

Layer 2: Orchestration and agents

This is your application code: prompt assembly, retrieval calls, tool calls and business rules.

• Cloud Run is a simple default for containerised APIs that scale to zero (what is Cloud Run).
• Vertex AI Agent Engine is a managed runtime for deploying and scaling agents (Agent Engine overview).
• Agent Development Kit (ADK) is Google's open-source framework for building agents (ADK docs); LangChain/LangGraph are common alternatives.
• Function calling lets Gemini request structured tool calls that your code executes (function calling). Your code, not the model, decides whether a call is allowed.
• Model Context Protocol (MCP) is an open protocol for connecting models to tools and data sources (modelcontextprotocol.io), useful when many agents need the same tools.

Design rule: side-effect actions (sending, updating, deleting, paying) need explicit authorisation checks and, usually, human approval.

Layer 3: Models on Vertex AI

Vertex AI provides Gemini models plus a catalogue of Google, partner and open models through Model Garden (Google models on Vertex AI). Architecture decisions here:

• Model routing: use a smaller, faster model for classification, routing and grading; reserve larger models for complex reasoning. Make the model name configuration, not code.
• Structured output: ask for JSON against a schema so downstream code is reliable (control generated output).
• Grounding: connect answers to your data or to Google Search where appropriate (grounding overview).
• Embeddings: pick one embedding model per index and record its version; changing it means re-embedding (text embeddings).
• Capacity: for steady, high-volume workloads, evaluate Provisioned Throughput.
• Cost and latency: reuse repeated long context with context caching.

Model versions change over time. Pin versions in production and upgrade through your evaluation gate (see LLMOps below).

Layer 4: Data and retrieval

Choose the vector store based on where the source data lives and who maintains it:

Whatever you choose:

• Keep the raw documents in Cloud Storage (or the source system) as the system of record; the index is derived and rebuildable.
• Store metadata with every chunk: source, section, updated date, and access-control groups.
• Combine keyword and vector retrieval, then re-rank (ranking API).
• Build an incremental ingestion pipeline so updated documents are re-indexed without a full rebuild.

Layer 5: Security and governance

Security reviewers will ask about identity, network, encryption, logging and content risk. Have an answer for each:

1. Identity (IAM): a dedicated service account per component with least-privilege roles; no user keys in code (IAM overview).
2. Perimeter: place sensitive projects inside VPC Service Controls to reduce data exfiltration risk.
3. Encryption keys: use customer-managed encryption keys (CMEK) where your policy requires control of keys, and check which services support CMEK.
4. Secrets: API keys and credentials in Secret Manager.
5. Audit: enable and retain Cloud Audit Logs, including data access logs where needed.
6. Content screening: screen prompts and responses for prompt injection, jailbreaks and sensitive data with Model Armor, and configure Gemini safety filters.
7. Document-level access: enforce ACL filters in the retrieval query so users can only retrieve what they're entitled to read.
8. Responsible AI: review Google's responsible AI guidance for Vertex AI and map risks with frameworks such as Google SAIF, the NIST AI RMF and the OWASP Top 10 for LLM Applications.

For Indian organisations handling personal data, also involve your legal or privacy team on obligations under India's data protection framework (MeitY data protection framework).

Layer 6: LLMOps

LLMOps is what keeps the system good after launch.

• Version everything: prompts, model names and versions, embedding model, chunking settings, and evaluation sets live in Git.
• Evaluation gate in CI/CD: each change runs the evaluation set with the Gen AI evaluation service or your own harness; releases below threshold are blocked. Choose metrics deliberately (determine evaluation).
• Pipelines: automate ingestion, re-embedding and batch evaluation with Vertex AI Pipelines.
• Monitoring: dashboards for latency, error rate, tokens and cost per request, retrieval hit rate, refusal rate and user feedback.
• Model upgrades: run the new version in shadow or on a slice of traffic, compare on the evaluation set, then switch.
• Feedback loop: negative feedback goes into a review queue; reviewed cases are added to the evaluation set.

A minimal request path

# Simplified request path (pseudocode). Each function maps to a layer above.
def handle(request, user):
    screen_input(request.text)                         # Model Armor / custom checks
    route = classify(request.text, model=SMALL_MODEL)  # cheap routing call
    passages = retrieve(request.text, acl=user.groups) # vector search + ACL filter
    passages = rerank(request.text, passages)[:6]
    answer = generate(request.text, passages, model=MAIN_MODEL, schema=ANSWER_SCHEMA)
    screen_output(answer, passages)                    # faithfulness, PII, policy
    log_trace(user, route, passages, answer)           # tokens, latency, cost, sources
    return answer

Design review checklist

Go deeper

Our AI Architect & Cloud Master Track takes you through these layers on Google Cloud: Gemini APIs, Vertex AI Search and Vector Search RAG, Agent Builder and ADK, Gemini multimodal pipelines, Model Armor security and LLMOps, with enterprise capstones. Live on Mon & Fri 10:00–11:30 AM plus Sat 12:00–1:30 PM IST, for 12 weeks.