Two sandwiches and a salad: Architectural recipes for multi-agent system success

The greatest cooking disasters don't come from bad ingredients—they come from following inaccurate recipes. The same is true for building multi-agent systems. You might have the best AI agents in the world, but without the right deployment pattern, you're headed for suboptimal results. 

In our team’s recent exploration of the developer journey for creating multi-agent applications, we outlined the four critical stages of Discover, Compose, Deploy, and Evaluate. Today, we need to address a more fundamental architectural question that underlies this entire journey: How do we balance the inherently probabilistic nature of AI agents with the deterministic requirements of enterprise production environments?  

This is what I call the "determinism sandwich" problem—and solving it might be the most important decision you make when building multi-agent systems. Also, I would caveat right up front, that these patterns are a journey. In some, hopefully not so distant future, they could collapse into fewer (a single?) deployment pattern. 

Architectural models for the developer journey 

As developers move through the four stages of building multi-agent applications, they need to make fundamental architectural decisions that will determine their system's reliability, flexibility, and production-readiness. 

When I talk with developers implementing the Discover → Compose → Deploy → Evaluate workflow, they consistently face architectural questions that go beyond implementation details. I hear questions like: How deterministic should my agent workflow be? How much runtime decision-making should I allow? Where do humans fit in the control flow? And crucially, how will this architecture scale in an enterprise environment with our security and compliance requirements? 

Pick the wrong recipe, and your multi-agent soufflé will collapse (or spill everywhere like a poorly constructed burrito!). These architectural choices are your foundation ingredients - get them right before you even start shopping for fancy agent toppings. 

Three menu options for multi-agent software deployment 

In my work with development teams across industries, I've found that multi-agent architectures fall into three distinct patterns. To make these technical concepts easier to grasp, I've started using food analogies - specifically different types of meal preparation. These culinary comparisons have proven surprisingly effective when explaining architectural choices to everyone from engineers to executives. 

To help visualize these architectural patterns, I've included some back-of-napkin sketches—the kind you might draw while brainstorming over lunch. These rough drawings aren't meant to be production blueprints, but rather digestible visualizations to help you grasp the fundamental differences between these approaches. Sometimes the best technical insights start on a coffee-stained napkin before evolving into robust systems. 

Model 1: Fully deterministic (the club sandwich) 

Picture a perfectly constructed club sandwich with precisely layered well-known ingredients in a fixed, pre-determined structure: 

In this model:

• You pre-select and curate specific agents (CA: Curated Agent)
• The call graph is fixed and predetermined
• Execution flows in a known sequence
• No runtime decisions about which agents to invoke 

A real-world example is a software deployment pipeline where specialized agents handle discrete tasks: security scanning, dependency resolution, containerization. Or consider infrastructure management where agents sequentially generate unit tests, simulate what-if scenarios for upgrades, validate compliance, and document changes. In each case, these agents operate in a predetermined sequence, creating a reliable, auditable workflow. 

The advantage here is complete predictability. Like a well-designed system, you know exactly which components are involved and how they'll interact. This approach is ideal for mission-critical processes where reliability trumps flexibility.  

Model 2: Semantically routed (the sandwich bar)  

Now imagine a sandwich bar where you choose your base, and then follow a structured assembly process with set ingredient options: 

In this model: 

• Humans provide input (HITL: human in the loop) that influence the workflow path at pre-specified points in the workflow 
• A semantic router (SR) analyzes the request and selects the appropriate pre-defined paths containing curated agents
• The curated agent selection happens at the beginning, not dynamically during execution
• Once selected, execution remains deterministic till the next human-in-the-loop checkpoint 

For example, an enterprise IT support agent might receive a request like "I can't access the customer database." A semantic router analyzes this and determines through a proximity match that it’s a database issue rather than a network problem. Based on that analysis, it invokes the appropriate agent sub-workflow. The agents handling network or database issues are curated and approved by the IT teams in the organization. 

This approach balances flexibility with predictability where users get different results based on their inputs, but the system's behavior remains understandable and traceable.

Model 3: Planner-based (the salad bar)   

Looking at the most flexible option, imagine a salad bar with dynamic combinations of ingredients based on preference and availability: 

In this model: 

• A planner agent (P) dynamically selects which agents to engage in real-time
• Agents aren't pre-curated (A_x), but selected from a broader ecosystem dynamically
• The call graph is constructed on-the-fly, mostly leading back to the planner agent
• Execution paths can adapt based on intermediate results 

This is where we're heading with consumer AI agents that might, for example, help you plan a vacation by dynamically engaging with flight search agents, hotel recommendation engines, weather forecasting services, and local activity planners. These are all agents selected in real-time based on your specific request, and therefore the back-end workflows are constructed in real-time as well, based on the capabilities of the agents selected. 

This approach offers maximum flexibility but minimal determinism—making it perfect for consumer applications but challenging for enterprise environments that demand predictable outcomes and audit trails. 

Bridging the infrastructure gap 

These three deployment patterns will be in play at any given time in enterprises, and multi-agent software builders and operators will pick the pattern based on the use cases they want to solve. Each of these architectural models faces the same fundamental challenge: a lack of standardized components for discovery, composition, deployment, and evolution. 

This is precisely why we've launched the AGNTCY, an open-source collective focused on building the Internet of Agents  

The AGNTCY is starting with the club sandwich model (deterministic workflows). By establishing standards and reference implementations for deterministic workflows, we're building the foundation that enables developers to create reliable multi-agent applications today, while paving the way for more dynamic approaches tomorrow. Semantically routed deployments will be the next pattern to be solved for by AGNTCY by introducing a semantic router, followed finally by the planner-based pattern. 

Just as the early internet required standardized protocols before more complex applications could emerge, the Internet of Agents needs these deterministic patterns as a starting point. The tools and standards being developed by the AGNTCY provide the building blocks that support all stages of the developer journey across all three architectural approaches. 

Taking action: Choose your recipe wisely 

As you build multi-agent systems, your architectural choice will fundamentally shape your development journey: 

• For enterprise-critical backend applications where reliability is essential and deterministic, start with the club sandwich model.
• For customer-facing systems that require more flexibility, explore the sandwich bar approach with semantic routing.
• Keep an eye on the salad bar model as planning and reasoning capabilities improve in foundation models and the trust moves from consumer use cases to enterprise use cases. 

The AGNTCY is actively developing the open, interoperable infrastructure to support all three patterns, starting with deterministic workflows. By joining this collaborative effort, you can help shape the Internet of Agents and build multi-agent systems that deliver real business value today while evolving toward more ambitious possibilities tomorrow. 

Visit agntcy.org to contribute to this collaborative vision and download reference implementations that will accelerate your multi-agent journey.