🦸🏻#6: The Role of Profiling in Agentic Workflows

In the dynamic world of AI agents, profiling, knowledge, and memory are tightly intertwined, shaping how these systems perceive, adapt, and respond to their environments and tasks. Profiling – rarely given its own category in agent design – is the bridge between an agent's static capabilities and its dynamic adaptability, based on programmed knowledge and more adaptive memory systems. It is the mechanism that enables intelligent agents to create detailed "portraits" of the environments, users, and tasks they engage with. By synthesizing what the agent "knows" (pre-existing knowledge) and what it "remembers" (historical and real-time data), profiling drives nuanced decision-making, personalized interactions, and seamless task execution. And then you throw in reasoning and planning, reflection, action, and communication – voilà – the whole agentic workflow is complete.

In this episode, we’ll dive into recent and older research papers that offer fascinating perspectives on the concept of an “agent profile.” Profiling deserves to be discussed as a distinct and critical core component of agentic workflows because it acts as the crucial layer connecting humans and machines in their communication. We’ll highlight some long-forgotten studies and explore how they inform contemporary approaches. Ready? Let’s go.

What’s in today’s episode?

_{We apologize for the anthropomorphizing terms scattered throughout this article – let’s agree they are all in ““. (In the Resources section, we provide a paper on anthropomorphizing that is a must-read for designing dialogue systems.)}

Profiling is not a very common term, but we’d like to stick to it since it perfectly encapsulates everything that agents need to do to be aware of their environment and their role in it. Profiling is the process of observing, analyzing, and interpreting the contexts in which agents operate. This isn’t limited to identifying the physical or digital environment – profiling encompasses evaluating performance metrics and understanding behavioral patterns, creating a multi-dimensional awareness that allows agents to act intelligently. In a nutshell, profiling is all about awareness. And how do we unpack this awareness? By asking all the right questions →

In their insightful paper “Generative Agents: Interactive Simulacra of Human Behavior,” researchers from Stanford and Google use the term Agent Avatar, representing a visual and interactive embodiment of a generative agent in a virtual sandbox world, linking its simulated behaviors to observable actions and dialogues:

This profile, this avatar, is what gives the agent its “character” and defines its role, which determines its behavior.

Understanding agent behavior begins with exploring how intelligent systems can balance immediate reactions with long-term goals. The Belief-Desire-Intention (BDI) model provides one influential approach to designing agents that deliberate, plan, and adapt. This model organizes agent decision-making into three core mental attitudes:

The philosophical roots of the BDI model trace back to Michael Bratman and his 1987 book Intention, Plans, and Practical Reason. Bratman introduced intentions as a crucial link between beliefs and desires, emphasizing how rational agents coordinate their actions over time while adapting to new information and circumstances.

Bratman’s theoretical insights inspired further development by researchers Anand Rao and Michael Georgeff in the early 1990s. They formalized the BDI model into a computational framework in their work A model-theoretic approach to the verification of situated reasoning systems, creating a structure for building rational agents that could operate in dynamic environments. Their work aimed to:

While BDI is not the only framework for intelligent agents, its focus on representing mental states and structuring rational decision-making has influenced both academic research and practical applications.

Rao and Georgeff extended the model to tackle the complexities of situated reasoning, where agents operate within environments that are constantly changing and unpredictable. They introduced the idea of agents navigating through branching paths of possible worlds, representing choices shaped by their beliefs, actions, and environmental uncertainties.

Central to their work is the concept of commitment, which determines how agents persist in pursuing goals:

These distinctions allow agents to balance determination with flexibility, a critical aspect of operating effectively in real-world scenarios.

As intelligent agents began to tackle safety-critical applications, the ability to verify their behavior became paramount. Rao and Georgeff introduced branching-time BDI logics to ensure agents adhered to specified properties, such as achieving goals or avoiding harmful outcomes. Their methods enabled efficient model-checking, offering linear or polynomial time complexity for verifying agent behavior.

Wooldridge and Jennings later – in their seminal work Intelligent Agents: Theory and Practice – extended the BDI framework into multi-agent systems, emphasizing social abilities like coordination, negotiation, and cooperation. This evolution reflected the growing complexity of agentic workflows, where agents interact not only with their environment but also with one another, often sharing beliefs and collaborating to achieve overlapping goals.

While BDI is just one approach among many, its emphasis on rationality and decision-making has provided a valuable lens for understanding and designing intelligent agent behavior. It illustrates how agents can not only act but also reason about their actions in pursuit of meaningful goals.

Behavioral profiling focuses on understanding and predicting actions based on historical data and observed patterns. It is a critical step toward creating agents that feel less mechanical and more intuitive in their interactions. For instance, a recommendation engine profiles a user’s browsing history to suggest products they might like. Similarly, a virtual assistant anticipates the next step in a multi-step task based on user inputs.

Agents don’t act/behave in vacuum. Environmental profiling becomes central to intelligent agent design. Without a clear understanding of their surroundings, agents cannot effectively interpret or interact with their environment. Whether in the physical world, like a drone in the sky, or in digital spaces, such as a financial trading algorithm, environmental profiling provides the situational awareness they need to function.

One of the most used frameworks (taught in universities) is PEAS – short for Performance measure, Environment, Actuators, and Sensors. It s a simple yet powerful framework used in AI to break down how an intelligent agent operates within its world. The term was introduced by Stuart Russell and Peter Norvig in their textbook Artificial Intelligence: A Modern Approach.

PEAS answers four key questions:

Here is Trading Algorithm’s PEAS:

Yes, performance is the first letter in PEAS, but performance evaluation and benchmarking have grown into a full-fledged industry since the bloom of generative AI, driving innovation and competition across AI research and development. However, they’ve also become a persistent headache for researchers and practitioners alike. Why? Because while benchmarks like MMLU or HumanEval provide a common yardstick, they often oversimplify the nuanced realities of AI agent performance.

Kapoor et al.'s research AI Agents That Matter speaks directly to this tension. Their paper reveals how benchmarks sometimes incentivize narrow optimization over broader capabilities, creating a mirage of excellence. For example, systems like STeP exploit benchmark-specific patterns in WebArena rather than demonstrating true understanding – a clear case of benchmarks being gamed rather than truly reflecting utility. Similarly, the cost-accuracy tradeoff explored in the paper highlights how leaderboards tend to prioritize flashy performance metrics while glossing over computational and real-world feasibility.

This growing "benchmarking industry" creates a paradox: while it fuels rapid progress, it also drives a race to top metrics that may not translate to real-world effectiveness. Kapoor et al.'s call for integrating cost-efficiency metrics and adopting standardized, reproducible practices is a crucial step toward resolving this headache—making the benchmarking industry more aligned with the actual goals of AI deployment and innovation.

Resource monitoring is vital for intelligent agents, enabling them to assess their computational, network, and physical limits. Often environmental profiling starts with resource monitoring. This process informs decision-making, ensuring agents adapt effectively to dynamic environments. Modern AI agents must carefully balance their capabilities against available resources to maintain optimal performance. This monitoring goes far beyond simple capacity checks – it's a sophisticated system of metrics, thresholds, and dynamic adjustments that ensures smooth operation under varying loads.

Core Resource Metrics

Profiling is more than a technical necessity – it’s the connective tissue that unites an agent’s knowledge, memory, reasoning, and actions. By mapping an agent’s identity, behavior, environment, performance, and resources, profiling lays the groundwork for intelligent, context-aware systems. It transforms static tools into dynamic collaborators, capable of nuanced decision-making and meaningful interactions.

As agents grow more sophisticated, their components must work seamlessly together to ensure precision, adaptability, and meaningful interactions. Profiling enables agents to understand their purpose, but knowledge deepens their expertise, memory stores their experiences, reasoning and planning guide their strategies, reflection refines their processes, and actions bring their decisions to life. These interconnected elements form the backbone of agentic workflows, driving the evolution of AI from rigid tools to dynamic, collaborative systems.

In upcoming episodes, we will continue unwrapping the core components of agentic systems and workflows individually, exploring how each contributes to the creation of intelligent agents. From the intricate designs of memory systems to the strategic capabilities of reasoning and planning, we will uncover the latest innovations and challenges shaping the future of AI.

As we stand at the intersection of technological innovation and human-centric design, with the accelerated advance of generative AI, the possibilities for AI agents are enormous. With thoughtful integration of their core components, these systems promise to transform how we interact, learn, and achieve together.