An argument for expecting distributed, multi-agent AI systems

Meta: I wrote this sometime in 2021, and although there is a couple of holes or fuzzy bits in the writing, I decided to post it here because a) it can be useful to see (one’s own as well as other’s) “track of ideas”, including faulty ideas; b) I still think that this piece makes some interesting points. To be clear, I am not claiming now (and wasn’t’ at the time) that this is sufficient of an argument for multiagent AI futures, but that it is one argument that might be productive to engage with.

***

This is a sketch of an argument for why we should expect highly advanced AI to manifest as a distributed, multi-agent system as opposed to some singleton AI.

It is presented as a deductive argument, however, I think it is not best to read as proof by syllogism. I don’t think the arguments (and its conclusion) is straightforwardly true, but I think it’s worth engaging with.

Here it goes:

1. Any intelligent system with human-level-or-beyond intelligence needs to continue learning upon deployment to remain effective. 

   1. The fact alone that the world keeps changing requires this.

2. Any intelligent system relies on data about the world to make decisions.

   1. (Also NB F. A. Hayek. The Use of Knowledge in Society)

3. In order to keep learning upon deployment, an AI system relies on

   1. a) sustained data input (i.e. it isn’t at any point “done” with learning), as well as on

   2. b) local data (i.e. data that is sourced locally; there are things about location X that you can only learn through data that stems from location X).

4. Sourcing data locally implies that data needs to be transported from where the data is sourced to where the data is being processed.

5. This data transport requires time and energy (compute, money, …).

6. Because there are strict limits to the speed at which information can travel, a system that requires (local) data sourcing and (central) data processing will experience some degree of “differentiation”.

1. For example, such a system might develop a hierarchical network structure with decentralized, meso-level processing units where data is being partially processed and compressed in order to make the transport to the central processing and control unit cheaper.

2. Even if the time differentials due to the required transport time is "small", these differences will be meaningful in the context of the operational speed of highly advanced AI systems.

7. The time differential is functionally equivalent to an “information differential”, i.e. at any one point in time, different parts of the AI system have access to different types of information.   

8. We should expect that this time/information differential will lead to the emergence of what, functionally, should be understood as differentiated “individuality” and, by extension, a multi-agent AI world.

1. See for some more discussion of this point at “what is individuality” below

(Comment: Assuming the above line of argument to be correct, something like a single AI super-systems can still exists (in some sense of the word “exist”); the claim here is that for some of the analytical purposes most relevant to this community, the more useful frame is to think of this super system is a multiagent one.)

Possible disagreements

2. Any intelligent system with roughly human-level-or-beyond intelligence needs to continue learning upon deployment. 

One might argue that, if you can get so good that you can simulate the entire rest of the world, you would no longer need this. A possible counter to this is that modelling the entire world is nondeterministic polynomial time (NP), thus requiring exponential time to solve.

8. We should expect that this time/information differential will lead to the emergence of what, functionally, should be understood as differentiated “individuality” and, by extension, a multi-agent AI world.

I expect some people will agree with the above line of reasoning, up to the last point: that all this implies multi-agency. They might argue instead that singletons can be distributed provided good self-refactoring capabilities or something like that. 

I would partially agree, in such that, given certain interpretations of "good self-refactoring", we can expect fairly high degrees of "coupling" of a distributed system, maybe so much so that it's legitimate to call it "singleton". However, I also expect that we tend to underestimate how quickly differential information coupling might lead to differential "identity" (and thus goals, purposes, strategies, ..) and their game theoretic implications.

A lot of this puzzle comes down to the question of what is individuality (i.e. the boundaries of an agent), so let’s talk about this some.

What is individuality? 

The single best way to share my intuitions on this question might be to recommend this paper (“The Information Theory of Individuality”). In essence, they consider biological individuality in terms of information theoretic principles, attempting to extract an algorithmic decomposition of system-environment boundaries, arriving at a definition of individuality in terms of ongoing, bounded information processing units rather than lists of static features or conventional replication-based definitions. 

A possible way to operationalize “individuality” is “acting with a unified purpose”. Here is one story for why systems act with a unified purpose: integrated milieus. In other words, what affects one subsystem (e.g. intake of a toxin, of food, of information) also affects the others. From a game theoretic perspective, this implies that, never mind how selfish the subsystems are, they are now selfishly interested in closely coordinating among each other and thus "act with a unified purpose". (For a more detailed account, see this article by Michael Levin and Daniel Dennett) 

According to this information-theoretic conception of what defines agent boundaries, “individuality” comes in degrees. For example, subsystems A and B might be fairly integrated, and thus from a game theoretic perspective, fairly interested in coordinating closely. (Say, they would be willing to share some safety-relevant information, or they are able to credibly commit to some stag hunt type scenario.) Subsystems A and C, however, might be less informationally integrated. (They might for example be unwilling to share safety-critical information, but entirely happy to exchange what they know about what the weather is going to be like tomorrow or whether they recommend going to the newly opened museum in town.)

A thought experiment

Philosophy knows a lot of classical thought experiments that are meant to inform our understanding of what it means to be an individual (e.g. 1, 2, 3). Some of these can also be issued to engage our intuitions about the (single or multiple) identity of AI systems. 

Say, we consider an AI super-system A sending "part of itself" to a mission to some other corner of the universe to do X. We might then ask questions like: 

•  When subsystem B arrives at that other corner of the universe, do A and B still share "one unified purpose"? What if, e.g,. the supra-system A made important updates since they were separated. 

• When the subsystem B comes back, will it simply be able to fully merge with the supra-system again? It seems like both A and B might at this point have reasons to be suspicious/cautious of whether the other systems started plotting against them in the meantime. They might thus decide that they won't (immediately) share safety/security-critical data with the other system anymore. 

• Are they then still "one system" or have they differentiated?

A few more pointers to related ideas

• The economic calculation problem

• Boundaries of firms

• On Being The Right Size

A taxanomy of strong optimzers; and "scaffolded optimizers"

Here is a fake taxonomy of “strong optimizers”. Note that I’m mainly interested in the idea of “scaffolded optimizers" (more below), but there are a few more bits needed to contextualize said concept.)

(Why is it (plausibly) interesting to have such a taxonomy? On one plausible view, AI risk can be understood as risks from strong optimizers. Strong optimizers will tend to, by their sheer nature, seek to have large effects on the world. This might turn out tricky (to say the least), unless we figure out how and what to turn such an optimizer “towards”. Remember, however: this is only one (plausibly useful) stance one can adopt towards the AI rik problem (also see).)

Back to the taxonomy. Remember: it’s fake.

1. Optimizers (general)

   1. Maybe something like this as definition of optimization:

      1. > An optimizing system is a system that has a tendency to evolve towards one of a set of configurations that we will call the target configuration set, when started from any configuration within a larger set of configurations, which we call the basin of attraction, and continues to exhibit this tendency with respect to the same target configuration set despite perturbations.

2. Agentic Optimizers

1. Here, agents are the sorts of things that have reasons, and whose reasons have motivational force (on the agent).

   1. (NB agency as “having reasons which have motivational force” is a pretty typical conception of agency in academic philosophy, and it’s also a conception that I expect most AI alignment folks to find weird and unsatisfying.)

3. Scaffolded Optimizers

1. Scaffolded Opitzers optimize by making use of external machinery, e.g. economic structures (the market, cooperations, money, …), or cultural-memetic infrastructures (e.g. (social) media, language, ). 

   1. I use this term in analogy to the way Godfrey-Smith uses the term “scaffolded reproducers” in Darwinian Populations and Natural Selection: “A third category [in the menagerie of reproductive processes found in different parts of the tree of life] I will call scaffolded reproducers. [...] Their reproduction is dependent on an elaborate scaffolding of some kind that is external to them. [...] Examples here include viruses and chromosomes. As part of cell division, a chromosome is copied; a new one is made from the old. The chromosomes cannot do this with its own machinery, or even largely with its own machinery. It is more accurate to say that the chromosome is copied by the cell.” (Godfrey-Smith himself uses his term leaning on Sterneley's notion of scaffolded learning which he understands as learning scaffolded by instructions, artefacts, and the active shaping of the learning environment (2003).

Footnote on Agent-boundaries vs Personal-Identity

Somewhere else, I specified that I am interested in how to draw boundaries between an agent and its environment, and not interested in questions of individuality in the sense of personal identity or selfhood. 

This is an important distinction to make. For example, depending on what question you want to pursue, you will judge a different set of methodological approaches appropriate for tackling your question. It is also relevant when motivating why you want to ask this question in the first place and the nature of the epistemic purchase you might hope to achieve in pursuing this question. 

Even so, - and this is important to note clearly - it is still possible (and we ought to remain open to the possibility) that, in clarifying how to draw boundaries around agents, we in fact clarify things related to personal identity and selfhood. That does not mean that those two questions are secretly the same question and that my earlier distinction is vacuous. Still, in thinking about it, we can realize that we ought not to be surprised if progress on the former question usefully informs discussions on the latter. 

Consider some of the most well-known philosophical thought experiments that are meant to illuminate, or at least motivate, questions concerning personal identity and selfhood. Imagine for example a technologically superior civilization in which it is possible to create an atomically identical copy of myself. (For many more examples of philosophical thought experiments about personal identity, see e.g. here.) What does that imply about my selfhood and personal identity? Are there suddenly now two “Me”s there that exist simultaneously and independently? But how can “I” be in more than one place at once? On the other hand, if we suggest that the two copies represent different identities, based on what can we claim them to be distinct (remember they are identical copies down to the level of atoms)? In other words, what defines my personal identity if not the sum total of everything that constitutes me? But then, we run into a new problem. If we want to claim that “Me” refers to a specific sum total of things that constitute me, in how far can we claim that me-now and me-from-a-few-seconds-ago refer to the same “Self”? It seems like, the more we think about these questions, the more we get into trouble. Patches we used to get to a reasonable-sounding answer in one place “come back to hunt us” by exposing new inconsistencies elsewhere. 

The question we are interested in here is whether better answers to the agent boundaries problem can help us become less confused about situations like the one described above? I don’t know that it will but I think certainly we have reasons to hope so. We might for example hope to get a more principled answer to the question “in how for do me-now and me-from-a-few-seconds-ago point to the same individual, or not?”. If we could come up with fully principled plausible answers in one place, we could use the same principled reasoning in other places. At the very least, this gives us a coherent (even if wrong) picture. From there, we can then figure out what seems wrong with the coherent-but-wrong picture, and make principled updates to the initial answers. This definitely seems better than trying to find ad-hoc patches for local inconsistencies in a way that does not systematically bring us towards a better comprehensive understanding of the question of personal identity.

Let’s be honest. The above consists of a lot of hopeful yet vague gesturing. However, the point I am trying to make is not about any specific claims concerning the personal identity and its relationship to the boundaries of agents. The point is simply to illuminate something about the relationship between these two lines of inquiry. In summary, I claim that while questions of agent boundaries and questions of personal identity are importantly distinct (i.e. not just different ways to ask the same underlying question), they are not independent in that they can inform each other. Personally, I expect that the flow of insight is predominantly directed from boundaries of agents to personal identity, not vice versa. Mostly, I find discussions about personal identity to be even more fundamentally confused than discussions of agent boundaries, and I see fewer solid epistemic strategies for making progress on the former compared to the latter. In other words, I believe we can learn about how to draw boundaries around agents in a principled manner by looking at the actual world (i.e. conducting some form of empiricism). In the case of personal identity, however, the empirically grounded path to the sight seems cluttered with more obstacles that we don’t currently have much traction on resolving anytime soon (e.g. how science can deal with subjectivity).

Notions of individuality

What does it mean to call something “an individual”? When is it appropriate to do so, and why? In particular, according to what criteria do we draw the boundaries of individuality between the agent and its environment?

Let me be more clear about what I do and don’t have in mind when I talk about individuality here. Specifically, I am not interested in individuality in the sense of personal identity or inquiries about “the Self”. Instead, I am specifically interested in answering how to draw boundaries around agents.

This may in part or entirely be a matter of epistemology and philosophy of science in that we are attempting to provide more conceptual clarity, or at least clearly point at an existing conceptual confusion, with respect to the explanatory role notions of inviability play in our scientific attempts of making sense of the world. However, it may also be the case that this is not “merely” a question of modelling conventions. In adopting, at least temporarily, an assumption of “realism about individuality”, we may ask ourselves whether there are substantive things to be understood about what makes something an individual. (In my own inquiries, I am certainly trying to take the realism hypothesis seriously.)

Many theories across virtually any scientific discipline rely, implicitly or explicitly, on some notions of individuality. However, it is my impression that individuality is largely under-theorized. This can lead to inconsistencies when trying to explain phenomena in a way that builds on (unconvincing or inconsistent) notions of individuality.

For example, if we don’t really know what we mean by “an individual”, we risk using slightly different interpretations at different places within the same theory. It may then appears as if inexplicable discrepancies or “paradoxes” emerge but in fact, the paradox would not be a paradox if we realized we were substituting the same term (“an individual”) for different things. Or, inversely, in inadvertently using different notions of the individual, we may patch over discrepancies in our theory that really are there and that we really should notice, but the patch can make it look like the theory is coherent or unproblematic. A different problem can arise when the concept of an “individual” is used differently in different theories, making them incommensurable. In the name of scientific progress and the ambition to reach an ever more comprehensive understanding of how the world works, we are interested in having theories from different fields be able to “talk to each other” and - eventually - be reconciled. Inconsistent usage of such fundamental concepts as individuality will hamper this undertaking.

Lastly, and this is something I have already noted elsewhere, as a philosopher I feel a special responsibility to contribute to the endeavour of conceptual deconfusion. If empiricism is the bread and butter of science, conceptual deconfusion is the kitchen utensils and the cutlery we need to make bread and spread butter. The idea of individuality - the idea that there is a correct answer to the question of how to draw a line between an agent and its environment - is so fundamental to virtually any theorizing that it affects our understanding of a lot of other concepts of key interest. How can we understand agency (or any notion of goals or goal-directed behaviour, for that matter) if we aren’t able to point at an agent? What is causality if not based on some fundamental distinction between things (that can cause each other)? How should we understand notions of rationality if we cannot coherently point at agents that are supposedly acting rational (or irrational) in the context of a given environment? What is it that the concept of freedom tries to refer to if not something to do with the relationship of an individual to its Umwelt? Any notion to do with coordination (be it altruism/egoism, competition, or symbiosis) is premised on there being several entities that can coordinate with each other.  

The list goes on. Looking at this list, it bugs me to think that we might not even know whether the “individual” that we are looking to identify and delineate in each of these cases (causality, agency, rationality, freedom, coordination, etc.) points at one and the same phenomena (c.f. realism about individuality), or whether it is merely a matter of linguistic conventions and path dependency that we are lead to assume they do. (To put it differently, we may wonder whether the term “individual” or “individuality” is just an umbrella term that refers to many superficially similar-looking but fundamentally different things (or phenomena or processes), or to one-and-the-same underlying and fundamental thing (or phenomenon or process).)

It is in this sense that I want to understand the notion (or notions) of individuality better. I am sympathetic to the hypothesis I formerly dubbed “realism about individuality” - the idea that there is a natural category that is individuality, governed by the same principles across different modalities of implementation/manifestation. That said, I also believe that different fields, despite using at times very different notions of individuality (or, at least, different ways to talk about and formalize it) likely have interesting insights to contribute to the question of individuality - not in small part thanks to a certain theoretical pragmatism (e.g. choosing notions of individuality that lend themselves to talking about the things they want to talk about). 

So, what do different fields' answers to the question of individuality look like? For substantial answers to this question, the reader will have to wait for future posts in which I will be doing deep dives on specific fields and theories. However, there are a few observations that I want to share here already. 

First, it is in fact not as straightforward as it might seem to identify whether a theory in question is about individuality in the sense we are interested in here (precisely because we don’t (yet?) quite know what individuality is). To the extent that we assume that physics, biology and the social sciences (for example) all have things to say about how to draw boundaries around agents (without needing to assume anything about whatever their respective propositions are of equal epistemic merit), it is in fact not surprising that the type of answers these fields provide often look extremely different. The challenge then consists in figuring out whether, in fact, those theories talk about different things or phenomena, or whether it is a legitimate and valuable enterprise to try to figure out how to render those different theories commensurable (based on the assumption that they are in fact pointing in essence at the same sort of thing). As a result, someone undertaking this endeavour should expect to look into a theory and sometimes learn something new and substantial about the nature of individuality, and sometimes learn that what they have learned about was (maybe interesting but) not about individuality (in the sense they are interested in it) at all.  

Second, when doing a high-level scan of how different fields theorize individuality, it quickly becomes apartment that they are different types of explanations that can be given. One distinction that I specifically want to point out at this point is between top-down, descriptive and bottom-up, generative explanations. For example, biology typically relies on notions of individuality which are based on a list of observable properties. For example, an individual is defined as that which constitutes a unity of reproduction, or a unit of metabolic activity, or a unit delineated by a cell membrane (or similar physical signature), etc. While this way of defining individuality can appear locally sufficient, it comes with certain epistemic drawbacks and inadequacies. In the case of biology, for example, it doesn’t take long for those top-down, descriptive notions of individuality to run into trouble, e.g. when they fail to give useful answers to how to draw agent boundaries in the context of uni- vs multi-cellularity, “superorganisms” like ant colonies, mechanisms of genetic transfer beyond chromosomal transfer, etc. 

We might think that instead of such top-down definitions of individuality, what we rather want is a bottom-up, generative framework for identifying the agent-environment boundaries (e.g. Krakauer et al., 2014, “Information Theory of Individuality“ or Levin, 2019, “The Computational Boundary of a “Self””). To the best of my knowledge, attempts of this type are as of yet relative rare and underdeveloped - but they sure seem worth keeping an eye on. 

Third, so far, I have talked a lot about the idea of looking at how different fields approach the questions of individuality and agent boundaries. This asks for an important methodological point of clarification. I am not (primarily) interested in different fields’ perspectives on this issue for reasons related to the history and sociology of science. More importantly, I am using scientific fields as a proxy for assemblages of knowledge about different types of natural systems. For example, the field of biology is delineated (largely) by the fact that its objects of study are biological systems. Respectively, the social sciences study social systems, etc. Related to the notion of “realism about individuality”, the endeavour I am pursuing here is about figuring out whether individuality is expressed by principles, processes or mechanisms that are essentially the same across their modality of implementation (i.e. whether they are implemented in a biological, social or physical system). Now, this is not a completely innocent idea. For example, someone might point out that any sociological system is essentially implemented on top of a biological system, and that any biological system is essentially implemented on a physical system. That certainly seems right to me. However, we might still think that, from an epistemological and certainly sociology of science point of view, different levels of explanations and different epistemic communities will be better placed to talk about certain systems more so than others. Therefore, the abstraction into social, biological, physical etc. systems may well be justified and useful.

What is individuality: why care?

Biology seems like a prime example of how the conception of individuality - the choice of how to draw boundaries around an agent and its environment - is less straightforward, and more riddled with puzzles than appears to be the case at first sight. For example, we might start out with a naive conception that, obviously, biological individuality is one organism - an elephant, a monkey or a tree. Those are the units of replication (evolutionary theory) as well as the units of metabolism (biochemistry, ecology). However, it doesn’t take long until we run into trouble. Consider the relationship between an ant and an ant colony. A worker ant is not the unit of replication, and yet it lays a critical role in how an ant population propagates. Or consider fungus. What looks to us like “the fungus” is really only a small part of the entire, largely subterrestrial fungus web (called the mycorrhizal network). [more examples] This is to illustrate, the question of what makes for an individual is much deeper and more intricate than one might assume. 

Not so fast… sure, different biological organisms come in lots of different shapes and modes of organization. But is this entire talk of individuality not just a substance-less philosophical debate disguised behind smart-sounding semantics? I think not, but I will need some space to explain why. 

First, it can be useful to notice the sheer ubiquity of the use of (different) notions of individuality in scientific theory across basically any field of study. In particular, the usage of the notion of individuality - or, more pragmatically, decisions about how to draw boundaries between agents and their environments - often play load-bearing roles in those theories. For example, individual agents (be that in biology, psychology or economics) usually represent the unit of analysis. The theory wants to say something about their agents, thus decisions about what counts as “agent” and what not are likely to have important implications on the claims produced by that theory. 

Second, we can notice how a number of other interesting concepts (or, our understanding of them) are tied up with our understanding of individuality. For example, any notion of agency is based on a respective notion of “an agent” and agentic or goal-directed behavior is characterized with respect to the actions and goals of some agent (in contrast to the environment). More generally, any theory or functional account of agent behavior in a given situation or circumstance (e.g. behaviorism, rational actor models, bayesian epistemology or game theory) implies some ways of drawing agent-environment boundaries. 

Similarly, normative frameworks as we know them for examples from practical and political philosophy, too, tend to rely on certain notions of individuality. Do not cause harm. Is it better to have more, happy people (adding additional people who are happy) or more happy people (making existing people more happy)? If justice has to do with distribution (of wealth, opportunity, capability, etc.), what or who are we distributing things in between? If a juste state fosters feedoms, whose or what freedom do we have in mind?

But even if “individual agents” (whatever that is) are not the central subject of a theory in question, choices about where we draw boundaries around individuals may still affect how our theory cashes out (what predictions it makes) or at least how we interpret it. In evolutionary theory, different notions of individuality come into play. Richard Dawkins affected a field-internal move towards considering the gene itself as the relevant level of analysis when considering the game theory of evolution. At the same time, we should remember that the phenotype distribution of a population is exhibited at a different level, usually what we would classically consider an individual organism. Controversies in discussions about levels of selection highlight how it may even be that some selection pressure is being executed at the level of kin groups. (See e.g. Okasha 2006, Evolution and the Levels of Selection)

This should go some way to motivate deeper inquiry into the notion of individuality, albeit more is (and will later) be said about it.