An argument for expecting distributed, multi-agent AI systems

Meta: I wrote this sometime in 2021, and although there is a couple of holes or fuzzy bits in the writing, I decided to post it here because a) it can be useful to see (one’s own as well as other’s) “track of ideas”, including faulty ideas; b) I still think that this piece makes some interesting points. To be clear, I am not claiming now (and wasn’t’ at the time) that this is sufficient of an argument for multiagent AI futures, but that it is one argument that might be productive to engage with.

***

This is a sketch of an argument for why we should expect highly advanced AI to manifest as a distributed, multi-agent system as opposed to some singleton AI.

It is presented as a deductive argument, however, I think it is not best to read as proof by syllogism. I don’t think the arguments (and its conclusion) is straightforwardly true, but I think it’s worth engaging with.

Here it goes:

  1. Any intelligent system with human-level-or-beyond intelligence needs to continue learning upon deployment to remain effective. 

    1. The fact alone that the world keeps changing requires this.

  2. Any intelligent system relies on data about the world to make decisions.

    1. (Also NB F. A. Hayek. The Use of Knowledge in Society)

  3. In order to keep learning upon deployment, an AI system relies on

    1. a) sustained data input (i.e. it isn’t at any point “done” with learning), as well as on

    2. b) local data (i.e. data that is sourced locally; there are things about location X that you can only learn through data that stems from location X).

  4. Sourcing data locally implies that data needs to be transported from where the data is sourced to where the data is being processed.

  5. This data transport requires time and energy (compute, money, …).

  6. Because there are strict limits to the speed at which information can travel, a system that requires (local) data sourcing and (central) data processing will experience some degree of “differentiation”.

    1. For example, such a system might develop a hierarchical network structure with decentralized, meso-level processing units where data is being partially processed and compressed in order to make the transport to the central processing and control unit cheaper.

    2. Even if the time differentials due to the required transport time is "small", these differences will be meaningful in the context of the operational speed of highly advanced AI systems.

  7. The time differential is functionally equivalent to an “information differential”, i.e. at any one point in time, different parts of the AI system have access to different types of information.   

  8. We should expect that this time/information differential will lead to the emergence of what, functionally, should be understood as differentiated “individuality” and, by extension, a multi-agent AI world.

    1. See for some more discussion of this point at “what is individuality” below

(Comment: Assuming the above line of argument to be correct, something like a single AI super-systems can still exists (in some sense of the word “exist”); the claim here is that for some of the analytical purposes most relevant to this community, the more useful frame is to think of this super system is a multiagent one.)

Possible disagreements

2. Any intelligent system with roughly human-level-or-beyond intelligence needs to continue learning upon deployment. 

One might argue that, if you can get so good that you can simulate the entire rest of the world, you would no longer need this. A possible counter to this is that modelling the entire world is nondeterministic polynomial time (NP), thus requiring exponential time to solve.

8. We should expect that this time/information differential will lead to the emergence of what, functionally, should be understood as differentiated “individuality” and, by extension, a multi-agent AI world.

I expect some people will agree with the above line of reasoning, up to the last point: that all this implies multi-agency. They might argue instead that singletons can be distributed provided good self-refactoring capabilities or something like that. 

I would partially agree, in such that, given certain interpretations of "good self-refactoring", we can expect fairly high degrees of "coupling" of a distributed system, maybe so much so that it's legitimate to call it "singleton". However, I also expect that we tend to underestimate how quickly differential information coupling might lead to differential "identity" (and thus goals, purposes, strategies, ..) and their game theoretic implications.

A lot of this puzzle comes down to the question of what is individuality (i.e. the boundaries of an agent), so let’s talk about this some.

What is individuality? 

The single best way to share my intuitions on this question might be to recommend this paper (“The Information Theory of Individuality”). In essence, they consider biological individuality in terms of information theoretic principles, attempting to extract an algorithmic decomposition of system-environment boundaries, arriving at a definition of individuality in terms of ongoing, bounded information processing units rather than lists of static features or conventional replication-based definitions. 

A possible way to operationalize “individuality” is “acting with a unified purpose”. Here is one story for why systems act with a unified purpose: integrated milieus. In other words, what affects one subsystem (e.g. intake of a toxin, of food, of information) also affects the others. From a game theoretic perspective, this implies that, never mind how selfish the subsystems are, they are now selfishly interested in closely coordinating among each other and thus "act with a unified purpose". (For a more detailed account, see this article by Michael Levin and Daniel Dennett) 

According to this information-theoretic conception of what defines agent boundaries, “individuality” comes in degrees. For example, subsystems A and B might be fairly integrated, and thus from a game theoretic perspective, fairly interested in coordinating closely. (Say, they would be willing to share some safety-relevant information, or they are able to credibly commit to some stag hunt type scenario.) Subsystems A and C, however, might be less informationally integrated. (They might for example be unwilling to share safety-critical information, but entirely happy to exchange what they know about what the weather is going to be like tomorrow or whether they recommend going to the newly opened museum in town.)

A thought experiment

Philosophy knows a lot of classical thought experiments that are meant to inform our understanding of what it means to be an individual (e.g. 1, 2, 3). Some of these can also be issued to engage our intuitions about the (single or multiple) identity of AI systems. 

Say, we consider an AI super-system A sending "part of itself" to a mission to some other corner of the universe to do X. We might then ask questions like: 

  •  When subsystem B arrives at that other corner of the universe, do A and B still share "one unified purpose"? What if, e.g,. the supra-system A made important updates since they were separated. 

  • When the subsystem B comes back, will it simply be able to fully merge with the supra-system again? It seems like both A and B might at this point have reasons to be suspicious/cautious of whether the other systems started plotting against them in the meantime. They might thus decide that they won't (immediately) share safety/security-critical data with the other system anymore. 

  • Are they then still "one system" or have they differentiated?

A few more pointers to related ideas