A taxanomy of strong optimzers; and "scaffolded optimizers"

Here is a fake taxonomy of “strong optimizers”. Note that I’m mainly interested in the idea of “scaffolded optimizers" (more below), but there are a few more bits needed to contextualize said concept.)

(Why is it (plausibly) interesting to have such a taxonomy? On one plausible view, AI risk can be understood as risks from strong optimizers. Strong optimizers will tend to, by their sheer nature, seek to have large effects on the world. This might turn out tricky (to say the least), unless we figure out how and what to turn such an optimizer “towards”. Remember, however: this is only one (plausibly useful) stance one can adopt towards the AI rik problem (also see).)

Back to the taxonomy. Remember: it’s fake.

  1. Optimizers (general)

    1. Maybe something like this as definition of optimization:

      1. > An optimizing system is a system that has a tendency to evolve towards one of a set of configurations that we will call the target configuration set, when started from any configuration within a larger set of configurations, which we call the basin of attraction, and continues to exhibit this tendency with respect to the same target configuration set despite perturbations.

  2. Agentic Optimizers

    1. Here, agents are the sorts of things that have reasons, and whose reasons have motivational force (on the agent).

      1. (NB agency as “having reasons which have motivational force” is a pretty typical conception of agency in academic philosophy, and it’s also a conception that I expect most AI alignment folks to find weird and unsatisfying.)

  3. Scaffolded Optimizers

    1. Scaffolded Opitzers optimize by making use of external machinery, e.g. economic structures (the market, cooperations, money, …), or cultural-memetic infrastructures (e.g. (social) media, language, ). 

      1. I use this term in analogy to the way Godfrey-Smith uses the term “scaffolded reproducers” in Darwinian Populations and Natural Selection: “A third category [in the menagerie of reproductive processes found in different parts of the tree of life] I will call scaffolded reproducers. [...] Their reproduction is dependent on an elaborate scaffolding of some kind that is external to them. [...] Examples here include viruses and chromosomes. As part of cell division, a chromosome is copied; a new one is made from the old. The chromosomes cannot do this with its own machinery, or even largely with its own machinery. It is more accurate to say that the chromosome is copied by the cell.” (Godfrey-Smith himself uses his term leaning on Sterneley's notion of scaffolded learning which he understands as learning scaffolded by instructions, artefacts, and the active shaping of the learning environment (2003).