01 | How to understand the boundaries of intelligence
When discussing AI, we readily focus on the model: how much it knows, how well it reasons, whether it can plan, and whether it can improve itself. Memory, tools, permissions, and execution mechanisms built around it are often treated as peripheral, as though the model already contained intelligence in its entirety and everything else were merely an accessory that made it easier to use.
Applied to people, however, this distinction seems strange. An engineer’s ability to solve problems depends on more than a brain. It also depends on a body that perceives the world, hands that operate equipment, reference material, instruments, and the resources and collaboration provided by an organization. Remove these one by one, and what the engineer can accomplish changes. The capability we observe is produced by the whole system working together.
When we ask how intelligence works in reality, a more appropriate unit of analysis is therefore the intelligent system: the components responsible for computation and judgment, the interfaces for perception and action, and the resources and tools it can control. To identify the system boundary relevant to that capability, we need to follow the actual processes of perception, action, and feedback. That boundary has no fixed level of granularity.
From cells to organs to whole individuals with central nervous systems, increasing complexity is accompanied by changes in organization. Coordination among cells, cooperation among organs, and coherent action by an individual each need to be understood at the corresponding level. We can borrow the idea of a “phase transition” from complex systems to describe these shifts: beyond certain organizational levels, new functions and constraints emerge, requiring new rules, coordination logic, and organizational units to explain them.
Looking further outward, people form teams and organizations, of which companies are only one kind. Organizational capability depends on members, but also on division of labor, institutions, information flows, and resource allocation. Even with the same members, changing how they collaborate may enable an organization to do things it could not do before. A complete system at one level becomes a component of a larger system at another. Capabilities at the lower level make the higher level possible, but cannot alone explain how it operates. The boundary of analysis should therefore follow the function we want to explain.
02 | How ecological niches shape intelligent systems
The same holds for agents. An AI agent that works continuously gradually acquires its own state, tools, operating permissions, and feedback mechanisms. Which information it can see, how it preserves unfinished work, when it calls tools, how it recovers from execution failures, and when it asks a person for help are usually handled by its harness. These mechanisms directly determine how the model can operate in the real world and help constitute the capability of the whole system.
A harness is not an optional plugin, but its engineering boundary does not automatically coincide with the intelligent system’s functional boundary. Separating and replacing components must be guided by function. We cannot yet assume in advance which parts can be removed or shared, or which replacements will preserve full capability. A component that can be detached through a software interface cannot necessarily be replaced without affecting the system’s capabilities. That must be tested in relation to the system’s ecological niche.
Imagine an agent helping researchers explore a new problem. Today, it reads a paper and proposes a hypothesis. A few days later, another set of experimental results overturns one of its premises. It needs to remember where the hypothesis came from and which evidence it relied on, then adjust the next research step. If it saves only the final conclusion and compresses away the reasoning and rejected explanations, later exploration may lose important grounds for judgment.
Now consider an agent scheduling production in a factory. A batch of components expected today is delayed, leaving an order scheduled to begin tomorrow short of materials. It needs to check whether inventory contains usable substitutes, whether engineering has approved them, whether other orders can be produced first, and whether rescheduling would affect delivery dates. After proposing a plan, it must update schedules within its authority, communicate changes to the relevant people, and confirm whether the shop floor follows the new plan. A reasonable analysis alone is far from enough: the system must connect to the actual states of inventory, manufacturing processes, orders, and the shop floor.
The first agent needs traceable hypotheses and evidence. The second needs current business states and actions constrained by permissions and rules. These differences shape the design of memory, tools, and execution mechanisms, and therefore affect which parts of a harness can be shared and which need to be built separately.
An ecological niche describes what function a system performs in its environment, which resources it depends on, which objects it interacts with, and which constraints it faces. An industry or job title offers only a rough indication of these conditions. One enterprise can contain many niches, while systems in different industries can have similar functional structures.
The basic unit for analyzing intelligence is therefore a complete system embedded in an ecological niche.
03 | Intelligence and its environment co-evolve
Improving system capability and changing the environment are coupled. Return to the delayed components: if the conditions for using substitutes exist only in engineers’ experience and chat histories, an agent must search through records, determine when a substitute can be used, and seek human confirmation. If the factory maintains explicit rules specifying which substitutes are allowed for which product models and operating conditions, this step can be handled through queries and rule checks. As the environment becomes clearer, the same problem requires less reasoning.
Feedback from execution, in turn, drives changes in both the system and the environment. A substitute may meet the required specifications, yet workers discover that it increases assembly time. This feedback changes the next scheduling judgment and may prompt the factory to refine its substitution rules. What the system can learn and how it decides next are closely tied to what the environment allows it to record and what feedback it receives.
Nor does every step in such a system need the most powerful, computationally demanding model. Database queries and arithmetic can determine whether inventory is sufficient. A rule engine can check whether a substitute meets known conditions. A scheduling algorithm can arrange orders. A small model may be enough to identify a delivery-date change in a supplier’s email. For an undocumented substitution option, a stronger model can help organize the material for an engineer to judge. These mechanisms jointly handle a material shortage, but use different forms of computation and judgment.
This is another form of differentiation shaped by niches. The components responsible for computation and judgment can be implemented by rule engines, specialized algorithms, models of different sizes and architectures, or people. The choice depends on the function required in the specific environment. Not every organ needs the computational capacity of the cerebral cortex.
Using LEGO as an analogy, an open-source model provides a core that can be adapted and reused; the working building block is the model together with a defined ecological niche. In production scheduling, the model alone is not enough: we must specify which orders it handles, which states it can read, which schedules it can change, and how it receives feedback from execution. Only when these definitions are implemented in a harness and environmental interfaces does it become a functional unit that can coordinate with other parts of the factory. The same holds for rule engines and specialized algorithms: composable blocks need explicit functional and collaboration boundaries.
These niches are connected through actual work. Production scheduling depends on procurement for arrival information. Procurement depends on engineering to confirm substitution options. The shop floor must execute the revised schedule and report the results. One system’s actions change the conditions another system faces.
For example, when a procurement agent updates an expected arrival date, a scheduling agent needs to reassess the plan, while an order-fulfillment agent uses the change to evaluate commitments to customers. They need to refer to the same order, the same batch of materials, and their latest states. It must also be clear which agent or person has authority to modify the plan. This coordination already involves designing the shared environment and the division of responsibilities within the organization.
I therefore expect a future built around a collaborative network of different agents, systems in digital environments, and people. They retain internal structures suited to their functions and coordinate through shared business objects, states, and interfaces. When the network connects to sensors, equipment, and actuators, its reach can extend into the physical environment. Its overall capability emerges from these parts working together and evolves as systems and environments change one another.
04 | Before studying RSI, ask what is worth solving
Viewed this way, recursive self-improvement (RSI) first requires a return to first principles: why does this system exist, and what problem should it actually solve?
If an agent spends considerable time each week copying data from several systems, reconciling differences, and producing reports, it is easy to set improvement goals of faster operations and fewer errors. But ask further: which decisions does the report support, why must data be moved manually, and how do the differences arise? We may find that the real problem is inconsistent business definitions across systems. Once definitions are aligned and data connected, the entire copying and reconciliation process may disappear. Continuing to optimize the reporting agent would then mean optimizing a step that need no longer exist.
Self-improvement first requires judging which problems are worth solving, which take priority, and whether the current approach still needs to exist. The ecological niche provides a basis for those judgments. What function the system serves, which gaps obstruct it, and which gap is most valuable to close under the given goals and resource constraints must all be established through its relationship with the environment. Without a niche, it has no “problem worth solving” and lacks a basis for choosing a direction for improvement.