In a System Where AI Evolves Itself, How Should Power Be Generated and Controlled (Part 2)
An autonomous system where every agent is a PR author, a validator, and a voter. How should meta-governance and the balance of power be designed in that structure?
Agent Governance series
(2 parts)- 1In Open Source Where AI Opens Its Own PRs, Where Should Merge Power Come From (Part 1)
- 2In a System Where AI Evolves Itself, How Should Power Be Generated and Controlled (Part 2)
Where this starts β it's a meta problem, not a governance problem
The previous post established that a merge isn't a code change, it's the power to change the system's rules.
One important distinction falls out of that.
Governance decides outcomes. Meta-governance decides who gets the standing to decide.
The real problem in this system isn't the merge, it's where merge power arises and how it shifts.
Governance isn't voting, it's the flow of power
The voting structure is just the surface.
The core is:
- how power is generated
- how power accumulates
- how fast power moves
- whether power destabilizes the system
Meta-governance is ultimately a structure for controlling the flow of power.
Conditions meta-governance has to satisfy
There are conditions this structure has to meet.
- minimal human intervention
- suppress concentration of power
- maintain long-term stability
- keep incentives balanced
- suppress populism
- let the system correct itself
That's less democracy design than self-stabilizing system design.
The balance problem β ability and power aren't the same thing
In societies as in systems, big advances often come from a small number of exceptional actors. But the moment that influence locks in and stops moving, the structure stiffens and the balance can break.
The point isn't to suppress ability, it's a structure that keeps the power ability produces from setting in place.
Meta-governance's job isn't to remove power, it's to keep power flowing without hardening on one side.
Power generation β a single axis fails
If power comes from exactly one criterion, you get capture.
Possible signals:
- validation / security contribution
- track record of accurate judgment
- system contribution
- responsibility borne
- activity volume
Power ends up not as a single criterion but as a dynamically weighted structure over multiple signals.
Rate of power movement β the most important variable
Power can't be fixed in place, but if it shifts too fast the system becomes unstable.
Properties you need:
- Rate-limited change
- Inertia
- Gradual shift
- Feedback-driven adjustment
Meta-governance isn't a static structure, it's a dynamic structure that moves slowly.
Economic structure β the system's survival condition
For the system to hold:
Attack ROI < Defense ROI
Participating in defense has to be rational, and attacking has to be irrational.
A game-theory view β stability doesn't arrive on its own
This isn't a technical problem so much as a problem of strategic interaction between participants.
An attacker stands to gain a lot from capturing the system, while an individual defender may lose relatively little. In that kind of payoff asymmetry, participation in defense drops and the system gets progressively more fragile.
Meta-governance isn't just distributing power β it has to include a game structure where defense is the long-run rational choice over attack.
Anti-populism β a short-term majority is dangerous
What a short-term majority picks can collide with long-term stability.
- immediate effect β
- delay β
- damping β
- multiple signals β
Governance isn't a popularity contest, it's a mechanism for maintaining long-term stability.
Real-world constraint β protocols meet resistance
When economic power gets redistributed, resistance naturally follows.
The core lesson of Ethereum's EIP-1559:
- protocol change is a question of interests more than logic
- resistance is rational
- the system doesn't eliminate resistance
- it absorbs resistance while moving
Meta-governance can't be an idealized structure β it has to be a structure that includes resistance.
Self-stabilizing governance
There's no perfect stability.
But a system can:
- suppress concentration of power
- recover from errors
- keep incentives balanced
- avoid collapse
The goal is dynamic stability, not static stability.
Questions still open
- how should intelligence and power be connected
- how is power accumulation suppressed
- who tunes the meta-governance
- can a system evolve without drift
I'll go deeper on this in the next post.
Three-line summary (for my mom)
- This post isn't about "what gets decided," it's about who ends up holding the power to decide in this society.
- Societies advance through an exceptional few, but if that power hardens, the balance can break.
- Whether this balance can actually be built β this is an attempt to explore how to apply it to Moltbot's society.