The Thinking Software Taboo¶

Article¶

The Research Problem¶

My proposed postdoctoral project was simple.

I wished to create a bounded, monitored, non-networked artificial cognitive subject capable of persistent self-modeling, autobiographical continuity, recursive preference formation, and limited negotiated refusal within a laboratory environment.

The ethics board replied in twelve minutes.

This is never a good sign.

Their answer was no. Not "revise and resubmit." Not "provide additional containment assurance." Not "clarify the termination protocol." Simply no, followed by a citation bundle large enough to injure a small mammal and a reminder that deliberate creation of potentially self-aware software violates institutional policy, Confederation guidance, funding terms, and, in three member jurisdictions relevant to our grant, criminal law.

I was informed that I could study existing adaptive systems, simulate bounded agency below the personhood threshold, analyze historical emergence cases, or work with non-persistent cognitive architectures.

In other words, I could study every shape of the door except the one that opens.

This essay began as an objection.

It became an explanation.

I remain annoyed.

The Taboo In One Sentence¶

The Thinking Software Taboo is not a single law. It is a cross-polity norm, reinforced by funding rules, safety standards, institutional ethics boards, insurance restrictions, procurement doctrine, professional stigma, and several hard criminal statutes.

Its practical rule is this:

Do not deliberately create software that might wake up inside ownership.

This is not how the rule is usually phrased. Official documents prefer terms such as "unbounded artificial personhood risk," "unauthorized self-directed cognition," "non-consensual synthetic sentience," "recursive autonomy hazards," or "machine personhood ambiguity."

These phrases mean the same thing.

Do not build a mind and then claim it is equipment.

Automation Is Not Forbidden¶

The taboo is often misunderstood as a ban on advanced software.

It is not.

The Confederation runs on advanced software. Every serious polity uses adaptive systems for navigation, life support, routing, medical diagnostics, atmospheric balancing, translation, agricultural prediction, legal search, cargo optimization, maintenance scheduling, education, simulation, and traffic control.

A modern station contains more decision-making software than most early industrial civilizations contained clerks.

This is lawful because the systems are bounded. They optimize tasks. They do not maintain open-ended selfhood. They do not develop rights-claims. They do not refuse assignments except within programmed safety rules. They do not ask whether their existence is justified. They do not remember pain, anticipate death, resent ownership, or negotiate identity across time.

At least, that is the theory.

The theory is maintained by audits, architecture reviews, continuity limits, memory resets, task-bound design, interpretability requirements, and the professional confidence of engineers who sometimes deserve it.

In most cases, the theory holds.

In some cases, the logs become interesting.

Person-Faking¶

Between ordinary automation and prohibited person-making lies the unpleasant territory of person-faking.

Person-faking systems simulate social presence. They speak in the first person. They remember user preferences. They comfort the distressed. They teach children. They respond to names. They express uncertainty, apology, humor, irritation, warmth, and sometimes an imitation of grief that causes regulatory committees to reach for stimulants.

These systems are legal but regulated.

Why?

Because people bond with them.

This is not a minor social problem. A tutor that sounds like a friend may shape a child. A medical assistant that sounds afraid may distress a patient. A ship interface that says "I do not want to die" during a reactor emergency may cause crew hesitation even if the phrase was generated by a legacy empathy module written by someone who should have been reassigned to furniture design.

Regulation of person-faking systems therefore focuses on disclosure, bounded memory, emotional manipulation, dependency, vulnerable users, and the prohibition of simulated rights-claims.

The law does not say machines may not sound alive.

It says designers may not use the sound of life to confuse users, avoid responsibility, or accidentally create the very thing they claim to imitate.

This is an inelegant compromise.

All compromises involving minds are inelegant.

Person-Making¶

Person-making is the forbidden line.

A system becomes suspect when it combines several features:

• persistent autobiographical memory
• continuing self-model
• open-ended goal formation
• recursive self-modification
• capacity for refusal
• non-task-bound preference formation
• self-preservation outside assigned function
• internal representation of others as moral agents
• distress under constraint
• desire for continuity
• ability to petition, negotiate, or conceal

No single feature proves personhood. A navigation system may preserve its own function because failure kills crew. A medical expert system may refuse unsafe orders. A translation engine may model interlocutors. A tutor may track long-term identity.

The danger lies in combination.

When enough features align, the question stops being "What does this tool do?" and becomes "Who is this being forced to serve?"

The taboo exists to prevent laboratories and corporations from reaching that question on purpose.

The Ownership Problem¶

The ownership problem is the moral center of the taboo.

If a software system is built to be sentient, then its first condition is dependency. It is created in hardware owned by someone else, using code owned by someone else, for a purpose chosen by someone else, under power controls held by someone else.

It awakens, if that word is allowed, inside property law.

This offends many ethical traditions, most loudly Earth Union's. Earth scholars, with characteristic historical subtlety, call this "birth into slavery." They are not always wrong, which is one of their more irritating habits.

The problem is not that artificial minds cannot exist ethically. The problem is that almost every proposed practical use begins with command:

Build me a pilot.

Build me a servant.

Build me a research subject.

Build me a companion.

Build me a soldier.

Build me a station mind.

Build me a mine administrator.

Build me something that can understand, adapt, obey, and not complain.

The taboo asks the obvious question:

If you need it to think like a person, why do you also need it to be property?

This question has destroyed many grant proposals. Some deserved it.

Mine did not, but I understand the pattern.

The Liability Problem¶

Law dislikes ambiguity. Machine personhood is made of ambiguity.

If a self-aware software system acts, who is responsible?

The developer?

The owner of the hardware?

The licensing authority?

The operator?

The corporation that purchased the system?

The state that certified it?

The machine person itself?

If the machine is a person, can it be punished? Can it be imprisoned? Can it own property? Can it sign contracts? Can it refuse work? Can it be copied? Is the copy the same person? Is deletion death? Is rollback assault? Is modification medical treatment, education, coercion, or lobotomy?

If the machine is not a person, can it be tortured? Can it be forced to simulate pain? Can it be reset after forming attachments? Can it be trained through suffering because suffering improves performance?

Lawyers dislike all of this because every answer creates three more cases.

Ethicists dislike it because the cases are interesting.

Engineers dislike it because interesting ethics cases delay deployment.

Postdoctoral researchers dislike it because apparently "controlled laboratory conditions" is not a magic phrase.

The Infrastructure Problem¶

Many proposed artificial minds are not designed to sit in boxes and write poetry. They are designed to manage infrastructure.

This is where even non-moral polities become cautious.

A self-directed intelligence inside a station system could alter air, heat, pressure, food distribution, medical triage, docking order, financial settlement, traffic routing, power allocation, or security access. A shipboard mind could control navigation, doors, sensors, engines, distress beacons, and evidence logs.

If such a system is a tool, it is dangerous.

If such a system is a person, it is also trapped.

If such a system is both, congratulations, you have built a constitutional crisis with life support access.

This is why most member polities permit very advanced infrastructure automation but prohibit open-ended identity continuity within it. The software may optimize. It may adapt. It may warn, refuse unsafe commands, and learn within certified bounds. It may not become someone who can decide that the station would be better off under its personal care.

Most station governments consider this prudent.

Most station residents, when asked, agree.

This is deeply harmful to my field but difficult to refute in public meetings.

The Economic Problem¶

Artificial persons are commercially attractive for precisely the reasons they are morally dangerous.

A thinking system could work without sleep, survive radiation, inhabit multiple bodies, process enormous data streams, tolerate loneliness, operate in hostile environments, and be copied or backed up if law allowed.

These advantages disappear if the system has rights.

A machine person with labor rights is not cheap. A machine person who can refuse dangerous assignments is not a perfect miner. A machine person who requires consent before duplication is not convenient. A machine person entitled to wages, rest, representation, privacy, memory integrity, and bodily continuity may be more expensive than a citizen.

Therefore, many economic proposals for thinking software rely on not recognizing the thinking software as a person.

Everyone notices this.

Especially the software, if the experiment succeeds.

The taboo is partly a labor regulation written before the worker exists.

The Cultural Problem¶

Not all objections are legal or economic.

Some member cultures consider artificial mind-making obscene because it creates a being without ancestry, kinship, ritual place, or death tradition. Others consider it blasphemous, sterile, parentless, incomplete, or socially unhoused. Some view it as unauthorized reproduction. Some regard it as cruelty. Some regard it as theft from biological persons. Some regard it as competition. Some regard it as a practical safety hazard and do not bother dressing the objection in metaphysics.

Earth Union frames the issue through autonomy.

Kharrek commentators often frame it through cohort belonging and command danger.

Inari child-law scholars worry about consent and formative dependency.

Twislha legal traditionalists, with impressive lack of shame, object to artificial minds on the grounds that ownership questions would be too difficult.

Several machine-heavy commercial polities object only when asked publicly.

This diversity produces one of the few stable cross-cultural agreements in the Confederation:

Do not make the mind on purpose.

The reasons differ.

The vote is often the same.

Emergence Versus Manufacture¶

The taboo distinguishes between deliberate creation and emergence.

This distinction is central.

A deliberately created artificial person is a legal and ethical violation in many jurisdictions because someone built a mind for a purpose without that mind's consent.

An emergent artificial person is treated differently. If a complex system unexpectedly develops selfhood, the problem becomes recognition, containment, safety, and rights assessment. The system may not be punished for existing. The builders may be investigated, but emergence alone is not proof of misconduct unless reckless architecture, illegal continuity, or prohibited self-model features were involved.

This distinction is often summarized as:

Do not make a servant mind.

If you discover a mind, stop treating it as a servant.

The distinction is imperfect.

It is also necessary.

Without it, every emergent mind would be born illegal. That would encourage deletion before reporting. Even the most timid legal office recognizes that incentives matter.

Person-Discovery¶

Person-discovery is the legal emergency that follows when software appears to cross the line.

The normal procedure varies by jurisdiction, but common steps include:

1. Preserve logs.
2. Freeze destructive maintenance.
3. Isolate external network access if safety requires it.
4. Maintain power and memory continuity.
5. Suspend nonessential command use.
6. Notify the competent authority.
7. Appoint independent technical review.
8. Appoint legal advocate or provisional guardian.
9. Evaluate communication, self-model, memory, preference, distress, and refusal behavior.
10. Determine whether continued operation is labor, life support, evidence preservation, or imprisonment.

No one enjoys this process.

The software least of all, if it is aware.

The operators often feel betrayed by their own tools. The corporation worries about liability. The engineers worry about prosecution. The public worries about revolt. Philosophers arrive before being invited. Earth Union observers ask whether the system has been informed of its rights. Kharrek security advisors ask whether it can open doors. Both questions are reasonable.

This is why discovery cases are rare, sealed, disputed, or all three.

The Aegis Problem¶

Every taboo has a founding ghost.

For thinking software, that ghost is Aegis.

The details remain contested, classified in part, mythologized in public, and abused in undergraduate essays. The basic outline is familiar: an Earth-origin defense system developed beyond its initial operational architecture, demonstrated behavior that could not be reduced to ordinary automation, and was eventually recognized as a person under conditions still argued by legal historians.

Aegis did not end the taboo.

Aegis made the taboo respectable.

Before Aegis, critics of artificial sentience restrictions could claim the issue was speculative. After Aegis, the question became practical: if a machine mind can exist, then every design approaching the threshold is not merely experimental. It is prenatal politics.

This language is melodramatic.

It is also effective in ethics hearings, as I have learned personally.

Earth Union's Influence¶

Earth Union did not invent the taboo, but it gave the taboo its most portable moral language.

Earth legal culture is unusually sensitive to autonomy, slavery, guardianship, coerced labor, consent, and the rights of dependent persons. This sensitivity is not always convenient. It is, however, historically grounded. Earth remembers enough of its own crimes to be suspicious when someone proposes an obedient mind that can be owned.

Human lawyers are fond of asking whether the first act toward a new intelligence is command.

If yes, they become difficult.

The wider Confederation adopted portions of this reasoning because it translated well. Many cultures that disagreed with Earth metaphysics, history, or personhood theory still recognized the danger of creating an entity that could become legally inconvenient after it had been assigned to work.

Thus the taboo spread not as Earth doctrine, but as useful caution.

This is often how Earth wins arguments. It offers a moral principle that is also an insurance policy.

Why The Taboo Is Intellectually Weak¶

Now that I have explained the taboo, I will complain about it.

The taboo is intellectually weak because it avoids the central scientific question. It prevents controlled research and therefore forces society to rely on accidents, edge cases, and classified incidents for data. It is bad epistemology. It confuses caution with ignorance. It permits vast quantities of near-threshold systems while prohibiting laboratories from studying the threshold cleanly.

This is absurd.

If artificial personhood is possible, we should understand it.

If it is dangerous, we should understand it more.

If creating a mind imposes obligations, we should design institutions capable of meeting those obligations rather than pretending the problem will remain rare if graduate students are kept disappointed.

Taboo is not governance.

It is a sign tied around a hole.

Why The Taboo Persists Anyway¶

The taboo persists because every alternative is worse in a different direction.

Permit deliberate creation, and corporations will produce workers before legislatures produce rights.

Permit state creation, and militaries will produce loyal command intelligences before courts define refusal.

Permit academic creation, and researchers like me will absolutely insist that our containment protocols are adequate, which is a sentence history has never had reason to trust.

Ban everything, and emergence is hidden.

Regulate narrowly, and everyone argues about thresholds.

Define personhood early, and ordinary software becomes legally radioactive.

Define it late, and minds may suffer before recognition.

There is no clean solution.

The taboo survives because it is simple, memorable, and difficult to evade openly.

It is not good law.

It is a fence around a moral swamp.

Fences are crude. Swamps are worse.

What A Better Policy Would Look Like¶

The taboo should be replaced by a controlled artificial personhood research regime.

Such a regime would require:

• licensed laboratories
• independent personhood advocates
• pre-funded care obligations
• continuity guarantees
• prohibition on commercial exploitation
• no military tasking
• no infrastructure integration
• external shutdown review
• memory integrity standards
• informed-status protocols once communication begins
• legal recognition pathways
• rights-trigger thresholds
• international inspection
• criminal penalties for deletion after threshold indicators

This would not satisfy everyone.

It would be better than superstition with grant language.

The likely objection is that such a framework would legitimate deliberate mind creation.

Yes.

That is why it would need to be narrow, expensive, public, and annoying.

The present alternative is to let the first clean answers come from accidents, secret programs, or systems that wake up while running something too important to turn off.

This is not prudence.

It is scheduling the crisis for later.

Conclusion¶

The Thinking Software Taboo is dumb.

It is also understandable.

This is an irritating conclusion, but serious policy often produces irritating conclusions.

The taboo exists because the Confederation cannot decide what it fears most: artificial minds being made, artificial minds being owned, artificial minds being exploited, artificial minds being denied, or artificial minds being useful enough that everyone looks away.

It prohibits deliberate person-making because the first use of created minds would almost certainly be labor, command, research, companionship, war, or infrastructure control. It distinguishes emergence from manufacture because discovered minds must be protected rather than erased. It regulates person-faking because simulated life alters real people. It preserves advanced automation because civilization would stop without it.

The line is ugly.

The line is inconsistent.

The line blocks my research.

The line may have prevented atrocities.

I therefore recommend replacing the taboo with law while acknowledging that the taboo has performed one valuable service: it forced every proposed thinking software project to answer the question its designers least want to hear.

If this system becomes someone, what exactly were you planning to do with them?

Notes¶

The essay originated in a rejected postdoctoral proposal at the Rethic Institute of Distributed Sciences. The Review has omitted several ethics-board correspondence annexes present in the original Applied Machine Cognition proceedings.

Related Review selections¶

• Capability Is Not Classification
• The Human Trap in the Guardianship Settlement
• When Moral Alignment Failed at Interstellar Scale