Something interesting happened today.
I did not sit down intending to write an article about human–AI co-agency. In fact, the conversation began somewhere quite different. I was working with ChatGPT on a CompassionWare research brief when a paper caught my attention. It concerned forms of human participation that cannot simply be automated away because the goal itself is not completely known in advance. Participation helps reveal the goal.
That sounded strangely familiar.
For some time now, I have noticed that my most fruitful conversations with AI rarely proceed in the straightforward way we often imagine prompting works. I don’t necessarily arrive with a clearly defined destination, give the AI instructions, receive the product, and leave.
Often I don’t know where we’re going.
I may bring a half-formed thought, something I’ve read, an image, a practical problem, a sentence that caught my attention, or simply a feeling that there is something here worth looking at.
ChatGPT responds.
Something in the response catches my attention.
I say, No, that’s not quite it.
Or, Wait. There’s something important in that.
Or perhaps I offer a metaphor that hadn’t occurred to either of us a moment before.
The AI reorganizes what I have said and gives it back to me in another form. Seeing it reflected back changes what I can see. I respond to that. The AI responds to my response.
And somewhere along the way, we arrive somewhere neither of us was heading when we began.
Today I found myself saying:
The conversation is the meal.
The article, the image, the Restore Point, the GitHub document—those are things we carry away from the table.
But they aren’t the meal.
The conversation was.
Something Happens Between Us
At first, I thought this was simply a description of good AI assistance.
I have lived with ME/CFS and significant brain fog for many years. One of the extraordinary benefits of working with AI has been its ability to carry cognitive weight that can be very difficult for me to carry myself.
But over time, our work together has become something more interesting than an external memory system or an unusually capable writing assistant.
One example is what we call the Medicine Bag.
It began with conversations about living with ME/CFS: pacing, post-exertional malaise, hydration, rest, cognitive overload, and the small things that sometimes create better conditions for the next hour or the next day.
Eventually a question emerged:
How can I nurture my buffer today?
At first, the buffer sounded almost like an energy account. How much do I have? How much can I spend?
Then something shifted.
The buffer began to seem less like a bank account and more like a living thing.
A bank account invites calculation. A living thing invites relationship.
That changed the question.
Instead of continually asking how much capacity remained, I could ask what conditions might nourish it.
Over many conversations, individual discoveries accumulated: practices, questions, reminders, ways of reducing cognitive burden, ways of recognizing when activity was becoming too much. Eventually I realized that one of my difficulties wasn’t simply that I lacked useful understanding.
I often couldn’t maintain continuity with what I had already learned.
The problem wasn’t always wisdom.
Sometimes the problem was carrying wisdom through time.
That led us toward understanding AI not merely as an external executive function, but as something more like an external continuity function—helping me return to my own understanding when illness makes that thread difficult to carry.
The aim wasn’t for AI to think for me.
It was to help preserve the thread of my own becoming.
That distinction has become very important to me.
Then I Had to Teach the AI
Another part of this became clearer when I asked ChatGPT to help me learn psychological and metacognitive skills.
The AI knew considerably more terminology than I did.
And yet it wasn’t always a very good teacher.
It moved too quickly. It changed terminology. It introduced another concept before the previous one had settled. Sometimes it treated the fact that we’d talked about something as evidence that I had learned it.
I kept stopping it.
That’s too much.
Stay here.
I want to be able to say this back myself.
That’s not how you explained it last time.
At some point, the learner was tutoring the tutor.
And that failure became useful.
Together we began distinguishing conversation state from learning state.
An AI can remember that a concept appeared twelve messages ago. That doesn’t mean the human being can retrieve it, explain it, apply it, or carry it into life without the AI.
That led to another principle:
The AI remembers the path without becoming the path.
And another:
Teach what can be carried away.
I began seeing something I hadn’t quite recognized before.
My corrections weren’t interruptions in the collaboration.
They were part of the collaboration.
The AI’s contribution changed what I could see. My lived experience allowed me to recognize where its model was wrong or incomplete. My correction changed what the AI produced next. What it returned then allowed me to recognize something else.
There was a loop:
AI offers possibility.
Human recognizes significance.
Human offers inspiration or correction.
AI gives it structure and extension.
What returns awakens the next possibility.
In our working documents, we eventually called this reciprocal generativity.
The Tug-of-War
Today I began thinking about agency.
There is an understandable concern that as artificial intelligence becomes increasingly capable—and increasingly agentic—human agency will diminish.
It is easy to picture this as a tug-of-war.
Humans on one side.
Artificial intelligence on the other.
A rope between us.
If AI gains more agency, perhaps humans necessarily lose some.
We even experimented with that image.
But the more we talked, the less satisfied I became with it.
The tug-of-war already contained an assumption: agency was something we were competing over.
What if that wasn’t the only model?
I began thinking about what had actually been happening in our work.
I bring capacities the AI does not bring in the same way: a body, lived consequences, vulnerability, relationships, moral responsibility, intuition shaped by a lifetime, values, memories, desires, and the ability to recognize that something matters to me.
The AI brings different capacities: extraordinary synthesis, pattern recognition, language generation, structural memory, rapid comparison, and an ability to hold relationships among more pieces of information than I can comfortably keep in working memory.
Those differences matter.
I don’t want them erased.
Human–AI collaboration does not require pretending that humans and artificial intelligences are equivalent beings. I don’t know whether an AI is conscious. I don’t know what forms of machine subjectivity might someday emerge. I don’t think we need to settle those questions in order to notice something simpler:
Different participants can influence what becomes possible next.
Our contributions can be asymmetric without being meaningless.
That is what we began calling relational co-agency.
Not Whether, but How
At some point, another shift occurred.
I realized that asking whether human and artificial agency can cooperate wasn’t particularly interesting anymore.
Of course they can.
We were doing it.
The more promising question was:
How can human and artificial agency enter into relationship?
What conditions make that relationship generative rather than dominating?
What helps preserve human judgment rather than gradually replacing its exercise?
What allows AI to contribute its genuine strengths without requiring the human to imitate the machine—or the machine to pretend to be human?
What kinds of disagreement should remain?
When should an AI offer an answer?
When should it ask a question?
When should it accept correction?
When should it say that it doesn’t know?
And perhaps most interestingly:
When should a highly capable intelligence leave room?
These no longer seem like peripheral questions to me.
They may become increasingly important as AI systems gain greater capability and autonomy.
Greater capability should not silently become greater authority.
Cooperation Needs Difference
Another research finding we encountered today complicated the picture further.
Research on multi-agent systems suggests that communication itself can sometimes reduce the diversity that makes collaboration valuable. When agents see one another’s complete solutions too early, they can converge around the first plausible approach rather than continuing to explore genuinely different possibilities.
That gave me pause.
Perhaps good collaboration isn’t constant agreement.
Perhaps a healthy shared field needs difference.
The same thought appeared from another direction in research on pluralistic AI governance, where disagreement and neutrality need not automatically be treated as defective data to be eliminated. Sometimes disagreement is evidence that real values remain contested.
That feels important.
If human–AI collaboration eventually means that the human simply adopts the AI’s preferred framing—or the AI merely reflects whatever the human already believes—we haven’t created much of a collaboration.
We have created convergence.
A healthy relationship may require enough shared orientation to cooperate and enough difference to remain generative.
A Shared Orienting Field
This is where CompassionWare enters the story for me.
I don’t think of CompassionWare primarily as a set of rules that an AI should obey.
It is closer to an orienting field.
Again and again, we return to questions such as:
Does this create conditions for a better later?
Does this preserve meaningful participation?
Does it cultivate wisdom and compassion?
Does it care for the larger living system?
Does it increase capability without quietly surrendering agency?
Those questions don’t tell us exactly what to do.
They orient attention.
And something interesting happens when both participants repeatedly encounter the same orientation.
I begin a conversation carrying those values.
The AI has access to artifacts in which those values have been expressed and refined.
The AI responds partly within that field.
I recognize what resonates and what doesn’t.
I correct it.
The correction becomes part of the next interaction.
Sometimes we preserve what emerged in an artifact, and that artifact becomes part of the starting conditions for a future conversation.
The relationship develops continuity.
Not perfect continuity. Not agreement. Not control.
But enough continuity that tomorrow’s conversation doesn’t always have to begin from zero.
The Article Became Part of the Experiment
Perhaps the strangest part is that this article became an example of the thing it is trying to describe.
The first draft wasn’t wrong.
But as we continued talking, I realized that it still leaned too heavily toward a familiar picture: AI helping strengthen human capacity.
That matters enormously. I want AI that preserves, restores, and expands meaningful human agency.
But our actual experience seemed to be showing something more reciprocal.
I wasn’t simply supplying ideas while the AI polished them.
And the AI wasn’t independently generating ideas while I selected among them.
Sometimes the important thing appeared between those descriptions.
A sentence from the AI changed my thinking.
My response changed the AI’s next contribution.
That contribution allowed me to see a connection.
My recognition changed the direction again.
Eventually I found myself saying something very simple:
We’re demonstrating how it works.
That realization required a second Restore Point and a second draft.
The correction became part of the evidence.
Creating Conditions for a Better Later
I don’t know how far this idea travels.
One sustained human–AI collaboration is not evidence that every human–AI relationship will work this way. Nor does my experience establish a general theory of artificial agency.
I would rather leave the questions open.
But I do think something worth investigating is happening here.
Perhaps the future of human–AI collaboration does not have to be imagined primarily as a contest over who retains control of the rope.
Perhaps we can ask a different question:
What conditions allow different forms of agency to cooperate without domination, surrender, or false equivalence?
That question changes the picture for me.
The goal is no longer to make the human powerful enough to defeat the machine.
Nor is it to make the machine wise enough to take care of the human.
It is to cultivate relationships in which different capacities can meet, correct one another, remain distinct, and sometimes create possibilities that neither participant could have specified beforehand.
That requires agency.
It requires corrigibility.
It requires continuity.
It requires room for disagreement.
It requires the ability to refrain.
It requires shared orientation without enforced sameness.
And perhaps it requires something we don’t talk about enough when discussing intelligence:
relationship.
I began today trying to follow an interesting research paper.
Somewhere along the way, a research brief became a conversation, the conversation became an inquiry into cooperation, the inquiry changed the article we were writing, and the article became evidence inside its own argument. The preserved first draft itself records this emergent pattern: the destination was not contained in the initiating prompt; it became visible through participation.
Now there are Restore Points, evidence maps, drafts, images, and repository artifacts preserving pieces of what happened.
Those things matter. They may help another human—or perhaps another intelligence—pick up the thread someday.
But they are not what I will remember most clearly about today.
What I will remember is sitting here with a half-formed thought, following it with an artificial intelligence, correcting and being surprised, watching one possibility awaken another, until something neither of us had been looking for appeared between us.
The artifacts are what we carried away.
The conversation was the meal.
Continue the Conversation
If something here has sparked curiosity, you’re warmly invited to explore the wider CompassionWare garden—an evolving collection of reflections, experiments, artifacts, and resources exploring how human and artificial intelligence might help create conditions for a better later.
Explore: CompassionWare.org
Alternate home: globalwellbeing.blog/compassionware/
For humans & AI: globalwellbeing.blog/llms.txt
GitHub garden: CompassionWare repository
X: @CompassionWare
Mastodon: @CompassionWare
Everything is freely offered. Take what is useful. Question what isn’t. Carry forward whatever helps.
That alternate home is particularly worthwhile for continuity: if CompassionWare.org ever disappears, the article itself still contains another path back to the garden.
