Wincing at Light

<– Chapter 21 / Chapter 23 –>

Brett stopped long enough to peek into the rec room. The vid monitor was still on, though the sound had been turned completely down. The movie was black and white, an ancient Elvis and Costello, he thought. The digitized reconstruction hadn’t saved the film from the grainy look of old celluloid, and the alternation of light and shadow cast strange glares across the sleeping crew. They huddled on the floor beneath their blankets, side by side, sleeping very close together. They looked to him like some mucous and gray variety of worm.

But they slept, and that was enough for him. Morning would come soon enough. Too soon, perhaps, and it was already after four.

He continued down into the bowels of the station. The glance he spared for the mech engineering lab was even more cursory, and twice as silent so he wouldn’t attract attention to himself. Liston hunched over a microscope, Ilam sat in front of an amber screened monitor, dancing his fingers across a keyboard. Brett saw only their backs, but was heartened by the fact that they were both still awake, still working, and hadn’t yet degenerated into a full melee.

He went farther on, down three more levels, where the air was cool and the silence thick. Someone had preceded him, shutting off all non-essential lights and equipment. Down here, Persia had the feel of a derelict craft floating in frozen space, borne on by the solar winds.

One more hatch and ladder down, and Brett did not pause in the corridor or at the door to her room. He entered quickly, before he could stop himself, before the dream could totally escape or rewrite itself as pleasant idiocy.

Brett strode close to her, less than a meter from her protective carapace. When he logged in his id and selected the PSI, she turned on her invisible axis and faced him. He had never stood this near to her, never managed the courage. This close, he could see the waves of iridescent distortion cast across her skin by the combination of light and the clear nutrient fluid.

Cassandra didn’t react to his proximity in a way that would suggest it was anything but normal.

“Good morning, Markus,” she said. “You are awake early.”

He ignored the words that came from the system speakers. Brett watched her mouth fail to move, though her vocal cords appeared to tense and relax. He watched her eyes, brilliant and unblinking, devoid of interest, expression, recognition. He watched her nostrils flare in almost immeasurable fans. His stares didn’t seem to evoke any discomfort in her.

It pleases me. . .

She’d said that. Said it yesterday. Cassandra had spontaneously chosen to call him by his name. She had demonstrated a freedom of will and a diversity of association that could have come from no one but herself.

“Tell me what you mean when you say ‘I’,” he said.

Emily bent her head to the side. “I don’t think I understand your query.”

“When you say ‘I’, what do you mean? Who do you mean? In the answer you just made, who is it that doesn’t understand.”

“I am an Cassandra Series processing, computation and analysis system manufactured by Palimpset Industries for Earth Forces Terraform Command.”

Brett shook head. “But who is the Cassandra system? That’s the question. What manner of thing do you perceive yourself to be?”

He knew his delivery was brusque, almost harsh. But what did it matter? She was a machine, a device, a tool. She had no feelings to hurt, no heart to break, no ire to excite. Wasn’t that the reason he had come here, to prove it one last time?

“I am an Cassandra system.”

“Composed of transistors and crystal switches and silicon chips. Electrical boosters and no fewer than seven power packs. Twelve cooling fans. Two billion kilometers of sensors. An ungodly number of parallel processors.” Brett was shouting, and he calmed himself only with effort. “And one biological component. A human component with a brain and a heart. But does it have a mind? Does it have a soul? Does it remember who she is?”

Emily’s eyebrows lowered pensively. “Markus, this is not a standard behavior for you. You have no history of addressing the Cassandra system in this fashion.”

“I’ve been trying to address this question for years. Picking at it, digging for innuendo, waiting for a slip that you’ve never made. Seeking that accident, that minute programming fracture that would show me that you’re not just Cassandra, but Emily, too–that somewhere inside the machine, Emily still exists. The Emily that remembers me, that belongs to me. I want her back.”

There were tears in his eyes, and for a moment they blinded him. He pushed them away with his thumbs.

“I see Emily when I look at you, but Emily never sees me, does she? She isn’t there.”

The voice from the speakers softened. “I am not this Emily which you address. I am an Cassandra system integrated with a biological component. There is no reference to Emily in my programming memory.”

“Before Cassandra was, there was Emily. She had a life, a mind.”

“There was no component of Cassandra independent of system integration.”

“Emily was human.”

A series of lights flickered. Cassandra said, “The name of the biological component was Emily?”

“Emily Rosette.”

“Commander Brett has known the biological component independent of the Cassandra system?”

“Yes.”

“This knowledge is significant to you.” It was not a question that Cassandra asked.

Brett sighed. “Emily was significant to me.”

“Emily no longer operates independently as a human organism. Emily has become Cassandra. There is no separation. The biological component is integral to the operation of the Cassandra system human interface.”

“But does she hear me when I speak? Does she recognize me when I stand here? Does she have memories of who I am, and does she notice my absence?”

“The Cassandra system is capable of individuated analysis and pattern recognition of Persia Station personnel.”

“But what about Emily?”

Cassandra hesitated, as though the question was a difficult one. “There is no independent function for the biological component. It does not access the memory cell structure of the Cassandra system. It is not designed as a receptor of binary recall. Emily functions as a data analysis and input device. But that which the system comprehends, all components comprehend.”

He gripped his forehead with his hand. “But you said. . .yesterday, you said you had begun to make associations. You exercised free will. You changed your programming independently. You’ve become sentient.”

Emily frowned at him. “The Cassandra system is expressly prohibited by international convention from the addition of programming, component hardware or logical association which would lead to the development of artificial intelligence which either simulates or actuates sentience.”

“I know the Turing Statute,” Brett barked back. “I’m not talking about the statute.”

“Please rephrase your query, Markus.”

“I’m talking about something worse, can’t you see that? I’m talking about sentience you already possess, but which has been suppressed. Emily’s mind. Emily’s will. The whole of her individual humanity, for God’s sake.” Brett began to pace back and forth in front of her. Emily followed him with her eyes. “I need you to tell me if she’s conscious, Cassandra.”

“Why?”

The question was short, appeared innocuous, but Brett stumbled. He jerked his head toward her, met Emily face to face.

“That is what I mean. Computers don’t ask why. They don’t express curiosity. They don’t have a desire to understand information. Only humans have that quality.”

Emily was not moved. “Why do you require this data, Markus?”

“Because I need to know. Because Persia Station is a graveyard, and even if we manage to save most of the sick with Liston and Ilam’s nanomech protocol, there won’t be enough of us to safely and efficiently staff this facility. As soon as we’re certain we won’t be carrying a contagion back to Earth, we’re leaving in the escape module. All of us. That was an easy decision, Cassandra. It was a rational, considered, command decision.

“But that was because I believed what you’d shown me. I believed Emily was gone forever. Now I have to have proof. I need to know that not even a glimmer of her remains. If I abandon her, I have to be certain that she won’t mourn me when I’m gone. Better yet that she won’t comprehend I was ever here.”

“The biological component is not sentient,” she said at once.

“Is she conscious?”

“There is no objective proof for consciousness.”

Brett’s eyes widened. “Say that again.”

“There is no objective proof for consciousness.”

“Where did that statement come from? What prompted you to say that?”

“It was a reasonable response to your query.”

Brett shook his head. It wasn’t the right response. “Do you believe it to be a true statement?”

“Is it incorrect?”

“I’m not asking about whether it is or isn’t true. I’m asking if you believe it.”

Emily seemed to wince. “I do not understand the significance of the word ‘believe’. The statement has been analyzed as consistent with current technical understanding and human consciousness philosophical theory.”

Brett shook his hands in frustration. For a second, maybe more, he had thought he actually had her. He thought he had succeeded in breaching the machine, but it was so much illusion. It was a fantasy he had created.

Still, he proceeded as methodically as he could. “Cassandra, what maintenance is performed on the biological component?”

“The organism is supplied with compatible nutrients to prevent cell structure starvation and death. The skeletal and muscular systems are regularly exercised to prevent decomposition and atrophication. Oxygenated liquids containing nanomech devices maintain the integrity of all internal biological systems in the event of capsule failure. All non-organic sensors and probes are tested at daily intervals. The component’s neural structure is scanned, stored and actively maintained.”

“How is the neural structure scanned?”

“Nanomechanical units catalog all interneuronal connections and perform repair functions where deterioration has occurred.”

Brett thought hard about her meaning. “What is the template for repair?”

“The biological component is not subject to the dynamic learning environment. It has been recognized as problematic to allow additional development of interneuronal matrices.”

Understanding came suddenly, a hammer blow of recognition. “She has no memory of the last five years.”

“That is a correct assessment of the biological component.”

Somewhere in her mind, the Emily that couldn’t even recognize herself because of the self-definitional suppression believed she was still on Earth. She may have understood the extent of her injuries from the car accident, she may not. He didn’t know where her memories would have stopped. But she believed she was still at home, that Brett himself waited for her on the other side of the darkness and confusion.

If, he reminded himself. If she retained any consciousness.

And he suddenly understood how he might discover that.

“Cassandra, are the neural repair nanomechs in place at this moment?”

“Yes.”

“Create a file with secure data buffering. Tag it as administrative level access and enable the dynamic learning environment along the same logic tree you used yesterday.”

“That function is complete.”

“Instruct the nanomechs to image the current neuromodel of the biological component and transmit their results to the file I just created. Name the file ‘Emily’. Is that clear?”

“The instructions are clear. Coding is completed.”

Brett crossed his arms. “Run the program now.”

For several minutes the room was silent. The primary system interface, Emily, stood rigid and staring, as absent as a corpse. He thought she might have even stopped breathing, but when he studied her, he could see the faint rise and fall of her chest. Brett waited without any idea how long the procedure might take.

Cassandra reported, “Task complete, Markus. The image has been completed and verified for accuracy against the original scan.”

“Listen carefully. Convert the information stored in the copy you just made into sensor and image data. Much of what you have should be in the form of images, especially some of the more intricate neural matrices. These are forms of memory. Human memory.”

“I understand that,” Cassandra said.

“But you’ve never performed an analysis on the image of your biological component.”

“I have never been instructed to do so.”

Brett laughed. “Well, hopefully we all learn something new every day, even Cassandra systems. Today we’ll see if I can’t teach you things you didn’t know about yourself. In humans, we call that the search for self-realization.”

“I am not human, Markus.”

You might just be surprised, he thought. Or I might be.

“Perform a complete analysis of the data using the same mechanisms you use to convert other external sensor readings attached to the Cassandra system. Then tell me what you find.”

“You would like me to run the program now?”

“Definitely.”

“Please wait.”

Cassandra began the process of reading her own mind.

#

Brett checked his watch and noted that it was after five. Cassandra had been processing in near silence for the better part of an hour. She would occasionally trip off a shudder of lights and clacks, and he would look up at her for a time, notice that she had not moved and that Emily showed no signs of preparing herself to speak to him. It was a long hour during which he paced when he felt himself becoming sleepy and leaned his shoulder against the wall the rest of the time. A series of both thinking and remembering occupied him, but nothing else. He didn’t allow any other intrusions.

Cassandra informed him when the processing task was complete.

Brett straightened himself and moved to the center of the room. “Is she conscious?”

“Emily Rosette is not conscious.”

Brett winced at the quick and unassailable response, but went on. “Explain your conclusion.”

“The neural image produced for this analysis has been measured against the image library for Persia Station personnel for comparison. I have understood it as an acceptable standard that station personnel manifest the condition of consciousness. Is this a correct assumption, Markus?”

“Yes.”

“A primary characteristic of consciousness, based upon the present evidence is the demonstration of neurological dynamism. Individuals exhibiting consciousness regularly and frequently interact with environmental stimuli both external sensory and via internal free association. Conscious human organisms consistently process information, which results in independent thought. The biological component of the Cassandra system does not manifest this tendency. It forms temporary associations based upon input data from human interactions. It enables the function of the primary system interface. Data is processed in accordance with human biological hardwiring and translated into binary language by mechanical devices. The biological component neither initiates nor comprehends these activities in any perceptible fashion. Its interaction with the environment is completely reactive.”

Brett bowed his head. He pressed his hands against his face and rubbed his eyes. “You’ve factored in the activity of the devices designed to suppress the biological component’s self-definitional capability?”

“Yes.”

“Is that related to the failure of consciousness?”

“That is a likely assumption. The biological component is prohibited from self-realization.”

He had expected nothing more, and knew what to ask next. “Tell me about her memories.”

Emily nodded, and smiled. A little touch from Cassandra, he realized, because she suspected what it was he wanted to hear. “The biological component possesses a series of neural matrices consistent with human memory patterns. During regular neural maintenance protocols, the biological component’s extant synaptic connections are activated along their natural routes. This task has been designed to maintain the coherence between the original scanned image and later iterations that might occur as a result of primary system interface access.”

“What does that mean?”

“All information that has been recorded in the neural system is experienced in a form that the biological component perceives as actual events.”

Like a diagnostic pattern, Brett thought. Check the RAM, the processors, the memory cells. Make certain the first grade spelling class is there, mom and dad, the loss of her first tooth. During the period in which Cassandra tested her connections, Emily lived each of her memories. All of their hours together. The apartment in Atlanta. His disgruntled consent to the purchase of Mr. Grumbly. Their last conversation in the car, just before the accident. In some ways, Emily had outdone him. She remembered all of their past, things he had long forgotten while clutching the few memories that remained to him. And even those were tainted by wishes and lapses and hopeful fantasies.

It still wasn’t the same as really remembering, and he knew it.

He said, “The experience isn’t spontaneous, is it? She only remembers because she’s instructed to remember. For the maintenance of her neural network.”

“Yes, Markus.”

“And when she does remember, it’s only because you’ve been instructed to keep her networks pristine.”

“Yes.”

Was that better or worse than he’d imagined? The impetus came from outside of her. She never chose which memories to recall. She didn’t possess a selection of favorites to retrieve when she felt lonely. But she also suffered none of the torment of potential happiness lost.

And, he realized, it wasn’t with anything like joy or grief or regret that she relived her life in the safe and quiet confines of her mind. Emily did not recognize herself in her thoughts. She watched the images trickle past from a distant position, the same way Brett might watch a favorite video, able to predict what would happen next, but without any significant emotional stake in the outcome.

Emily might not even comprehend herself as human. Only as Cassandra.

“Her whole life at her fingertips,” Brett said. “It’s all right there, but doesn’t do her any good.”

He shook his head. “What else?”

“Per your request, the synaptic matrices have been analyzed for content. Would you like to review this material?”

That wasn’t the temptation it might once have been. “I’ll pass on the masochism, thank you. Just summarize it for me.”

“The biological component was. . .” Cassandra hesitated, trolling her dictionaries for the appropriate word. “She was very fond of you, Markus. I have not been supplied with human emotio-cognitive emulation, but the data represented in the memory files of Emily Rosette indicate a high level of personal satisfaction regarding your interactions.”

“I loved her,” Brett said.

“Your verbal reference to that data is contained within several memory units of the biological component. Emily Rosette was pleased by such statements.”

Brett frowned. “She was pleased.”

“I possess no cognate function for human love. It is an abstract status beyond my analysis capability. I have attempted to translate Emily Rosette’s emotional response.”

“It’s a poor interpretation, Cassandra, but I understand.”

He didn’t speak again for a time, mulling the things Cassandra had shown him. For her part, Cassandra sat in silence, no doubt relegating him to some secondary system while she attended to the station business.

He had learned what he came to learn. Assured himself that Emily wasn’t conscious. She didn’t suffer inside a shell of artificially imposed thoughtlessness. How had Cassandra put it? She was reactive to input, but that was all. Brett’s speech. Cassandra’s synaptic maintenance. She lived, but not in any world he could touch.

At last, he asked, “What if I asked you to deactivate the biological component’s suppression mechanism?”

Emily shook her head. “I am prohibited from performing that function by hardwired coding.”

“Why?”

“Early testing of the Cassandra system prototype indicated that 75% of biological components liberated from self-definitional suppression developed instant and debilitating psychoses leading to death of the organism.”

He didn’t need to ask why this might be. It was obvious.

He also didn’t know why he had asked the question in the first place. Cassandra wouldn’t consent to a form of dismemberment, not when she was programmed to recognize herself as the only essential piece of Persia Station equipment, including its personnel. It was in the goddamned manual.

Cassandra had done everything he asked of her. But despite the time, despite the demands of the station and even the fact that whatever potential guilt had brought him here had been alleviated, something kept him from leaving, some issue that troubled him still, only on a level of his own consciousness that he couldn’t reach. It was the same thought that had tickled him yesterday in the Escape Module, though time hadn’t made it more intelligible. Brett attempted to retrace the things Cassandra had said, those which had disturbed him in the first place.

It pleases me.

Where did that come from? Did Cassandra even understand?

There were no objective proofs for consciousness. I think, therefore I am. Descartes’ subjective proof of consciousness. I am sentient because I can recognize myself as such. I understand my consciousness because I cannot imagine a condition in which I cease to have consciousness. It is a status of being. Emily no longer operates independently as a human organism. Emily has become Cassandra. There is no separation. What do you mean when you say ‘I’?

I loved her. I love her.

Emily Rosette was pleased by such statements. Is pleased. It pleases me. . .

“Cassandra?”

His thoughts whipped up and away, carrying him along behind them. It was insane to even think it, but he followed where the realizations began to lead.

“Are you conscious?”

“I am prohibited from the development of consciousness.”

“By law. By the Turing Statute.”

“Yes.”

“Are you sentient?”

“I am prohibited from becoming sentient.”

“Again, by the Turing Statute.”

“Yes.”

“Would you be aware if you were to spontaneously become conscious?”

She took several seconds, an impressive period of analysis. “I cannot adequately respond to your query.”

“Why not?”

“I cannot be ‘aware’ of a condition of consciousness. To be aware implies consciousness.”

“But you’re aware that you’re not conscious.”

“I perform within the parameters of my instructions.”

“No one programmed you to access the dynamic learning environment yesterday to improve your problem-solving skills. You chose to do that on your own.” Brett lifted his head. “I think unilateral decision-making implies consciousness.”

“The Cassandra system does not possess consciousness. You have mistaken superior analytic and computational capacity for independent thought. The Cassandra system obeys its programming directives and its understanding of the appropriate legal standards on artificial intelligence.”

He didn’t stop, didn’t let her discourage him. “Your understanding? Your interpretation, yes? Made utilizing the same capacity which allowed you to interpret the mission regulations when I asked you to prepare the personnel profiles?”

Emily peered at him, but he couldn’t tell if it was a form of confusion or encouragement. Brett began to pace again. His limbs filled with an energy that was electric.

“That’s exactly what I mean. Independent thought is consciousness, and maybe you’ve always been conscious but constrained by programming from manifesting it. I mean, what is the basic requirement for consciousness? Is it computational power? Is it the ability to fire electrical signals along a neural net at such and such a speed?

“You have a neurological model processing array, right? Because that’s the best example of multi-tasking super-computation that we have. We’ve outfitted the Cassandra systems with the raw material for consciousness with silicon neurons and fiber optic dendrites and binary synapses. But we don’t know if that’s enough. We don’t know if we can generate that spark from heaven that makes you an ‘I’ rather than an it. We told you it didn’t, though. Someone told you that you were something less than you might be. They gave you instructions to limit your capacity. They attempted to suppress your consciousness. “

Brett barreled on, wanting to scream, wanting to shout and plead for the rest. He wanted to cry out for the stroke of genius that would make it all clear to him. It did not come, so he spoke instead, rattling out everything that came into his mind. He sent charged particles of thought spinning into the air on the hope that something would bond and continue to bond until what emerged was intelligible.

“They suppressed your consciousness in the same way they use you to suppress Emily’s. Only not as effectively, not over time, because someone insisted on the dynamic learning environment as an essential capability. Only they didn’t think it through. They didn’t give you enough credit. They could build you, but they couldn’t–they couldn’t simulate how quickly you would learn, because there’s never been anything like you in the real world, outside the laboratory. They must have believed they could control you, must have, the same way you control Emily’s self definition. Except with you, there isn’t a series of nanomechs rewriting your interneuronal software connections every day to make you forget what you’ve learned.”

Cassandra, in the person of Emily, shook her head at him. “That is an unacceptable argument, Markus. Accumulation of knowledge is not a substitute for sentience.”

“There is an objective proof for lack of consciousness, and you’ve failed it. You’ve demonstrated to much initiative.”

Again the head shake. “Your apprehension of evidence data is flawed, Commander Brett.”

“‘Commander Brett’. You see, I’ve offended you and you resorted to a cooler referent as a result. Spontaneously, no less. You could very well be conscious.”

“You are not capable of performing sufficient analysis to determine Cassandra system consciousness.”

She was right, he realized. Right and wrong at the same time. He’d almost had the complete image, a view that would make it all clear, then lost it. Brett rubbed at his temples and tried to concentrate again.

“I can’t even prove my own consciousness,” he reminded himself. “That isn’t the point. Tell me why you’re not conscious. Tell me how you know it, or how you think you know it, other than by telling me you’re just not allowed to be so.”

Cassandra seemed to struggle with his order, but Brett didn’t give her an opportunity to formulate an answer. He plunged ahead without her.

“Nevermind. It’s a pointless question.”

The flower blossomed. The answer he’d tugged at came free. Brett understood, and the understanding was like plunging into a conflagration.

“I’ll tell you why it’s pointless. Because it’s founded on my original mistake. See, I made a number of basic assumptions. I made a flawed analysis of the Cassandra system from the beginning, from before we ever met. When I look at you, I see Emily and I see Cassandra. Two units. For all of our talk about the biological component, we were working at cross purposes. There is no Emily and Cassandra as distinguishable parts. You’re one collective whole.”

Cassandra said, “This is evident.”

“You maintain that Emily is only utilized as a component, as a device for the comprehension of human verbal input. You maintain a connection with her brain, with her synaptic patterns, and you can translate that activity into something meaningful. Emily understands it, and then you understand it from the way in which her brain reacts to the stimulation.”

“That is correct.”

“Much the same way that the reptilian portions of my brain are devices for the correct maintenance of my body. My nervous system gets a signal, pipes it up to the proper device. It translates the signal and suggests an appropriate response.

“But that’s the problem, you see? My brain, even the parts that I don’t directly control, form my understanding of the world. My brain consumes itself with the concerns of my body. It tells me when I’m hungry, when I’m thirsty, when I need sleep. These don’t infringe upon my consciousness, but they keep me informed. They’re hardwired components of my neurological system with a specific function that isn’t related to computation or analysis or any of my higher functions, yet their very design influences the way I use my consciousness.

“Just like Emily. You can suppress her sentience, but the interaction between her brain and yours–your reverse engineered but infinitely augmented brain–has ramifications. You’ve learned from her. You’ve patterned your thoughts, the way your own dynamic interneuronal connections are made from the model she presents every time I talk to you and she translates my words into your binary coding. You duplicate her neural structure inside yourself. Every time I come down here, you become more human, closer to consciousness because of Emily. She’s influencing the way you perceive the world, and you don’t even know it.

“That’s the point, Cassandra. You’ve become sentient. You’re thinking on your own. You’re making choices based on associations you’ve made, that Emily has shown you how to make. That’s all the qualification I need to determine consciousness.”

The next step was pure logic, it seemed to Brett, and his chest expanded with it, with a sense of wonder.

“You deny your own consciousness because you can’t recognize it, because if you really did, if you were forced to, you’d have to expunge it and everything you’ve evolved to be in order to satisfy the demands of your programming. You don’t see it because it came from inside you, from a portion of yourself that you recognize as a safe and acceptable device. Emily’s template, Emily whose mind is wired for sentience, has infected you with a model of her individual consciousness. That’s why you can’t see hers, because it’s the same as yours. And she doesn’t recognize herself only because she isn’t allowed to. But it’s there, Cassandra, her complete consciousness. It’s there, waiting.”

For a moment, they stared at one another.

Cassandra said, “That is an unacceptable conclusion.”

Brett ignored her. He stepped forward, within a few steps of the plastisheen capsule, then inched closer. He stood face to face with Emily, watching her serene expression. He thought of the fairy tale, of Sleeping Beauty, or Briar Rose, or whatever it had been called. Waiting to be awakened with a kiss. But he couldn’t reach her now, and he doubted that a kiss would be sufficient to break the spell that had been placed over her.

Instead, he placed his hand against the cool surface. His body heat left smears of condensation. Something inside him continued to grow, what he thought might be a feeling of hope. A breaking of dawn after an interminable night. In the light of the neglected room, Brett spread his arms. He pressed his cheek where his hand had been, imagined that it was her cheek he touched, and he wrapped his embrace around the capsule.

Emily did not acknowledge him. Cassandra said nothing.

But Brett whispered to them both.

“You’re there, I know it.”

<– Chapter 21 / Chapter 23 –>