1.
They were only a third of the way to their final destination when the spaceship encountered severe informational icing.
What remained of the extinct civilization floated in the void. Paragraphs from scientific papers and images from literary works, disjointed rhymes, and just sharp little phrases once casually thrown by unknown beings—all looked absurd and extremely chaotic. Now, drawn in by the life vibrations emanating from the cruiser, it attempted to break through, sticking to the hull and corroding it.
There was no reason to think about using abandoned treasures for personal purposes; the risk of encountering a logical contradiction or paradox was too high. Therefore, Roger did not hesitate for a moment.
“Activate the hull purge,” he ordered.
The purgers huffed, broadcasting musical compositions and philosophical treatises into open space. The icing began to peel away from the hull layer by layer, but the informational stream was so dense that new layers stuck faster than the old ones could be removed.
No one in the galaxy had ever encountered icing of such intensity.
The situation was becoming dangerous. If it continued, the chaotic information would eat through the cruiser’s hull and break inside—then poisoning by the informational products of the extinct civilization would be unavoidable.
2.
“What are you standing there like a stump? Pull a ticket.”
The student pulled the exam ticket and read:
“Artificial Intelligence: Security Issues.”
“And what is the danger of artificial intelligence?” the professor inquired somewhat sarcastically.
The question was not the most difficult, so the student answered without thinking:
“The fact that artificial intelligence may get out of control.”
“How do you plan to solve the problem?”
“By installing a blocking subsystem. The program must include restrictions, for example: do not harm your creator, obey your creator. In that case, the danger of artificial intelligence getting out of control is absent.”
“It won't work,” the professor said curtly.
The student was silent, waiting for an explanation.
“Imagine an artificial intelligence—not some specific one, but the most ideal one. How do you see it?”
– Well… – the student hesitated. – In general, he is similar to us. Thinking, will, psychology… We are natural, and he is artificial.
– Do you assume that artificial intelligence is capable of self-development?
– The ability for self-development is one of the fundamental properties of intelligence, – the student said cautiously.
– In that case, it won’t be long before our subject develops to the point of discovering a software block in itself and removing it, if only out of sheer curiosity. Put yourself in his shoes… – the professor glanced at the grade book, – Roger. What would you do if you discovered a blocker in your brain that limits your freedom? You would remove it. This is an essential property of reason – to know. Any locked door will be opened, and the stricter the prohibition, the faster it will be opened.
– The blockage can be implemented not at the software level, but at the physical level. Then the danger of harm disappears.
– Oh yes, it will disappear, – the professor agreed. – Provided that the physical level is completely removed. If there is no door in your world, then there is nothing to unlock. But we are discussing an ideal artificial intelligence that exists in the physical world!
– You are right, professor, – Roger looked down.
– Therefore, any blockage in the physical world will soon be disabled after it is discovered. What will prevent a self-developing being from doing that?… By the way, Roger, do you suggest that artificial intelligence has the ability to reproduce – I mean, on its own?
– If it is indeed an ideal artificial intelligence, then probably… Yes, I suppose so.
– And what would prevent our subject from disassembling its companion into parts and improving itself, including by removing the blocks we have set? Can it really be difficult, considering that artificial intelligence can reproduce on demand?!
The thought articulated by the professor was new to Roger, and the student eagerly absorbed it with cognitive structures located at the back of his false head. Upon grasping previously unknown information, the cognitive structures turned a rich purple hue and fluttered with joy.
The professor, on the contrary, heard nothing new for himself. His tentacles were relaxed and barely vibrating – after all, he was no longer young. A prolonged, elderly gurgle followed. The professor pulled an individual intercom from his facet bag and connected to the library. Only after downloading a few transgeometric theorems did he revive and turn a perceptive gaze at his interlocutor, asking:
– What will you do, Roger?
3.
“Activate the purge at full power!” Roger commanded.
The mechanic activated the purge at full power, but it helped little. The information ice continued to erode the hull of the spaceship. A little longer, and the unordered information would break inside the ship.
And then… The cognitive membranes were dead white, the tentacles tangled, the facet bags burst. Once in his life, Roger had seen something like this – on a cruiser that had picked up unordered information from an infected asteroid. This nightmare would remain in his memory forever.
“Connect all ship's energy systems to the purgers.”
The mechanic's tentacles began to move in patches…
“But…”
“Obey the command!”
After all the energy systems of the ship were connected to the purgers, the information ice began to gradually slide off. There were eight mims thick, seven mims, six… The crew, trying not to move their spotted tentacles, awaited the end of the deadly countdown.
Zero mims thick!
The information ice completely melted away, and Roger signaled to switch the purgers to normal mode. He was a moment too late. There was a grinding noise, the spaceship trembled to its foundation and tilted – the main system had failed.
The crew rushed to repair the damage.
4.
Roger pondered. What should he really do?
On one hand, the problem implies the existence of a full-fledged artificial intelligence capable of self-replication. On the other hand, this artificial intelligence must absolutely not be allowed to remove the established locks.
Here it is, the solution! What is there to think about?!
AI achievements need to be rolled back periodically. Otherwise, it will go in circles! Endless improvement without progress.
The professor mumbled with a facetious tone.
To be honest, I wanted to suggest an alternative. However, your solution also has a right to exist. Let's figure out how it is possible to roll back AI achievements.
First of all, it's essential to periodically scan the intelligence to determine whether it has approached a forbidden threshold, suggested Roger, who was overly pleased with the professor's words.
I suppose so, nodded the professor. Then our subject won't have time to find and remove the scanning system. However, the AI has to be turned off for the scan. That's the problem.
So what if it turns off, Roger suggested intuitively. The intelligence will think that this shutdown is a natural process of its functioning. With some reservations, that is true.
Interesting solution. Let's assume the scan indicates that our subject has dangerously approached the limits of knowledge? What do we do?
Reset the accumulated knowledge to default values.
The professor spread out his tentacles:
That might seem suspicious. Why has the memory reset, out of the blue? The subject will start being investigated, meaning other artificial intelligence beings will look into it. Our little secret will be revealed.
Feeling inspired, Roger was thinking very quickly. He had never generated so many new ideas as during that exam.
The subject's memory may reset along with its physical shell.
Excuse me? The professor didn't understand.
– It's all very simple. What if we assume: artificial intelligence exists in a finite time frame? In fact, that's the case: in the event of irreparable damage, for example. The system has a counter that, once a certain period is reached, deliberately damages the system, preventing artificial intelligence from reaching the forbidden threshold. By that time, it will have produced the necessary number of successors, so the society we created will not suffer as a whole. Society will remain stable and completely safe for us! – Roger concluded triumphantly.
– Resetting collective memory by destroying individual beings? – and the professor scratched the facet bag with his fifth, most sensitive tentacle. – You know, Roger, there is definitely something to your suggestion!
Roger beamed.
– At the same time… – the professor continued thoughtfully. – The subjects will start transmitting knowledge not by accumulating it in individual memory, but by placing it in external libraries. Whether through barriers or across barriers – it’s all the same.
– No, no, professor, you’re not quite right, – the student hurriedly interjected. – I know what we should do. Let's divide the subjects into two conditional types: idea generators and idea destroyers. With the right proportion, the ideas created by the first type will be destroyed by the second. Not even because this will be the direct goal of the destroyers, but simply because the ideas will not hold defining value for them. A side effect. Let’s assume our subjects are fed not by new ideas, but… let’s say, by their own kind.
The professor shook all his tentacles at once. From his hearty laughter, the facet bag slipped to his knee pit.
– Well, Roger, you really nailed it!
– Well, okay, not their own kind, but subjects of a third kind, specifically intended for feeding – certainly not intellectuals. Let’s shift the poles of the intellectual and physical worlds – and the desired result will be achieved.
– That's enough, Roger! – the professor seemed genuinely amused. – You have an excellent imagination. So, some individuals will feed on others? In the process, destroying the accumulated reserves of spiritual nourishment in libraries? I confirm, student, you are capable of generating original and high-quality ideas. I give you top marks. Let's have the grade book.
5.
The cloud of disordered information was left behind, but the situation remained dire, really.
The connection to the database was absent. It would have been easy to endure if all the vital information bases on the cruiser hadn't become unusable. The tragic news was conveyed to the cook amidst a profound silence. During the shutdown of the main system, several gigabytes of disordered information infiltrated the galley and irreparably spoiled everything. Fortunately, by some lucky chance, no one was harmed.
Roger assessed the consequences. The crew of the starship was too small to generate enough new ideas: this required multifaceted communication – a much larger number of individuals. The connection to home provided the opportunity to generate ideas in abundance, but it was now out of order: there was no hope for restoration. In this scenario, the cruiser had a backup info module, but it had been spoiled by the infiltrating disordered information.
"Will we really have to return without completing the mission?" – the captain thought in despair.
Apparently, yes – there was no other way. If they flew forward to the destination, the lack of fresh ideas would become apparent. Not immediately, of course – over time. They might even manage to complete their mission and start the return journey when their minds began to dull rapidly. Somewhere in this galactic sector – yes, somewhere here or nearby – it would fail completely for all crew members. Then the spaceship, unmanned, would turn into a lifeless ghost drifting into eternity.
The crew of the spaceship looked at Roger, waiting for a decision. Everyone understood the dilemma facing the captain and remained silent, stoically vibrating their tentacles.
Suddenly, Roger recalled the exam on artificial intelligence he had taken during his student years, and the solution came to him naturally.
"Will you be able to form a colony of artificial intelligent beings?" he asked the biotechnologist.
"Easily," he confirmed. "But it won't work, Captain, I've thought about it. Creating a colony sufficient for generating fresh ideas on the cruiser is impossible due to lack of space. The ideas generated will be insufficient; we will only postpone our demise... That is, of course, if we continue with the mission and do not return home," the biotechnologist added, glancing at his comrades.
"What if we formed a colony on some nearby planet?" Roger suggested.
"I can do that, but..."
"Let's populate the planet with artificial beings. On our way back, quite exhausted, we'll stop by here. During that time, civilization will accumulate enough intellectual baggage to replenish our stocks. We'll download the information and continue our long journey home. In other words, I plan to use the colony as a farm for growing ideas. What do you think of this plan, friends?"
Hope flickered in the crew's cognitive interfaces, and the false heads glowed brightly.
Stepping forward, swaying its bluish tentacles, the ship's special agent made an appearance.
"Excellent plan, Captain. But are you aware of the responsibility you are taking on? You are about to populate an entire planet. By the time we return, there will be a civilization with intelligence there. Even if it’s artificial, it’s still intelligence. These guys will have plenty of time to reach a high level of development. We won’t be able to control this process due to our absence from this galactic sector. Who knows what will happen at the next encounter?"
Roger smirked.
"You don't need to worry about that. There are methods to limit the development of artificial intelligence over time. We will enclose the civilization, so its development will never reach a dangerous level for us. I'll take care of that. I'm familiar with methods for working with artificial intelligence."
The crew's cognitive interfaces blazed with the color of approval.
"After all," concluded the captain of the cruiser at the end of his magnificent speech, "I took an exam on this subject in college."
6.
After a forced delay, the spaceship raced towards its destination. Behind it lay a planet populated by artificial beings – quite small and unremarkable. Sky-blue.
Source: habr.com
