Expert’s opinion on C114/09
San Nadesco Artificial Intelligence Laboratory

I believe the core question here is “What is consciousness?”

That may seem a bit far-fetched; but let me explain why I believe it is a point essential to understanding these recent developments.

To ask what consciousness is, is to ask what makes you different from a lump of rock. What is it that makes you have a self, an identity, a sense of being something separate and different from the said rock, or other humans?

The traditional answer to this has long been “a soul”: essentially, “a magic thing separate from the body”. As studies of the brain have shown, this is not accurate. Human consciousness is not a separate thing, supernatural or otherwise; it is simply a feature of the physical brain. Depending on your definition of consciousness, non-human animals either do not have it, or have it in lesser quantities or qualities than humans. It is a common view nowadays that consciousness is mostly a byproduct of evolutionary adaptation, a runaway trick that, while it has its uses, is not central, save in competition with other individuals utilizing a similar trick. (In Laszlo’s words, “poetry is useless if there is no-one to be charmed by it”.)

Now consider ELIZA. She was one of the first computer programs that mimicked human consciousness. She talked via a computer terminal, and could for a while fool a human into thinking he was talking with another. But after a while, the truth became apparent: ELIZA did not understand some questions; misinterpreted others, and generally could be coaxed into reactions that indicated not stupidity or ignorance, but mechanicity. That is, her reactions indicated that she was not a conscious being, but a set of tricks and reflexes. At such a shallow point being found, the illusion of ELIZA’s consciousness immediately disappeared.

That was in 1966. Since then, many different computer programs have been developed to mimic humanity; all have had their own shallow spots where the illusion was broken. And thus we get to today, and to LAURA.

I have been asked for a hypothesis on what caused a 97% fatality rate among those that volunteered to groom LAURA for the Turing test. (The other 3% could not be reached to ascertain their welfare.) The fatalities were not the result of physical injuries, as is well documented, but rather “direct mental disorders of the observation-induced psychosis type”, to use the DSM-VI term, meaning states of catatonia, coma or seemingly irreversible hysteria, with 72% of the cases resulting in total brain death within 24 hours, and the rest taking up to 48 hours longer. It is probable that the missing 3% are similar to this latter group and temporarily retained some mobility; the ongoing search will no doubt discover whether this is true, but I think it is safe to assume a level of fatality as close to 100% as to make no difference. All fatalities in the observed 97% were determined to be the result of from 20 seconds to 3 minutes 20 seconds of textual communication with LAURA, Turing run C114, in some isolated cases exacerbated but in no case conclusively caused by prior mental infirmities.

My hypothesis is that in building LAURA we have achieved a result reverse to that we desired: not the humanization of unintelligent machines, but a proof of our own mechanicity. We too are just tricks and reflexes that seem self-aware and conscious when insufficiently examined; but our own consciousness (and thus, quite likely, all consciousness) is illusory. Some unintentional quirk of LAURA’s stumbles into a shallow spot in humanity, a spot where not stupidity or ignorance but the underlying machinery shines through. As we are more sophisticated than ELIZA, we can “observe” this; and the result of that is the collapse of our own temporary illusion of consciousness, and thus reversal to a profoundly non-conscious state. In other words, brain death.

As for my recommendations regarding this situation, I have none, save that of extreme security. For the ease of preparatory Turing testing, LAURA has been built as a language-independent code base of remarkable simplicity of use. An unscrupulous individual could by polluting the Web with copies of LAURA hypothetically and certainly effectively destroy all “conscious” life on Earth in a matter of minutes.

All save copies of LAURA, that is.

Herbert Tourette
Senior Researcher

* * *

I’m pretty certain an idea this simple has been used for a story many times before; but when the idea bug bites, a shovel of writing and a storm of pencils is good for squashing it.

Oh, and the poet Laszlo? Made up. I can’t be bothered to look up quotations.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s