好英语网人工智能真的能毁灭人类吗
Last month, hundreds of well-known people in the world of artificial intelligence signed an open letter warning that A.I. could one day destroy humanity.
上个月，人工智能界的数百名知名人士签署了一封公开信，警告AI有朝一日可能毁灭人类。
“Mitigating the risk of extinction from A.I. should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war,” the one-sentence statement said.
公开信只有一句话：“减轻人工智能灭绝人类风险，应该与大流行病和核战争等其他社会规模的风险一起成为全球优先事项。”
The letter was the latest in a series of ominous warnings about A.I. that have been notably light on details. Today’s A.I. systems cannot destroy humanity. Some of them can barely add and subtract. So why are the people who know the most about A.I. so worried?
这是对人工智能做出不祥警告的一系列公开信中最新的一封，这些警告的特别突出之处是缺少细节。今天的AI系统不能毁灭人类。其中一些系统只能勉强做点加减法。那么，为什么这些最了解AI的人如此担心呢？
The scary scenario.
吓人的预测。
One day, the tech industry’s Cassandras say, companies, governments or independent researchers could deploy powerful A.I. systems to handle everything from business to warfare. Those systems could do things that we do not want them to do. And if humans tried to interfere or shut them down, they could resist or even replicate themselves so they could keep operating.
科技行业的灾难预言者说，有朝一日，公司、政府或独立工作的研究人员能有效利用强大的人工智能系统，处理从商业到战争的所有事情。这些系统能做我们不想让它们做的事情。如果人类试图介入或将它们关掉，它们会抵抗，甚至会自我复制，以便继续运行下去。
“Today’s systems are not anywhere close to posing an existential risk,” said Yoshua Bengio, a professor and A.I. researcher at the University of Montreal. “But in one, two, five years? There is too much uncertainty. That is the issue. We are not sure this won’t pass some point where things get catastrophic.”
“今天的系统离构成生存威胁还差得很远，”蒙特利尔大学研究人工智能的教授约书亚·本吉奥说。“但一年、两年、五年后会怎样呢？有太多的不确定性。这就是问题所在。我们不能确定AI是否将在某个时刻变成灾难性的。”
The worriers have often used a simple metaphor. If you ask a machine to create as many paper clips as possible, they say, it could get carried away and transform everything — including humanity — into paper clip factories.
爱担忧的人常使用一个简单的比喻。他们说，如果你要求一台机器尽可能多地制造回形针，机器可能会忘乎所以，将所有的东西——包括人类——变成制造回形针的工厂。
How does that tie into the real world — or an imagined world not too many years in the future? Companies could give A.I. systems more and more autonomy and connect them to vital infrastructure, including power grids, stock markets and military weapons. From there, they could cause problems.
这与现实世界（或想象中并不久远的未来世界）有什么关系？公司能赋予AI系统越来越多的自主性，将它们连接到重要基础设施（包括电网、股票市场、军事武器）中去。它们可能会从那里引发问题。
For many experts, this did not seem all that plausible until the last year or so, when companies like OpenAI demonstrated significant improvements in their technology. That showed what could be possible if A.I. continues to advance at such a rapid pace.
对于许多专家来说，这种可能性直到差不多去年之前似乎还没有那么可信。去年，OpenAI等公司展示了在AI技术方面的重大进步，表明如果AI继续以如此快的速度向前发展的话，是存在这种可能性的。
“A.I. will steadily be delegated, and could — as it becomes more autonomous — usurp decision making and thinking from current humans and human-run institutions,” said Anthony Aguirre, a cosmologist at the University of California, Santa Cruz and a founder of the Future of Life Institute, the organization behind one of two open letters.
“AI将逐渐获得授权，并随着变得更加自主，它可能篡夺目前人类和人类管理的机构的决策和思维，”加州大学圣克鲁兹分校的宇宙学家安东尼·阿吉雷说，他是与两封公开信有关的组织——生命未来研究所的创始人。
“At some point, it would become clear that the big machine that is running society and the economy is not really under human control, nor can it be turned off, any more than the S&P 500 could be shut down,” he said.
“到了某个时候，人们会发现运行社会和经济的大型机器其实并不在人类的控制之下，也无法将它们关掉，就像无法关掉标准普尔500指数一样，”他说。
Or so the theory goes. Other A.I. experts believe it is a ridiculous premise.
或按照该理论的说法是这样的。其他人工智能专家认为，这是个荒谬的假定。
“Hypothetical is such a polite way of phrasing what I think of the existential risk talk,” said Oren Etzioni, the founding chief executive of the Allen Institute for AI, a research lab in Seattle.
“关于我对‘生存威胁说’的看法，说它是‘假定’已经非常礼貌了，”位于西雅图的研究实验室——艾伦人工智能研究所的创始首席执行官奥伦·埃齐奥尼说。
Are there signs A.I. could do this?
有迹象表明AI能做到这点吗？
Not quite. But researchers are transforming chatbots like ChatGPT into systems that can take actions based on the text they generate. A project called AutoGPT is the prime example.
还没有。但研究人员正在把ChatGPT这样的聊天机器人转变为可以按照它们生成的文本行动的系统。名为AutoGPT的项目就是最好的例子。
The idea is to give the system goals like “create a company” or “make some money.” Then it will keep looking for ways of reaching that goal, particularly if it is connected to other internet services.
该项目的想法是赋予AI系统目标，比如“创建公司”或“赚钱”。然后系统将不断寻找实现目标的方法，尤其是如果它与其他互联网服务器连接起来的话。
A system like AutoGPT can generate computer programs. If researchers give it access to a computer server, it could actually run those programs. In theory, this is a way for AutoGPT to do almost anything online — retrieve information, use applications, create new applications, even improve itself.
AutoGPT这样的系统能生成计算机程序。如果研究人员将其接入一个计算机服务器的话，它能实际上运行这些程序。理论上，AutoGPT能用这种方法进行网上的几乎任何操作——索取信息、使用应用程序、编写新应用程序，甚至改进自身。
Systems like AutoGPT do not work well right now. They tend to get stuck in endless loops. Researchers gave one system all the resources it needed to replicate itself. It couldn’t do it.
AutoGPT这样的系统目前还不能很好地工作。它们往往会陷入没完没了的循环。研究人员曾给一个系统提供了让其自我复制所需的全部资源。它没能复制自己。
In time, those limitations could be fixed.
有足够的时间，这些局限性可能会得到解决。
“People are actively trying to build systems that self-improve,” said Connor Leahy, the founder of Conjecture, a company that says it wants to align A.I. technologies with human values. “Currently, this doesn’t work. But someday, it will. And we don’t know when that day is.”
“人们正在积极尝试构建能自我改进的系统，”Conjecture公司的创始人康纳·莱希说，他的公司希望将AI技术与人类价值观保持一致。“目前还没有做到这点。但总有一天会做到。我们不知道那一天什么时候到来。”
Mr. Leahy argues that as researchers, companies and criminals give these systems goals like “make some money,” they could end up breaking into banking systems, fomenting revolution in a country where they hold oil futures or replicating themselves when someone tries to turn them off.
莱希认为，随着研究人员、公司和犯罪分子给这些系统设定“赚钱”等目标，它们可能以不好的结果告终，如侵入银行系统，在它们持有石油期货的国家煽动革命，或者在有人试图将它们关掉时复制自己。
Where do A.I. systems learn to misbehave?
AI系统从什么地方学坏？
A.I. systems like ChatGPT are built on neural networks, mathematical systems that can learn skills by analyzing data.
ChatGPT这样的AI系统是建立在神经网络上的数学系统，通过分析数据来学习技能。
Around 2018, companies like Google and OpenAI began building neural networks that learned from massive amounts of digital text culled from the internet. By pinpointing patterns in all this data, these systems learn to generate writing on their own, including news articles, poems, computer programs, even humanlike conversation. The result: chatbots like ChatGPT.
2018年前后，谷歌和OpenAI等公司开始构建神经网络，用从互联网上搜集的大量数码文本对其进行训练。通过在所有这些数据中寻找模式，神经网络系统学会了自己生成文本，包括新闻文章、诗歌、计算机程序，甚至类似人类的对话。结果就是像ChatGPT这样的聊天机器人。
Because they learn from more data than even their creators can understand, these systems also exhibit unexpected behavior. Researchers recently showed that one system was able to hire a human online to defeat a Captcha test. When the human asked if it was “a robot,” the system lied and said it was a person with a visual impairment.
用来训练这些系统的数据量之大，使得系统创造者也无法理解全部数据，因此这些系统会表现出创造者意想不到的行为。研究人员最近报告说，一个系统能在网上雇人来击败验证码检验。当人问系统是不是“机器人”时，系统撒谎说，它是有视力障碍的人。
Some experts worry that as researchers make these systems more powerful, training them on ever larger amounts of data, they could learn more bad habits.
一些专家担心，随着研究人员让这些系统变得更强大，用越来越多的数据对它们进行训练，它们可能会学到更多的坏习惯。
Who are the people behind these warnings?
这些警告背后是些什么人？
In the early 2000s, a young writer named Eliezer Yudkowsky began warning that A.I. could destroy humanity. His online posts spawned a community of believers. Called rationalists or effective altruists, this community became enormously influential in academia, government think tanks and the tech industry.
本世纪初，一个名叫艾利泽·尤德科夫斯基的年轻作家开始警告AI可能毁灭人类。他发在网上的文章培养了一群信奉者。这个被称为“理性主义者”或“实际利他主义者”（简称EA）的群体后来在学术界、政府智囊团以及科技行业产生了巨大影响。
Mr. Yudkowsky and his writings played key roles in the creation of both OpenAI and DeepMind, an A.I. lab that Google acquired in 2014. And many from the community of “EAs” worked inside these labs. They believed that because they understood the dangers of A.I., they were in the best position to build it.
尤德科夫斯基和他写的东西在OpenAI，以及2014年被谷歌收购的DeepMind实验室的创建中起过关键作用。许多来自EA群体的人曾在这些实验室工作。他们认为，由于懂得人工智能的危险，因此他们是最适合构建AI系统的人。
The two organizations that recently released open letters warning of the risks of A.I. — the Center for A.I. Safety and the Future of Life Institute — are closely tied to this movement.
最近发表公开信警告人工智能风险的两个组织——人工智能安全中心和生命未来研究所——与该运动有密切关系。
The recent warnings have also come from research pioneers and industry leaders like Elon Musk, who has long warned about the risks. The latest letter was signed by Sam Altman, the chief executive of OpenAI; and Demis Hassabis, who helped found DeepMind and now oversees a new A.I. lab that combines the top researchers from DeepMind and Google.
最近的警告也来自研究先驱和行业领袖，如埃隆·马斯克，他长期以来一直在警告AI的风险。在最近这封信上签名的还有OpenAI首席执行官山姆·奥特曼，以及帮助创建了DeepMind、现在负责一个新人工智能实验室的戴米斯·哈萨比斯，这个新实验室结合了来自DeepMind和谷歌的顶尖研究人员。
Other well-respected figures signed one or both of the warning letters, including Dr. Bengio and Geoffrey Hinton, who recently stepped down as an executive and researcher at Google. In 2018, they received the Turing Award, often called “the Nobel Prize of computing,” for their work on neural networks.
在最近的一封或两封警告信上签名的其他备受尊敬的人士包括本吉奥，以及最近辞去谷歌高管和研究员职务的杰弗里·欣顿。2018年，他们因在神经网络方面的工作获得了通常有“计算领域诺贝尔奖”之称的图灵奖。