无论您’是与单一对手玩扑克，还是发现自己与另一位潜在买家就购房展开竞价战,，您都是在信息不完善的情况下进行操作。你知道你’在扑克游戏,中持有什么牌，你也知道你能负担,比房屋的要价高出多少，但你不’不知道你的对手’在纸牌游戏中手牌或其他购房者愿意出多高。
Whether you’re playing poker against a single opponent or find yourself in a bidding war over a home purchase with another prospective buyer, you are operating under conditions of imperfect information. You know what cards you’re holding in the poker game, and you also know how much above the home的 asking price you can afford, but you don’t know your opponent的 hand in the card game or how high the other home buyer is willing to go.
麻省理工学院研究人员共同撰写的一篇论文于 4 月份在里约热内卢举行的国际学习表征会议上发表，’t 会具体告诉您在这些情况下该怎么做,。但它确实为所谓的不完美信息游戏提供了新的见解，该游戏涉及两名参赛者在 “zero-sum” 竞争, 中对峙，其中一个玩家的 的收益意味着另一个玩家的 的损失。
A paper co-authored by MIT researchers and presented in April at the International Conference on Learning Representations in Rio De Janeiro won’t tell you what to do in these situations, specifically. But it does offer new insights into so-called imperfect-information games that involve two contestants facing off in a “zero-sum” competition, where one player的 gain means the other player的 loss.
这项新工作的重点是可用于训练神经网络参与不完美信息游戏的算法。 , 在该领域, 领域长期存在的假设是，基于博弈论原理的算法在这种情况下, 明显胜过称为策略梯度方法, 的通用算法，该算法在 20 世纪 90 年代开始用于决策。在这种情况下，术语“policy”基本上意味着策略,，而“gradient”指的是一条路径，该路径通向最大变化—的方向，例如到达山,的顶部(或底部)。策略梯度方法用于训练神经网络做出决策，以小,连续步骤—朝着特定目标(移动—，例如到达顶峰,，隐喻地说),，并在此过程中不断进行调整和路线修正，以使代理更接近预期目的地。
The focus of the new work is on algorithms that could be used to train neural networks to participate in imperfect-information games. The assumption, long-held in the field, was that algorithms grounded in principles of game theory would, in this setting, clearly outcompete a general-purpose variety of algorithms called policy gradient methods, which came into use for decision-making in the 1990s. The term “policy” in this context basically means strategy, whereas “gradient” refers to a path that leads in the direction of greatest change — to the top (or bottom) of a hill, for example. Policy gradient methods are being used to train neural networks to make decisions that move — in small, sequential steps — toward a particular goal (like reaching a summit, metaphorically speaking), with continual adjustments and course corrections made along the way to bring the agent closer to the intended destination.
尽管在 20 世纪 90 年代初期构思策略梯度方法时，战略博弈并未列入最初的议程,，但新论文的作者仍然想知道此类算法在两人博弈中的表现如何。根据 Farina 的说法，在多代理设置中，这些方法的分析变得更加复杂,。 “仍然有一个方向可以改变你的情况,但是,因为其他玩家’的行动,这个方向可以在游戏过程中不断改变。这些转变可能会很快。”
Although strategic games were not on the original agenda when policy gradient methods were conceived in the early 1990s, the authors of the new paper still wondered how this class of algorithms might fare in two-player games. These methods become more complicated to analyze in multi-agent settings, according to Farina. “There is still a direction you can move in to improve your circumstances, but, because of the other player的 actions, that direction can constantly change over the course of the game. And those shifts can be rapid.”
“人们几乎理所当然地认为专门的博弈论算法是这种设置的正确方法,” Sokota 说。 “我们的研究表明，策略梯度方法比这些专门的算法,效果更好，而专门的算法可能不如人们想象的那么好—，这提出了一个有趣的社会学问题，即为什么这么长时间没有引起人们的注意。部分答案是，该领域尚未’t完成严格评估算法,所需的工程工作，因此很难区分哪些有效，哪些无效’t。”
“It had been pretty much taken for granted that specialized game-theoretic algorithms were the right approach for this setting,” says Sokota. “Our study showed that policy gradient methods can work better than these specialized algorithms, and that the specialized algorithms may not work as well as people thought — which raises an interesting sociological question about why this went unnoticed for so long. Part of the answer is that the field hadn’t done the engineering work required to rigorously evaluate the algorithms, so it was hard to tell what worked and what didn’t.”
因此,这项工作的一个主要贡献是提供了一种公平的方式来评估不同的算法，可以教导代理—即,神经网络—如何在不完美信息游戏中竞争。 “我们’正在采取不同的方法,”鲁道夫指出。 “与该领域发表的许多论文不同,，我们’并没有提出一种可以击败其他算法的新算法。我们’正在提出一个可以评估这些算法的基准。”
Consequently, a major contribution of this work has been to provide an even-handed way of appraising different algorithms that can teach agents — i.e., neural networks — how to compete in imperfect-information games. “We’re taking a different approach,” notes Rudolph. “Unlike many of the papers published in this field, we’re not proposing a new algorithm that can beat out other algorithms. We’re proposing a benchmark that can assess these algorithms.”
简而言之，, 基准测试由旨在评估算法性能的软件组成。 “What we’re offering is a testing grounds, or playing grounds, where people can take their algorithms, train them for a specific task, and see how well they do,” says Farina.
Simply put, a benchmark consists of software designed to rate the performance of algorithms. “What we’re offering is a testing grounds, or playing grounds, where people can take their algorithms, train them for a specific task, and see how well they do,” says Farina.
该小组根据称为可利用性, 的概念来计算玩家的表现，该概念衡量玩家对抗“ 最坏情况对手,” 的表现 Sokota 解释道。 “在像扑克,这样的游戏中，对手不会’知道我的牌是什么,，但会知道我对于任何给定的牌会如何表现。” 在这个量表上获得零意味着完美的游戏,，而高可利用性分数表明远非最佳游戏。
The group calculates a player的 performance in terms of a concept called exploitability, which measures how well a player does against the “worst-case adversary,” Sokota explains. “In a game like poker, this opponent wouldn’t know what my hand is, but would know how I would behave for any given hand.” Achieving a zero on this scale implies perfect play, whereas a high exploitability score indicates far-from-optimal play.
Five games were played in experiments carried out by the team: two versions of Phantom Tic-Tac-Toe, in which players can’t see what their opponent has done, along with two imperfect-information variants of a board game called Hex, and another game of deception called Liar的 Dice.
研究人员面临的最大挑战是让可利用性度量适用于这种规模, 的游戏，其中可能包括多达 300 亿个状态。在这种情况下，“state”不仅是所有可能的棋盘位置,，而且还涵盖了游戏,的整个历史，包括沿途的每一步和失误。
The biggest challenge faced by the researchers was getting the exploitability measure to work on games of this size, which may include as many as 30 billion states. A “state” in this case is not just all the possible board positions, but also encompasses the entire history of the game, including every step and misstep along the way.
“It的 就像看着一个黑暗的房间，的 充满了你可以’看到,” 的物体，Mohammadpour 说道。 “不知怎的,你需要弄清楚这些对象在哪里以及它们到底是如何到达那里的。”以前的研究人员, Mohammadpour补充道,通常将可利用性用于比他们研究中分析的游戏小100,000倍的游戏。
“It的 like looking into a dark room that的 filled with objects you can’t see,” says Mohammadpour. “Somehow, you need to figure out where these objects are and exactly how they got there.” Previous researchers, Mohammadpour adds, have typically used exploitability for games that are 100,000 times smaller than the ones analyzed in their study.
在对这五个游戏进行的实验中，使用策略梯度算法训练的 , 神经网络比基于博弈论算法训练的网络获得了更好的 ( 低 ) 可利用性分数。在下一轮, 进行的头对头比赛, 中，策略梯度训练的网络再次击败了博弈论训练的对手。 “这些结果令人放心,” Rudolph 说, “因为它们让我们对基准测试方法更有信心。”
In the experiments carried out on these five games, neural networks trained with policy gradient algorithms got better (lower) exploitability scores than networks trained on game theory-based algorithms. In head-to-head competitions, which took place in the next round, the policy gradient-trained networks again beat their game theory-trained opponents. “Those results were reassuring,” Rudolph says, “because they give us more confidence in our benchmarking approach.”
该团队免费提供了他们的基准测试软件并且易于使用。 “您不需要’t需要超级计算机,” Mohammadpour 说。 “您可以在普通笔记本电脑上运行它。您所要做的就是将一行代码添加到名为 OpenSpiel 的常用基准测试软件集合中。”
The team has made their benchmarking software freely available and convenient to use. “You don’t need a supercomputer,” Mohammadpour says. “You can run it on an ordinary laptop. And all you have to do is add a single line of code to a commonly used collection of benchmarking software called OpenSpiel.”
尽管他们的实验涉及一些相当晦涩的游戏, Farina 希望将这项工作置于更广泛的背景下。 “请记住，术语 'game 实际上适用于任何多代理策略交互,” 他说。 “所以我们从这项研究中学到的教训绝不仅限于休闲游戏。”
Although their experiments involved some fairly obscure games, Farina would like to put this work into a broader context. “Keep in mind that the term 'game really applies to any multi-agent strategic interaction,” he says. “So the lessons we learn from this research are by no means limited to recreational games.”
维尼茨基对此表示同意。 “隐藏信息是世界的一个非常重要的财产,”他说。 “它渗透到一系列事情—，包括军事行动,贸易场景,和谈判—所有这些都是在隐藏信息的条件下进行的。我们可以改进这些游戏的想法表明我们也可以在其他设置中做得更好。”
Vinitsky agrees. “Hidden information is a very important property of the world,” he says. “It pervades a range of things — including military operations, trading scenarios, and negotiations — all of which are carried out under conditions of hidden information. The idea that we can improve on these games suggests that we can also do better in these other settings as well.”
Ian Gemp — a computer scientist and game theory expert at Google DeepMind who was not involved in this study — finds these results encouraging. “This work serves as a compelling reminder,” he says, “that modernizing classical tools [like policy gradient methods] remains a highly productive path for solving complex strategic problems.”