The idea of non-human devices of great power and great ability to carry through a policy, and of their dangers, is nothing new. All that is new is that now we possess effective devices of this kind.
In the past, similar possibilities were postulated for the techniques of magic, which forms the theme for so many legends and folk tales. These tales have thoroughly explored the moral situation of the magician.
In all these stories the point is that the agencies of magic are literal-minded; and that if we ask for a boon from them, we must ask for what we really want and not for what we think we want.
The new and real agencies of the learning machine are also literal-minded. . . . We cannot expect the machine to follow us in those prejudices and emotional compromises by which we enable ourselves to call destruction by the name of victory. If we ask for victory and do not know what we mean by it, we shall find the ghost knocking at our door
Norbert Wiener (1961) Cybernetics, 2nd ed.
No comments:
Post a Comment