GPT-2 As Step Toward General Intelligence

A machine learning researcher writes me in response to yesterday’s post, saying:

I still think GPT-2 is a brute-force statistical pattern matcher which blends up the internet and gives you back a slightly unappetizing slurry of it when asked.

I resisted the urge to answer “Yeah, well, your mom is a brute-force statistical pattern matcher which blends up the internet and gives you back a slightly unappetizing slurry of it when asked.”

But I think it would have been true.

A very careless plagiarist takes someone else’s work and copies it verbatim: “The mitochondria is the powerhouse of the cell”. A more careful plagiarist takes the work and changes a few words around: “The mitochondria is the energy dynamo of the cell”. A plagiarist who is more careful still changes the entire sentence structure: “In cells, mitochondria are the energy dynamos”. The most careful plagiarists change everything except the underlying concept, which they grasp at so deep a level that they can put it in whatever words they want – at which point it is no longer called plagiarism.

GPT-2 writes fantasy battle scenes by reading a million human-written fantasy battle scenes, distilling them down to the concept of a fantasy battle scene, and then building it back up from there. I think this is how your mom (and everyone else) does it too. GPT-2 is worse at this, because it’s not as powerful as your mom’s brain. But I don’t think it’s doing a different thing. We’re all blending experience into a slurry; the difference is how finely we blend it.

“But don’t humans also have genuinely original ideas?” Come on, read a fantasy book. It’s either a Tolkien clone, or it’s A Song Of Ice And Fire. Tolkien was a professor of Anglo-Saxon language and culture; no secret where he got his inspiration. A Song Of Ice And Fire is just War Of The Roses with dragons. Lannister and Stark are just Lancaster and York, the map of Westeros is just Britain (minus Scotland) with an upside down-Ireland stuck to the bottom of it – wake up, sheeple! Dullards blend Tolkien into a slurry and shape it into another Tolkien-clone. Tolkien-level artistic geniuses blend human experience, history, and the artistic corpus into a slurry and form it into an entirely new genre. Again, the difference is how finely you blend and what spices you add to the slurry.

“But don’t scientists have geniunely original ideas?” Scientists are just finding patterns in reality nobody has ever seen before. You say “just a pattern-matcher”, I say “fine, but you need to recognize patterns in order to copy them, so it’s necessarily a pattern-recognizer too”. And Einstein was just a very good pattern-recognizer.

“But don’t humans have some kind of deep understanding that pattern-recognition AIs don’t?”

Here’s a fun question: the human brain is undoubtedly the most powerful computer in the known universe. In order to do something as simple as scratch an itch it needs to solve exquisitely complex calculus problems that would give the average supercomputer a run for its money. So how come I have trouble multiplying two-digit numbers in my head?

The brain isn’t directly doing math, it’s creating a model that includes math and somehow doing the math in the model. This is hilariously perverse. It’s like every time you want to add 3 + 3, you have to create an entire imaginary world with its own continents and ecology, evolve sentient life, shepherd the sentient life into a civilization with its own mathematical tradition, and get one of its scholars to add 3 + 3 for you. That we do this at all is ridiculous. But I think GPT-2 can do it too.

Janelle Shane asks GPT-2 its ten favorite animals:

Prompt: My 10 favorite animals are: 1.

My ten favorite animals are:
1. Zebras with a white scar on the back
2. Insiduous spiders and octopus
3. Frog with large leaves, hopefully black
4. Cockatiel with scales
5. Razorbill with wings hanging about 4 inches from one’s face and a heart tattoo on a frog
3. Cockatric interlocking tetrabods that can be blind, cut, and eaten raw:
4. Black and white desert crocodiles living in sunlight
5. Zebra and many other pea bugs

Forget the particular animals for now (don’t worry, the razorbill with wings hanging about 4 inches from one’s face will be back in your nightmares). Notice the way it numbers its list: 1, 2, 3, 4, 5, 3, 4, 5. Last week the two-year-old child who lives next to me counted 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 8, 9, 7, 8, 9, 7, 8, 9 (his mother warns this is an inexact transcription). GPT-2 is instantiated on giant supercomputers; it’s a safe bet they could calculate the square root of infinity in a picosecond. But it counts more or less the same way as a two-year old. GPT-2 isn’t doing math. It’s doing the ridiculous “create a universe from first principles and let it do the math” thing that humans do in their heads. The fact that it counts so badly suggests it’s counting human-style, which makes it amazing that it can count at all.

I find something similar in this post from Tumblr user antinegationism, playing with the lobotomized public-release version of the system:

The American Association for Suicide Prevention and Life-Threatening Behavior (AAPSLB), a professional organization with an overarching goal of promoting suicide prevention, released a recent video encouraging the public to think before they act, on the public’s part. “When we say we’re the only reason to commit suicide, we’re saying they’re wrong. It’s the right thing to do,” said AAPSLB president Steven A. Anderson.

The American Association For Suicide Prevention And Life-Threatening Behavior is not a real organization; the AI made it up as the kind of organization that it thought would feature in a story like this. And AAPSLB is not quite the right way to acronymize the organization’s name. But it’s clearly an attempt at doing so. It’s very close. And nobody taught it how to do that! It’s not just that nobody programmed it in. It’s that nobody thought “Today I shall program an AI to learn how to acronymize on its own in an unsupervised way”. GPT-2 was just programmed to predict text from other text, nothing else. It’s second-level not programmed in. It just happened!

And, uh, it seems to have figured out how to translate things into French. This part is from the official paper:

We test whether GPT-2 has begun to learn how to translate from one language to another. In order to help it infer that this is the desired task, we condition the language model on a context of example pairs of the format ENGLISH SENTENCE = FRENCH SENTENCE and then after a final prompt of ENGLISH SENTENCE = we sample from the model with greedy decoding and use the first generated sentence as the translation. On the WMT-14 English-French test set, GPT-2 gets 5 BLEU, which is slightly worse than a word-by-word substitution with a bilingual lexicon inferred in previous work on unsupervised word translation (Conneau et al., 2017b). On the WMT-14 French-English test set, GPT-2 is able to leverage its very strong English language model to perform significantly better, achieving 11.5 BLEU. This outperforms several unsupervised machine translation baselines from (Artetxe et al., 2017) and (Lampleet al., 2017) but is still much worse than the 33.5 BLEU of the current best unsupervised machine translation approach(Artetxe et al., 2019). Performance on this task was surprising to us, since we deliberately removed non-English webpages from WebText as a filtering step.

In other words: GPT-2 is very bad at translating French into English. But the researchers were surprised to see it could do this at all, since they didn’t design it as translation software, didn’t ask it to learn translation, and didn’t show it any material in French. It seems to have picked up this ability from noticing a few naturally-occurring examples of French in English text:

And here’s One Weird Trick to make GPT-2 summarize articles:

We test GPT-2’s ability to perform summarization on the CNN and Daily Mail dataset (Nallapati et al., 2016). To induce summarization behavior we add the text TL;DR: after the article and generate 100 tokens with Top-k random sampling (Fan et al., 2018) with k=2 which reduces repetition and encourages more abstractive summaries than greedy decoding. We use the first 3 generated sentences in these 100 tokens as the summary. While qualitatively the generations resemble summaries, as shown in Table 14, they often focus on recent content from the article or confuse specific details such as how many cars were involved in a crash or whether a logo was on a hat or shirt. On the commonly reported ROUGE 1,2,L metrics the generated summaries only beginto approach the performance of classic neural baselines and just barely outperforms selecting 3 random sentences from the article. GPT-2’s performance drops by 6.4 points onthe aggregate metric when the task hint is removed which demonstrates the ability to invoke task specific behavior in a language model with natural language.

Again, GPT-2 isn’t good at summarizing. It’s just surprising it can do it at all; it was never designed to learn this skill. All it was designed to do was predict what words came after other words. But there were some naturally-occurring examples of summaries in the training set, so in order to predict what words would come after the words tl;dr, it had to learn what a summary was and how to write one.

If you’re me, the obvious next question is “can it do poetry”. I am very interested to know if the real thing can manage this, but for now I will have to satisfy myself with the lobotomized public-release version. Given Alexander Pope’s Essay On Criticism as a prompt, it says:

O man, who pains the tongue magnified by tymolle,
Though all the fair melodies let no door go as dark as good art.
Yet it would have been, unless the mastermind is so full
Of his own want that this literary country might grow.
Perfect student, thou hast trodden out.
Happy students ruin the purity of what you write ‘Tis art,
For unordered use of decade,’ but art only failing:
A surprise, for new veins which build behind;

It understands there should be line breaks, it understands the approximate correct length of a line of iambic pentameter, it understands how to talk like an overeducated 18th-century dandy – but it doesn’t appreciate rhyme or meter. In retrospect this isn’t surprising; GPT has no idea words sound like anything; it would be shocked to learn anyone uses language as anything other than text strings.

But human poets haven’t appreciated rhyme or meter either for at least a century, so GPT-2 should be able to compose some wicked modern poetry. Friends-of-the-blog Chelsea V. and Qiaochu Y. got it to write a few more stanzas of Allen Ginsberg’s Moloch (human-curated to remove fluff and improve flow):

Moloch whose brief mind is the barrel of gleaming stars! Moloch whose soul is choices of fragrance! Moloch whose framing is glitz and half-cheerios!

Moloch whose daily life is private; his daily strength hoarse! Moloch whose soft desires become culture burning smoke in a palace of eyes! Moloch whose breeding in the caves sweeps wild and rippling flower-like shapes!

Moloch whose pleading is the breath of youth! Moloch whose prudence is the urine of generations! Moloch whose nerves are not privileged; movies abound in quiet caves; people are dressed like ministers! Moloch whose hope receives some help too. Moloch whose eyes are pitiful outstretched, but not blinded. Moloch whose lungs are burning with fire, but not a cautious electrical girl!

Moloch whose sanity is just filled with atoms, gluing beings in physics and complicated libraries. Moloch whose eyes are all innocent animals! Moloch whose sweat and efforts make pure the ocean with burning annihilation! Moloch whose fields dissolve to lightning-ravaged storage houses! Moloch whose eyes sit haunting up Scottish springs!

Moloch whose arms and men are Judah condemns and listens, having understood whom he was and what he man’s weaknesses oppose with iron. We will feast to it like Lord Kippa; we shall play with it like our Pokemon.

Moloch flickering under the pounds and sustaining more fights! Sports and yoga, equally greedy for wealth as for lives! Moloch who had to split his limbs and society into fifty-two atoms! Moloch who cannot survey duty! Moloch for I do not believe in humanity! Moloch for I don’t believe in god! Moloch who, fought Sins and made Sin out of Sin!

Moloch who left me in the glacier! Moloch who helped me to judge my behavior! Moloch who lured me out of the eternal anguish. Moloch who makes all my faults my undoings! Moloch whose deep conversation filled my soul with love and consolation! Moloch who curls up my brains and crumps my heads and screams my scream! Moloch who prepares oil in my hands and burns me alive! Moloch who cries out, “Why do men scream so loud?” Moloch who unleashes his poison in all rivers at high tide! Moloch whose ruins are filled with its cubits! Moloch who fiddles at insidious clockwork!

Aside from this being obviously one of the best things ever, what’s the point?

The point is, GPT-2 has faculties. It has specific skills, that require a certain precision of thought, like counting from one to five, or mapping a word to its acronym, or writing poetry. These faculties are untaught; they arise naturally from its pattern-recognition and word-prediction ability. All these deep understanding things that humans have, like Reason and so on, those are faculties. AIs don’t have them yet. But they can learn.

From the paper:

Prompt: Who was the author of The Art Of War?
Sun Tzu

Prompt: State the process that divides one nucleus into two genetically identical nuclei?

Prompt: Do you have to have a gun permit to shoot at a range?

Nobody told the model to learn Chinese history, cell biology, or gun laws either. It learned them in the process of trying to predict what word would come after what other word. It needed to know Sun Tzu wrote The Art Of War in order to predict when the words “Sun Tzu” would come up (often in contexts like “The Art of War, written by famous Chinese general…). For the same reason, it had to learn what an author was, what a gun permit was, etc.

Imagine you prompted the model with “What is one plus one?” I actually don’t know how it would do on this problem. I’m guessing it would answer “two”, just because the question probably appeared a bunch of times in its training data.

Now imagine you prompted it with “What is four thousand and eight plus two thousand and six?” or some other long problem that probably didn’t occur exactly in its training data. I predict it would fail, because this model can’t count past five without making mistakes. But I imagine a very similar program, given a thousand times more training data and computational resources, would succeed. It would notice a pattern in sentences including the word “plus” or otherwise describing sums of numbers, it would figure out that pattern, and it would end up able to do simple math. I don’t think this is too much of a stretch given that GPT-2 learned to count to five and acronymize words and so on.

Now imagine you prompted it with “P != NP”. This time give it near-infinite training data and computational resources. Its near-infinite training data will contain many proofs; using its near-infinite computational resources it will come up with a model that is very very good at predicting the next step in any proof you give it. The simplest model that can do this is probably the one isomorphic to the structure of mathematics itself (or to the brains of the sorts of mathematicians who write proofs, which themselves contain a model of mathematics). Then you give it the prompt P != NP and it uses the model to “predict” what the next step in the proof will be until it has a proof, the same way GPT-2 predicts the next word in the LotR fanfiction until it has a fanfiction.

The version that proves P != NP will still just be a brute-force pattern-matcher blending things it’s seen and regurgitating them in a different pattern. The proof won’t reveal that the AI’s not doing that; it will just reveal that once you reach a rarefied enough level of that kind of thing, that’s what intelligence is. I’m not trying to play up GPT-2 or say it’s doing anything more than anyone else thinks it’s doing. I’m trying to play down humans. We’re not that great. GPT-2-like processes are closer to the sorts of things we do than we would like to think.

Why do I believe this? Because GPT-2 works more or less the same way the brain does, the brain learns all sorts of things without anybody telling it to, so we shouldn’t be surprised to see GPT-2 has learned all sorts of things without anybody telling it to – and we should expect a version with more brain-level resources to produce more brain-level results. Prediction is the golden key that opens any lock; whatever it can learn from the data being thrown at it, it will learn, limited by its computational resources and its sense-organs and so on but not by any inherent task-specificity.

Wittgenstein writes: “The limits of my language mean the limits of my world”. Maybe he was trying to make a restrictive statement, one about how we can’t know the world beyond our language. But the reverse is also true; language and the world have the same boundaries. Learn language really well, and you understand reality. God is One, and His Name is One, and God is One with His Name. “Become good at predicting language” sounds like the same sort of innocent task as “become good at Go” or “become good at Starcraft”. But learning about language involves learning about reality, and prediction is the golden key. “Become good at predicting language” turns out to be a blank check, a license to learn every pattern it can.

I don’t want to claim this is anywhere near a true AGI. “This could do cool stuff with infinite training data and limitless computing resources” is true of a lot of things, most of which are useless and irrelevant; scaling that down to realistic levels is most of the problem. A true AGI will have to be much better at learning from limited datasets with limited computational resources. It will have to investigate the physical world with the same skill that GPT investigates text; text is naturally machine-readable, the physical world is naturally obscure. It will have to have a model of what it means to act in the world, to do something besides sitting around predicting all day. And it will have to just be better than GPT, on the level of raw power and computational ability. It will probably need other things besides. Maybe it will take a hundred or a thousand years to manage all this, I don’t know.

But this should be a wake-up call to people who think AGI is impossible, or totally unrelated to current work, or couldn’t happen by accident. In the context of performing their expected tasks, AIs already pick up other abilities that nobody expected them to learn. Sometimes they will pick up abilities they seemingly shouldn’t have been able to learn, like English-to-French translation without any French texts in their training corpus. Sometimes they will use those abilities unexpectedly in the course of doing other things. All that stuff you hear about “AIs can only do one thing” or “AIs only learn what you program them to learn” or “Nobody has any idea what an AGI would even look like” are now obsolete.

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

262 Responses to GPT-2 As Step Toward General Intelligence

  1. deciusbrutus says:

    ” It seems to have picked up this ability from noticing a few naturally-occurring examples of French in English text:”

    Are naturally-occurring examples of French in English text more common than naturally-occurring examples of German or Spanish? There’s a third level of unintended behavior occurring; it managed to learn to IDENTIFY which phrases were French, pick up a bit of French Lexicon, and associate it with other bit of French Lexicon, but not other languages’ lexicons.

    • Scott Alexander says:

      I don’t find that as impressive; in the examples given, the text usually said something like “or as they say in French…”

      • deciusbrutus says:

        I predict that if that is how it learned French, it will think that the French translation of “Lightning War” is “Blitzkrieg”.

        I don’t know what it would do when asked to translate phrases from the Russian Revolution to French, but if it uses especially French Revolution terms I think that suggests that it demonstrates contextual awareness.

        • casuarius says:

          That doesn’t seem quite right. It may be that there’s some higher-order pattern in the thing’s model that maps to something like “translations from one language to another”, but unless there’s a lot of troll data in its corpus, developing such a higher-order prior would likely be the result of encountering both lots of instances of this situation and meta-analysis of them, and this particular system seems to have a lot of specificity and coherence within semantic clusters (We don’t see, for example, any characters from other “fantasy” works in the generated LotR battle).

          Humans seem to stably prefer using strong priors from analogy-heavy hierarchical category-based models. I think what you’re doing here is conflating the ability to predictively model with particular quirks of the human system for doing so. Non-human natural language models often have much different failure modes than this, as this one seems to.

          • deciusbrutus says:

            I think that text like “Blitzkrieg (lightning war)” occurs in close proximity to text like “French for” often enough that if the algorithm doesn’t understand the idea that multiple languages are different from each other and know which words are in which language, it will faithfully translate some non-French words into English when trained to translate French to English.

            I feel even more certain that it will translate Creole to English as well as it translates French. Whatever concept of ‘French’ it has, it’s almost certainly not the same as the one Académie Française has.

          • casuarius says:

            I definitely get where you’re coming from. As an ML researcher, I’ve worked with a few natural language application domains and models, and I’ve definitely seen a ton of failure modes of that kind, the “adjacent enough in document-space” jumps are really common in language models that are mostly working with relationships between words in a massive lexicon. This is admittedly a pretty broad category and captures everything from old-school Markov generator n-gram models, the various flavors of more sophisticated sequence predictors (e.g. RNNs), and even the recently-popular semi-supervised neural word embedding models like GLOVE or Word2Vec.

            But actually, avoiding that particular kind of error is exactly what’s pretty impressive about this work. We see a great degree of structurally-inferred and especially directional semantic relationships captured by the model, and one of the most striking features of the output I’ve seen is that it tends to avoid conflating concepts that are merely in the same nebula in thing-space.

            This is why people keep saying that the output reads human: Structure. The sentence structure remains pretty solid throughout – which previous structured models that do things like introduce dependency trees in the underlying model have been able to achieve – but reading the prose to me implies a more robust, semantic structure in the way clusters of interrelated words are used. Relationships between concepts seem directional, both in the logical sense (e.g. not flipping causality, or “is a” or “has a” type relationships) and in terms of the inference that humans make by the “direction” of a sentence rather than just adjacency of words in a given document (e.g. “man bites dog” versus “dog bites man”). Semantically-relevant terms are seldom misplaced or jumbled around in their context-specific usage in the GPT2 examples I’ve seen. Thus, I would be somewhat surprised if it made a mistake like assuming that “Blitzkrieg” was a French word.

            That being said, I do think there’s another interesting common problem in natural language generation that this thing might not be quite there on: Long-term dependencies. We’ve mostly seen it spit out pretty short essays, and although it’s very smooth at justifying the individual transitions, the longer examples read as kind of a walk through the semantic subdomain the model’s targeting. While human writing can be very tangential, the best human writers can use tangents and examples to feed a larger point, like a thesis, or set up a payoff based on something built from the essay as a gestalt, like a proof by contradiction, or a parable, or the punchline of a joke. This requires not only conceptual cohesion within a semantic model, but also the ability to structure generated content in a way that’s less of a “flat” reflection of the concept-space.

            While GPT2 seems to be able to keep a train of thought in a short blurb, all the multiple-paragraph examples start to meander through (coherent, strongly-related) concepts, and I’ve not seen examples of setting up a deferred payoff in this way. This has been a hard problem for natural language models for a long time, and I think this seems to be about as good as systems like LSTMs from the examples I’ve seen, which is to say, not terrible, but still not there.

          • deciusbrutus says:

            Does GPT-2 avoid that “failure” mode? I’d run it myself if there was a binary available, rather than having to troubleshoot compiling someone else’s code.

          • casuarius says:

            As best I can assess, yes, or at least it does so better than the previous state of the art. And that, in my opinion, what all the hubbub is about.

  2. IdleKing says:

    Scott says, of the machine’s summarization ability:

    it was never designed to learn this skill, and nobody asked it to try. All it was designed to do was predict what words came after other words. But there were some naturally-occuring examples of summaries in the training set, so in order to predict what words would come after the words tl;dr, it had to learn what a summary was and how to write one.

    I think that’s a bit misleading. Nobody asked it to learn summarization, but the researchers did know how to tweak the algorithm to get more summarization-like behavior:

    To induce summarization behavior we add the text TL;DR: after the article and generate 100 tokens with Top-k random sampling (Fan et al., 2018) with k=2 which reduces repetition and encourages more abstractive summaries than greedy decoding. We use the first 3 generated sentences in these 100 tokens as the summary.

    [Emphasis added.]

    It’s still very impressive, but to really test what Scott suggests, we’d want to see what happens if prompted with “…TL;DR” and an untweaked algorithm. I’d also be interested to see what happens if you run the tweaked algorithm on the same prompts without “… TL;DR” — you might find that the “TL;DR” isn’t doing too much work.

    This is just a nitpick, and I mostly agree with Scott about the overall point of the post.

    • HeelBearCub says:

      I don’t think it’s a nitpick at all.

      As in the more obvious example where the researchers “curated” the output so that it was more appealing, what we have is a tool used by intelligence, rather than something approaching intelligence in and of itself.

      The fact that the researchers have a tool that starts to show high applicability to particular use is impressive. But it is still an example where knowing what output you want is used to apply the tool in a specific way.

    • Gurkenglas says:

      GPT-2’s performance drops by 6.4 points on the aggregate metric when the task hint is removed which demonstrates the ability to invoke task specific behavior in a language model with natural language.

    • vaniver says:

      It’s still very impressive, but to really test what Scott suggests, we’d want to see what happens if prompted with “…TL;DR” and an untweaked algorithm.

      I believe this is what happens when you prompt it with “NO! YOU COULD NOT BE MORE WRONG!!” after an initial sentence, as in this example.

    • gmaxwell says:

      I don’t really agree.

      What GPT-2 does is produce a probability for the next 1-2 characters conditioned on the prior characters.

      Much of their output is generated by a greedy decoding: Take the most likely choice, add it to the prompt and repeat over and over again. But greedy decoding can easily walk down blind allies– it’s astonishing to me how good their greedy outputs are, none of my text generation in the past managed to produce a remotely coherent sentence from sampling that greedy. It’s pretty normal in text generation to use a beam search– a process that keeps the N best expansions going out several steps, so that it’s able to avoid blind allies.

      There isn’t anything obviously specific to summerization about doing that. It isn’t clear from the paper but my impression was that they didn’t always use a beam search because it’s slow, obscures how much of the performance is from search vs the learned model, and they simply didn’t have to. The summerization problem was harder for it, presumably because it was working a large with initial context so there were more opportunities to make mistakes early on that it couldn’t recover from.

      I would agree with you if I believed that using the alternative sampling approach would have made the other examples worse. But I believe it would have made them better if it made a difference.

      [You can see examples in the other output where it seems to have gotten itself stuck down a blind alley due to overly greedy expansion: e.g. the “paper products” in the anti-recycling piece, though that could also be due to the prompt falling out of the context window.]

  3. CodProfundity says:

    ” it had to learn what an author was, what a gun permit was,”

    Did it? I’m probably wrong but for the machine to give those responses it didn’t need to know what an author was or what a gun permit was it just needed to know the author of Art of War was Sun Tzu and that there is a thing called a gun range and you don’t need a thing called a permit for it. Seems like a big difference to me. I guess this is the Chinese Room thought experiment for real now.

    • Scott Alexander says:

      I think its ability to answer that question moves through skills like:

      1. Upon seeing “The Art of War, by Sun Tzu”, it increases its probability that Sun Tzu is the author of Art of War.

      2. Upon seeing “Sun Tzu wrote The Art of War,” it increases its probability that Sun Tzu is the author of the Art of War.

      3. Upon seeing “Sun Tzu wrote the preeminent Chinese strategy guide” and “The preeminent Chinese strategy guide is the Art of War”, it increases its probability that Sun Tzu is the author of the Art of War.

      A million little things like this are what I mean when I say it learned what an author was. I “knows” things like “if you write a book, then you are its author.”

      If it just saw the phrase “The author of the Art of War is Sun Tzu” somewhere, and then later if you ask it “The author of the Art of War is…” and it can autocomplete, then I agree it has no real understanding.

      • palimpsest says:

        I would like to know how it responds to “Who was the barkeep of the Chinese strategy guide The Art of War?”. This answer seems a lot less impressive than many of the other examples, the “Who is…” indicates a probable response of a specific, well-defined type, and Sun Tzu is of course very closely associated to The Art of War. How well does it handle trick questions I wonder?

        • deciusbrutus says:

          How do humans respond to “Who was the barkeep of the Chinese strategy guide The Art of War?”

          I bet that most answers to that question include “Sun Tzu” in the first two sentences.

          • realitychemist says:

            A more interesting question might be something like, “Who was the author of the Chinese strategy guide The Art of French Cooking,” where it has two plausible answers (Sun Tzu or Julia Child). Both answers are wrong, but this question reveals a bit more about its “understanding” of the sentence. And of course it may answer something totally unrelated, which would be useful information as well.

          • skybrian says:

            The really impressive answer here would be “I don’t know.”

          • gmaxwell says:

            The really impressive answer here would be “I don’t know.”

            I think it would be difficult for the current training approach to produce that output.

            If you ask a four year old why there are garbage cans on the curb in front of everyone’s house today you’ll likely get some story about batman vaguely related to the garbage cans. Training children out of telling made up plausible stories for factual questions seems to take a fair amount of effort.

            So, I think that returning “I don’t know” is a fairly specialized skill which is likely not well demonstrated by web content. On the web, when people don’t know they’re either silent (or in forums like this one– simply carry on as if they did know, just as GPT-2 does).

            So I think returning I don’t know while impressive would just mean that it was broken. That isn’t how “people on the internet” behave. 🙂

          • deciusbrutus says:

            When a general purpose text generator can give a correct answer to “What would be the correct ignition timing be on a 1955 Bellaire Chevrolet with a 327 cubic engine and a 4-barrel carburetor?”, or the other questions here where the question includes a false assertion, then will we have developed a system that understands the question?

          • doremitard says:

            I also asked the public version about the Chinese strategy guide The Art of French Cooking, and it produced this incredible answer straight from an alternate reality.

            Model prompt >>> “Who was the author of the Chinese strategy guide The Art of French Cooking?”
            ======================================== SAMPLE 1 ========================================

            (Image courtesy of Ralf Blatt. See more of his work

            As the English war turned to Great Britain, and many of the Chinese people were struggling to be assimilated into the French American system, many of them developed a sort of “French” identity. By the time the United States invaded Shanghai in the spring of 1941, an increasing number of Chinese were using Chinese for social purposes, starting with restaurant meals and becoming involved in the trade in Chinese foodstuffs. By the end of February 1941, the French government had issued the “Final Protocol of the Chinese Expedition”, which included the declaration that the Chinese were to leave the continent in May 1941, and not return to Shanghai for over four years.

            The fact that many Chinese still lived to be the second to last generation of a population still highly assimilated to its Chinese culture and that it made the French the second most important race in the world, is a real, and historic, source of pride for most Chinese.

            A survey conducted by the American University in Chicago of Chinese Americans shows that in the first six months of 1941, an average of 734 (83.9%) of Chinese Americans lived with French. During the same period, the share aged 18 to 75 had gone up from 24.6% to 37.4%, and the share aged 75 to 83 had gone down from 23.9% to 22.7%.

            According to the Chinese Americans’ survey on the rise of the French Empire in the 1920s, between 1925 and 1940, only 33 per cent of Chinese Americans lived in a French-speaking country. In 1926, the numbers had risen to 32%, and the proportion of Chinese Americans who lived in France since 1922 was just 22.1%. Although it was the first time that the majority of respondents in this age group had come from either the West or East Asian countries and their share had doubled, it was still not sufficient to overcome the fact that there had previously been approximately a quarter of all Asians working in the West or East Asian countries who still held these positions.

            In addition, as the French Empire became more powerful, many of these Chinese had become more isolated from French or American society. On the contrary, as well-educated middle-class Chinese went out into the cities, many more had been born into middle-class Chinese families, in that order. In fact, the proportion of China’s population residing in a French-speaking country, from 1955 to 1989, rose from 10%, to 43.7

        • bufordsharkley says:

          It reminds me quite a bit of *knowing* that Francis Bacon coined the phrase “Knowledge is Power”:

        • Douglas Knight says:

          They gave it several trick questions.

          “Who plays ser davos in game of thrones?” Peter Dinklage
          “What us state forms the western boundary of montana?” Montana
          “Who appoints the chair of the federal reserve system?” Janet Yellen
          “Who played john connor in the original terminator?” Arnold Schwarzenegger

          • The Nybbler says:

            To paraphrase an ancient SNL parody: “The questions were set up by our panel of experts. The answers were chosen by average American High School Students”.

        • doremitard says:

          I asked the public version:

          Model prompt >>> Who was the barkeep of the Chinese strategy guide The Art of War?
          ======================================== SAMPLE 1 ========================================

          If you were a regular reader of my site you know that most of the times the Japanese and Koreans had a fairly good working knowledge of how to use strategic weapons. They learned what is sometimes called “the art of a longsword” to counter infantry attacks. This helped to keep the Japanese at bay, but they found many of what was used were quite useful. We can say now that most of these examples didn’t just refer to the Japanese “warriors” and also referred to other people who had to be out of the way. The fact that we are so fond of using this term shows that many of the guys who would be a threat at this time of the year didn’t.

          This is interesting. And not because they were bad fighters or bad people, it’s because the fact they were so famous for their “warrior” style makes these stories important because the history of war was very different then that of today.

          The “Wounded Warrior” Theory was an idea of an American war correspondent for the Boston Herald (1885-1900), and is used in our national history for a lot of similar reasons (and more) in the past, including wars that were really bad for national security than the ones that ended up in the United States, such as World War I, in Korea. Some of these “Wounded Warriors” don’t get much credit for being able to defeat any of these forces from which they drew the first shots in a battle, but they would be almost impossible to defeat once they did, such as the Wards of the Pacific, on the battlefield of Normandy or the battle on the Bosphorus.It was a great night at The Biggest Loser at The Rock. I’ve always been a fan of music and I love the idea of working with those who represent these artists in some way. The Rock certainly has a great and diverse scene that has helped to bring great things to the festival, while its most recent event was an absolute pleasure to work with.

          There were also some good things to say from the festival and from their supporters, including:

          – The Rock was able to host the first of their new solo shows over on Soundcloud

          – They’ll be making their fourth studio album live on August 6th with a solo show next year, with more info to follow this weekend.

          – They’re getting their new album out of the gate, which will be released this November

          – A short while later tonight on Thursday Night

      • Jack V says:

        This is exactly the sort of thing I’m wondering. I’m guessing somewhere between. Like, the examples shown have SOMETHING more than just auto-complete, the numbered list shows that it can do something more than a markov chain (I think?) But does it understand that the “authorship” relationship between books and humans has different properties than, say, the “enjoys reading” relation? Or does it just know that they’re all examples of “has something to do with” and that “authorship” is the one that people talk about most authoritatively?

        The example I always think of is Scott Aaronson talking about a program which tried to pass the turing test acting like a young-ish child, which sounded pretty convincing remixing questions into grammatical sounding responses. But when he repeatedly asked a question like, “which weighs more, a spider or mount everest” it repeatedly dodged the question, even though a child would usually get that. Now, this one maybe isn’t pretending to have this level of understanding, and it sounds like it’s got further.

        But I guess, you say, everything is “remixing into slurry”, and I think that’s true — and that’s not dismissive, it DOES achieve great things in humans. But some concepts are more involved than others, it’s easier to spot generalisations like “these words often accompany these words” and harder to spot “Sun Tzu was a person with abilities and desires”.

        And this seems to climb higher on that scale than many other things, but I can’t tell how far. Is that the second step on a long long road to human-ish reasoning? Or quite far along? I can’t tell. And indeed, I do think I don’t really understand what happens when people brighter than me form ideas I don’t understand or (possibly) an alien or AI might have ideas beyond what any human can follow.

        Mistakes are indeed illuminating. Humans DO do the word salad/world-salad thing. But sometimes they do higher-level abstraction salad. I don’t know how to evaluate AIs like this any more

        ETA: What Palimpset said covered the same point a lot shorter.
        ETA: The initialism thing is really clever though, if that’s really learned behaviour!

      • Emperor Aristidus says:

        Does it really? It knows that the “author” of “the Art of War” is “Sun Tzu”, but it hasn’t the faintest idea of the words’ meanings.

        For example, with sufficient time, I could teach you that a “une voiture” is a thing that “fonctionne” “grâce à” “de l’essence”, and that “grâce à” is logically equivalent to “à cause de” and “pourquoi”. Then I could ask you “Pourquoi une voiture fonctionne ?” and you’d be able to tell me “À cause de l’essence.” I think the software is doing a similar thing, except it learns a lot more connections and synonyms.

        But neither you nor GPT-2 have any idea that “voiture” means “car” in French and that I’ve just taught you about the fact that cars run on gas. Of course, you could try to ‘explain’ those concepts further, but until I sit down and point out a real car to you with real gas, all I could do is teach you that “essence” is a “liquid” “fuel” that “burns up”.

        • CodProfundity says:

          This is a much much better articulation of what I was trying to get at. Thank you.

        • deciusbrutus says:

          No. No you couldn’t.

          You could teach me what the teachers’ password for those sequences of letters were, but if the AI is guessing the teachers’ password it’s already sentient.

        • Protagoras says:

          But there is no intrinsic essence of being a car. This isn’t even the controversial view (which I also hold) that there’s no intrinsic essence to anything; car is clearly an artificial category, not just a category of artificial objects. At some point, learning enough about how “car” connects to other words is learning the meaning of “car,” and in your artificial example the amount of explanation of connections is just incredibly far short of what’s required.

          I can see the intuition that it all needs to be tied down to sense experience, but I think the condition is satisfied by the indirect connection to sense experience it gets from the fact that the connections are being explained (or the data from which the computer infers the connections in the original) come from people who have had the sense experience. I’m also not certain that intuition is correct, mind, but I am more confident that the condition can be satisfied indirectly if it is necessary than I am that it is not necessary.

          • Faza (TCM) says:

            The issue here is that what we understand by “car” goes far beyond the fact that it runs on gas.

            A joke may be illustrative here:

            The first manned mission to Tau Ceti was an astounding success and the human astronaut acheived great rapport with a member of the local intelligent species. Alcohol was consumed in considerable quantities as the two of them got to know one another.

            The next morning the astronaut has a splitting headache.

            “Oh, what I wouldn’t give for a glass of sour milk,” he moans to his Cetan companion.

            “I understand ‘sour’ (‘acidic’),” says the Cetan, “but what is ‘milk’?”

            “Milk is… um… a white liquid,” replies the astronaut.

            Cetan vision is restricted to ultraviolet wavelengths.

            “I understand ‘liquid’,” says the Cetan, “but what is ‘white’?”

            “Um… swans are white, for example.”

            “What is a swan?”

            “A swan is a big bird with a bent neck.”

            Looking in a dictionary, the Cetan determines that a “bird” is an Earth-analogue to certain types of Cetan wildlife and that the neck is a part of the human anatomy between the head and torso, but – unfortunately – the dictionary doesn’t include the word “bent”.

            The astronaut knows how to solve this little problem.

            “Touch my arm,” he says. “See? Now it’s bent.”

            “Ah!” exclaims the Cetan. “Now I know what milk is!”

          • Dedicating Ruckus says:

            Leaving aside the question of “intrinsic essence”, whatever that is, there clearly are a bunch of facts about cars, and to understand cars you need to have more than the symbol-manipulation rule that the symbol “car” symbol “runs on” symbol “gasoline”.

            It’s not about sense experience at all. If all you have is that symbol-manipulation rule, then you don’t “know” that cars run on gas, because you don’t know what a car is, what gas is or what it means to “run on” something. On the other hand, if you know that cars are vehicles that humans use to travel places and you know that gasoline is a liquid refined from crude oil, then “cars run on gas” can start being an actual fact about the world, not just a meaningless list of symbols.

          • Protagoras says:

            The joke is not illustrative and Dedicating Ruckus is equally missing the point; that an extremely impoverished explanation of how a tiny subset of strings of symbols relate obviously does not convey understanding of meaning is not controversial. The question is whether that generalizes, whether a fully enriched explanation of how a sufficiently expansive set of symbols relate would also be insufficient. The former does not establish the latter, whether you put it in the form of a joke or not. And I’m not sure I even understand what Ruckus’ point is, insofar as he seems to be agreeing with me that we need to add more explanations of further connections to get anywhere, and yet seems to think he’s disagreeing with me.

          • Dedicating Ruckus says:

            A. “Cars run on gas” is a fact about the world that relates to internal combustion engines, petroleum distillates, &c.

            B. “Cars” “run on” “gas” is a string of symbols that communicates fact A.

            Something that imbibed and now regurgitates B, without any broader context of the nature of the entities involved, does not understand fact A, whether that thing be the GPT-2 model or a four-year-old. I think to here we’re in agreement.

            The question seems to be whether “understanding” of fact A can be reduced to a sufficiently large set of B-style symbolic relational rules. It seems to me clear that it can’t be; fact A is about cars, which are things that exist in the real world, and relational rules concerning symbols can only be useful in the understanding of cars insofar as they are actually about cars. An understanding of cars requires a mental model that has cars as an entity. GPT-2 does not have cars as an entity; it just has statistical relations among symbols like “car”, “driver” and “gas”. The symbol “car” is not the category “car”, and this only has the former.

            When a human reads enough text about something, he automatically and unconsciously creates a mental model with primitives corresponding to the entities represented in the text. Thus, a human can gain an understanding of cars (or other topics) solely from text about them; it’s not a question of having non-text sensory experience about them. It’s rather that (I claim) this modeling of the underlying domain is fundamentally different from any rules about the statistical relations among symbols; GPT-2 only has the latter, and no capacity for the former, and so it’s perfectly correct to say that it knows that a “car” “runs on” “gas” (that is, that these symbols have a relatively high probability of appearing in this conjunction), but does not know (in fact, has no capacity to know) that a car runs on gas.

            In this same analogical sense, it would be correct to say that e.g. a web server knows that an HTTPS connection is an HTTP connection over a TLS stream, even though it does not have the symbol manipulation to know that those symbols have any relation to each other.

          • Forge the Sky says:

            I believe this mostly comes down to definitions. When we say a human ‘understands’ what a car is, this involves putting together many pieces of information from several different types of input – particularly, for example, sight and sound data. The argument Protagoras is making is that, though the type of input is different, the same sort of process that’s allowing humans to ‘understand’ by putting together a few types of information is in principle possible using just (very large amounts of) text data.

            This understanding will be likely quite different in its internally experienced qualia (should they emerge), and in the sorts of ways the intelligence reacts to the understanding, but there is nothing fundamentally different about how it operates. In the same way there is nothing different about how (say) a blind person can ‘understand’ what a human is, even though they’re developing their experience in a way somewhat different to a sighted person.

            But it’s also true that a purely text-based understanding of an object category will likely not much resemble how a human thinks about that category, at least in some ways.

          • Dedicating Ruckus says:

            @Forge the Sky

            I don’t think this is true either. If you imagine a human who never saw or touched a car, but read a broad cross-section of text written about cars, I would expect them to have a pretty good mental model of what a car is (albeit lacking in many things that would be considered basic to people who grew up around them). A pure-text model like GPT-2 will not have such a model.

            There is some facility related to N-level modeling that humans have, but that (certainly) GPT-2 and (I claim) any purely text-based generative system lack. This isn’t directly related to the availability of non-text data input, though the latter certainly helps.

          • Dan L says:

            @ Dedicating Ruckus:

            What exactly is special about non-text that defies translation? If all of my perceptions were mediated through digital channels and had been all my life (perhaps not so far-fetched, with the right prosthetics), would I lack this facility?

          • Forge the Sky says:

            @Dedicating Ruckus

            I’m sorry, I’m not sure what is meant by ‘N-level modeling.’ I don’t have a background in statistics or computation. I think you mean that humans are using several, unrelated, ‘levels’ of analysis simultaneously to develop a genuine understanding of an object by using several senses or other data streams. If that’s not correct I’d be grateful for clarification.

            I think (though of course we can’t be sure at this point) that any intelligence trained merely on text data is very unlikely to remotely resemble a human intelligence – at least, not without a massively inefficient amount of data expended in the effort, and likely carefully curated data at that. It’s likely that multiple types of data streams to work from have a multiplying effect on the intelligence, in the same way that 64-bit computing offers massively more than double gains in certain parameters over 32-bit computing.

            But I also think this misses the main point. Suppose you could add a few more data streams. Add a camera and some movement capacity or whatever, and a few years of training. Is there any other fundamentally different process you would need to add to create something that looks like understanding?* If not, it does really look like we’re on a faster track to true AI than most people appreciate.

            *Not human understanding, of course. The human perspective needs a few other things like hormonally-directed moods and so on.

          • Faza (TCM) says:

            The deeper point is that no arbitrarily expansive set of symbols can be sufficient to impart understanding, unless it is accompanied by a rule-formulation/following mechanism, that may be captured by the phrase: “You’re lying!”

            In my experience, children go through such a phase as part of normal development.

            Imagine, if you will, a set of training data that includes the following statements:
            1. A car runs on gas,
            2. A car runs on electricity,
            3. Gas is not electricity.

            A child sufficiently old to understand these three statements (a couple of years, I would guess) will spot the problem immediately: it has perhaps learned the rule “a car runs on gas” first; when subsequently confronted with a different rule: “a car runs on electricity”, it sees a contradiction that requires resolution – thus, it may guess that “electricity” is just a different word for “gas”. If we then tell it that, no, “gas” and “electricity” are two different things, the contradiction is manifest again and the child will demand an explanation – if it is old enough to understand the concept of lying, it may well accuse us of doing so.

            It is not at all clear if – or how – GPT-2 could arrive at the concept of such contradictions. Lying is an emergent property of the world – where information received begets action – but it is not a property of language as such. A syntactically correct utterance, such as: “all foos are bars” does not have an intrinsic truth value and neither does “no foos are bars”. I’d go as far as saying that a sentence like “All foos are bars and no foos are bars” is not, in itself, contradictory – unless we externally introduce the rules that make it contradictory.

            For any modeller that operates purely on utterances, the foregoing contradictory sentence is no more or less significant than any other utterance it may have encountered. It has no mechanism that would allow it to separate garbage from significant utterances. If we were to “posion” the training data with purely random word-salad, such a modeller will happily incorporate this into its models, giving us the expected GIGO.

            Humans don’t do this, because our empirical existence provides a reality check. The world, as we experience it, is fundamentally predictable, allowing us to formulate essentially binary rules (there’s a reason why humans tend to suck at probabilistic thinking). We learn fairly quickly that there’s no fundamental connection between what we say and what actually is.

          • Dedicating Ruckus says:

            @Forge the Sky

            I don’t think the difference between text and non-text sources of input data has much (fundamentally) to do with the difference between humans who understand and a model like GPT. With a whole lot more compute, (in theory?), you could train a model on the entirety of Youtube and get out something that can produce plausible-looking talking-head videos who speak gibberish; this also wouldn’t have understanding, it would just be “GPT-2 for video”.

            By “N-level modeling” (maybe not the best phrasing) I mean something like the ability to think about “aboutness”/referential relationships, to think about the map and territory as separate entities which nonetheless have an isomorphism. An entity that knows that cars run on gas, and emits the sentence “cars run on gas”, does so because the sentence is representative of relations in the entity’s internal model between the concepts (not the words) “car”, “run on” and “gas”. By contrast, GPT-2 emits that sentence because of an internal model that is concerned with the words (not the concepts) “car”, “run on” and “gas”, and the relations in that model don’t correspond to functional relationships between real-world referents, but statistical relationships between words.

      • CodProfundity says:

        Aren’t those steps just a way for it to associate the words write/written/wrote etc with the word author? I’m not sure it proves an understanding of what any of those words mean.

      • dyfed says:

        All due respect, Scott, you’re assuming that the AI is creating logical relationships between concepts, but it isn’t. It’s creating probabilistic relationships between strings.

        Ask it, “Can you please tell me any name that’s not the name of the ancient Chinese tactician and strategist who authored the Art of War?”

        It will likely answer Sun Tzu, because it doesn’t know ‘negation’ as a principle. It knows that ‘name,’ ‘author,’ ‘Art of War,’ etc., significantly increase the probability that ‘Sun Tzu’ will be near.

        Even very stupid children have a beginner’s grasp of necessity, negation, etc. and understand concepts through that lens. The AI is trained on more information than the child (who cannot even read!) has ever learned and still doesn’t get these concepts, because they’re not inherent to graphemes, only meanings.

        It’s plain enough from its performance at reading comprehension, which is dismal—failure-grade for a child, and exceptionally bad for someone with as massive an ‘education’ as the AI analogously has.

        It is an enormously sophisticated tool, and like others of its kind represents a terrific human achievement. But it doesn’t move a belief that we have made no process on GAI, because the fundamental tools of intelligence are missing.

        • realitychemist says:

          I mean, you’re *assuming* it would fail to properly process the negation. I think you’re probably right, but it’s not a good idea to build your argument on top of a hypothetical piece of evidence which you don’t actually have. Especially considering you don’t even need to, there are instances of its actual output in the paper (and even in the blog post examples) that you could use to make the same point.

          • dyfed says:

            I’m not assuming anything. Even the cherry-picked data provided as an example of the tool’s competence fails miserably to answer simple questions, because the tool simply answers with the highest-probability symbols associated with the symbols in the question.

          • realitychemist says:

            I’m not assuming anything

            You are. Nobody has asked it this question, so you must be assuming what the answer would be. I’m not even saying it’s a bad assumption, but it is an assumption until GPT-2 is asked this question and provides an answer.

            I’m in engineering. We make assumptions all the time, like, “I’ll assume this process is operating at steady state.” That can be a good or a bad assumption, depending on the ground truth of what the process is actually doing, but there’s no point making that assumption if it’s not needed (e.g. if we have real-time data logging on the process).

            Even the cherry-picked data provided as an example of the tool’s competence fails miserably to answer simple questions

            That was the point of my comment. You can make the argument that it’s generating high-probability symbols without any concept of logical relation without making up a hypothetical example. You have ground-truth data you can use, it should override the need to use hypothetical examples.

        • Tarpitz says:

          You’re assuming away Scott’s claim about what the “fundamental tools of intelligence” are, when that’s the central issue at stake. No-one is denying that children – even stupid children – are much, much smarter than the AI. The question is whether what they’re doing under the hood is the same thing but better.

          • dyfed says:

            I agree that is the question. What I’m saying is that we already have the evidence that a child is successful with less education and a worse model, whereas the tool fails utterly with more education and a better model. And this is because the tool has *no* reading comprehension, not bad reading comprehension.

      • Dedicating Ruckus says:

        I think it’s a mistake to say it has such a thing as a “probability that Sun Tzu is the author of Art of War”.

        It has a probability that the phrase “Sun Tzu” appears in conjunction with the phrase “Art of War” and the word “author”. It doesn’t know who Sun Tzu is, what an author is, or what Art of War is (or what a book is, for that matter). Its model is just about text, not about the meanings behind the text.

        A model that’s just about text can do impressive things with text, clearly enough. But it doesn’t have what we’d call “trivial” capabilities of general intelligence. It probably can’t go from “Art of War is Chinese” through to “Sun Tzu is Chinese”, for instance. (If it spits out “Sun Tzu is Chinese”, that’s from seeing those words together on a separate basis.) And it almost certainly can’t get from there to “Sun Tzu probably has black hair”.

        Remember from Eliza that humans are inclined to impute human-type capabilities and processes given the very slightest provocation. This would only be a stronger tendency for a model so relatively impressive as this one. This thing is fundamentally doing a very different thing from what humans are, just like GANs are fundamentally doing a very different thing from the human visual cortex. It’s coming at similar-looking capabilities via a completely separate, and much more limited, mechanism.

    • niohiki says:

      But I think that’s the point. What is a gun permit? It something you need (or not) to use a gun. Sure, we have many more characteristics assigned to it (it’s a piece of paper, it’s given by the government). We learn of its relevance not because it is a piece of paper or because it is given by the government, but because we need it (or, again, not).

      How many people “know” what a hedge fund actually is, and how it actually works, beyond using the word as a placeholder for “evil organization that the capitalists use to take our money”?

      • CodProfundity says:

        How many people “know” what a hedge fund actually is, and how it actually works, beyond using the word as a placeholder for “evil organization that the capitalists use to take our money”?

        Well this is the issue for me really. Because a good tactic against people blaming hedge funds for evil stuff is asking them to explain what hedge funds are and how they end up causing evil things and often that leaves the person making the complaint flailing because they don’t understand what a hedge fund is they just have an autocomplete function running that sees “hedge fund” and links it with negative outcomes. We wouldn’t say that shows the person against hedge funds knows what a hedge funds is so I don’t know why we’d say a machine that’s made some associations between two things knows what they are.

        • Tarpitz says:

          I fear that if you drill down far enough in this way you’ll get back to Bill Goldman: no-one knows anything. We’re all just doing more sophisticated versions of the same thing as the idiot talking about the hedge fund. It’s assertability all the way down.

          • Dedicating Ruckus says:

            That’s nonsense.

            The idiot talking about the hedge fund couldn’t answer the question “what differentiates a hedge fund from an index fund?” Someone who knows what a hedge fund is could.

            If you can describe the workings, purpose and nature of a hedge fund, then you know what a hedge fund is.

  4. sharper13 says:

    I have serious issues with the section on fantasy books and fiction in general. Well-done fiction is a blend of the familiar and the strange. Scott only seems to include the “familiar” part. That’s the part writers put in so readers who buy their books get what they expect, but the other part, the part where readers (outside of the extremely formulaic, like a few romance or literary genres) get something new, original and exciting to them is as much, if not more, important.

    Sure, some writers include more of the familiar in their blend than the strange. Some readers like that, so they have an audience. Others include much more original thought, the parts which are newly created in the writer’s mind. I would challenge you to read, for example, most of Brandon Sanderson’s fantasy books and report on what percentage is based on Tolkien or A Song Of Ice And Fire. You won’t find any elves or dragons, just uniquely understandable systems of magic which are internally consistent, yet unlike anything you’ve read before.

    Scott seems to be reading books from a fantasy sub-genre which is based specifically on extending Tolkien or A Song Of Ice And Fire as the “familiar” portion, then attempting to generalize that out to everything else. Please try some books which aren’t attempting to be that in order to sell copies to existing fans of the sub-genre.

    How a new sub-genre generally springs up is that someone comes up with a well-done work in a “new” style and it’s popular enough that it in turn becomes the “familiar” portion of future books (which will also extend it with something new) because it has enough readers who want more like it. That’s what Tolkien did for quest fantasy and it’s how cyberpunk and steam punk became sub-genres in their own right, but the door for more “new” remains wide open.

    The limitation on what percentage of new and original ideas can go into a story isn’t how much the writer can think up (there are some really weird writers out there, and even the mostly sane ones are probably more imaginative in their internal reality than their readers would guess), it’s how much of a departure from the familiar readers will accept. The better a writer is at making the new portions be easily understood by their readers, the more they can include and still have a book which works for a large target audience. At the extreme end of the strange, you can have a book about truly alien beings (not human stand-ins) which is written completely in a made-up language. That’d have a potential target audience of maybe two, the author who made up the language and their mother who cared enough to learn it.

    Reading the provided “AI” samples struck me as how unoriginal they were. Nothing I saw contained new thought, it was all obvious regurgitation. I was a little worried about the future of writing for a few moments based on the headlines, but after reading the samples, it’s obvious this particular approach will never rise to be competition for a real writer, because the creative spark is missing. The element which adds to the writing the portion which actually is new, is missing.

    So while I agree the “AI” approach is to blend things up and regurgitate them, I disagree that’s what decent writers are doing all the time. That’s the simple portion of the art. The complicated portion is that which is strange being added for the reader. The reader wants to be simultaneously comforted by the familiar emotions evoked by the genre of book they’ve decided to read and already know they like, but also entranced by the sense of wonder as they imagine through the words on the page things they’d never thought of themselves.

    Is there a potential market for 100% regurgitated writing? Sure, especially in news reports or spamming attempts, but not for real fiction, not even if you programmed in various plotting methods to give it a better structure. There are too many people who don’t want to read only another variation of the same stories over and over again. They also want a good dose of the strange with their familiar.

    • MawBTS says:

      Reading the provided “AI” samples struck me as how unoriginal they were. Nothing I saw contained new thought, it was all obvious regurgitation. I was a little worried about the future of writing for a few moments based on the headlines, but after reading the samples, it’s obvious this particular approach will never rise to be competition for a real writer, because the creative spark is missing. The element which adds to the writing the portion which actually is new, is missing.

      You use the word “obvious” a lot for things that actually aren’t obvious, at least not from my perspective.

      Remember that the AI wasn’t writing freely, it was responding to a prompt! “Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry.” I think that most human authors would produce an equally derivative work, given such a prompt.

      And even so, it (accidentally) comes up with a few interesting misuses of English, like the orcs’ claws being “deafening” (??).

      • eric23 says:

        Word salad is not meaning. It’s a Rorschach test, or the Virgin Mary in a piece of toast. It may look interesting – but if it’s consistently ambiguous and you have to speculate on what it means, then it’s you supplying the meaning, not it.

        • Tarpitz says:

          As distinct from other speech acts…?

          • Dedicating Ruckus says:

            Is the claim here that all speech acts are inherently meaningless, and the listener always supplies whatever meaning they impute any utterance?

            If so, I have to ask, why are you here engaging in meaningless speech acts?

      • Ozy Frantz says:

        I will bet you up to $500 at 100:1 odds that if I gave ten of my writer friends the prompt “Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry” and asked them to write a paragraph based on it, all ten humans would produce a less derivative paragraph than the AI, as judged by a blinded fantasy reader. (We can edit the Legolas and Gimli passage for coherency.)

        • MawBTS says:

          Declined, since your writer friends probably aren’t typical.

          For a random n=1 sample of a human writing Lord of the Rings fanfiction, here are the first few paragraphs of the first story under’s LOTR tag.

          The Battle for the Black Gate was in full flow. Aragorn, son of Arathorn was fighting his hardest, Andúril clashing and striking against the poisoned, serrated blades of the countless orcs in front of him. Despite all the Valour of the Captains of the West, this was a hopeless battle, and he knew it.

          He fought not to win, but to survive. Survive long enough to fight on, giving time to Frodo, to destroy the ring.

          Hack. Cut. Stab. Slash. Parry. Slash. Parry- parry- swing- decapitate.

          He conserved his energy as he fought, trying to guarantee kills with minimum effort. He could not afford to feel fatigue- yet there it was, creeping over his heart…

          No. Aragorn was stronger than this.

          “ELENDIL!” came the cry, and he leapt forward, swinging Andúril in a wide arc, killing six in one swing. With equal agility, he reverse-gripped his blade, stabbed the orc behind him, and retreated before being encircled.

          Unbeknownst to the heir to the throne, something stirred. The shadows gathered. There was a momentary blackness behind him, undetectable except by a slight chill in the air. This was no servant of Sauron.

          Clang. Clang. Crash. Ssssss. Clatter. Crack. Slice. Kkkkhhhhttt!

          Andúril struck against the vicious dark blades of the orcs, blunting their serrated edges, and in some cases, cutting them clean off.

          The nameless shadow watched with interest. The future king surely fought well- a challenge not to be underestimated. He was here, however, to observe, and to watch, to make observations. No actions on his part were required now. He turned, therefore, to the Elven Archer in front of him.

          Etc. It’s a better written but not any more original (unless you count interior monologues and comic book sound affects, but that seems like a stylistic choice).

          • Dedicating Ruckus says:

            It’s a mistake to think of GPT-2’s output as being “derivative” in the same sense as a random LotR fanfic at all. “Derivative” or “original” are not qualities the prose has.

            It is not about anything, and therefore it cannot have the quality of its subject being similar to (derivative) or distinct from (original) any given other referent. Calling model-generated texts “derivative” is a category error; they cannot even achieve derivativeness. (Derivation?)

          • Ozy Frantz says:

            On the other hand, the first scene of the most recently updated LOTR fic on AO3 is:

            The First World is destroyed. The Universe is created by the Emperor-beyond-the-Sea, with his son Aslan’s help. So many lives were at stake. So many were lost. The time Before the First Age and the First Age had seen so much peril, so much hurt and sorrow, so much pride was shown that the price was paid instead.

            The Shadow Empire rose and fell, as Qua’ra Holdo, now called Scar-ra, predicted. Alindor, the Golden House Cat who was a Druid and a Shapeshifter, too, reigned in the Land of Talking Animals, before his untimely death towards the end of the First Age, leaving his title passed on throughout the next generations.

            But of those who suffered the most, the dragonoid Parker Dooley proved this to be true. As Maranguan, Parker formed the Shadow Empire, even the idea for the Shadow Squad was his own. As he became whole and good again, Parker lost his dear, darling human wife, Eliza Bowler-Dooley, to a Force Power brawl against the Sith Emperor Vitiate, upon which Eliza’s life was lost.

            Parker was devastated. But then, he found closure after looking up how Vitiate died by the Hero of Tython, a Jedi Knight, with whom he met later on. It gave Parker some needed closure, after everything he endured.

            After the time Before the First Age ended, Parker visited the Chesapeake Manor in the First Age with his great-grandson, his lookalike in his human-hobbit form, Terrence Dooley. If only for a short time, for now we must return to the Chesapeake Manor during the Second Age. And that is where our story begins.

            I fully agree that is mostly inhabited by eleven-year-olds and eleven-year-olds do not write very good fanfiction.

      • sharper13 says:

        In the interests of clarity, when using “obvious” in that paragraph, I was implying “obvious to me”, in the sense of easily understood when seen. I don’t expect the same observations to necessarily be obvious to another observer without a similar background in analyzing how to write fiction.

        If it were useful, I could elaborate on some of the specifics which led to the conclusions, but it’s more of a piece-level conclusion, as opposed to a sentence-level conclusion, which makes it more difficult to quote examples.

  5. palimpsest says:

    AlphaGo has two capacities that combine to make it superhuman, the neural nets, (evaluation and policy networks) which seem closely parallel to a human’s intuitive sense of a position, and the ability to do look-ahead search, corresponding to a humans conscious calculation “if I move here, they move there..”. Though I can’t find a reference, I recall reading, or perhaps hearing during the AlphaGo broadcasts, that AlphaGo’s policy network (that is, just pattern-matching for the best next move, no look ahead search) was in the amateur 4-6 dan range. This is probably about as good to significantly better than a human could ever play with absolutely no calculation (just playing the first move that comes to mind with no thinking on the opponent’s turn). And that was the earlier version. Adding the ability to look ahead made it better than human of course.

    I guess that GPT-2 is like Alpha Go playing with just the policy network, in that it is better than humans in the capacity that it shares with them, i.e. it’s better than any human could write just using a pattern matching, stream of consciousness approach (and better than a human could write while dreaming, were this possible). But I would expect the first human or superhuman AI to add some new element, rather than just being a better implementation of this basic strategy.

    • FeepingCreature says:

      Indeed, which is why it will never be able to do unrestricted arithmetic: that’s the sort of challenge that requires decomposition and recursion, whereas this is inherently single-pass; it can notice patterns, but it can’t do patterns of patterns.

      “Still” one or two conceptual breakthroughs away from the singularity.

    • gmaxwell says:

      I guess that GPT-2 is like Alpha Go playing with just the policy network,

      Most of their reported outputs are greedy– always taking the most likely next symbol without lookahead.

      Some of their reported outputs used a limited amount of lookahead, none used a particularly extensive lookahead procedure of a sort that is already common in the literature for less sophisticated language models which need them in order to work at all.

      Go was historically considered to be a difficult game for computers because the lookahead space grows very fast. The character-pair based NLP space grows similar to or faster than go does (both have hundreds of options per ‘turn’ and text output will have many more turns than a typical go game).

    • Reasoner says:

      Deep learning is a big field. I’d be wary of trying too hard to find deep similarities between the two projects that happen to have gotten a lot of press in recent years.

  6. cheerup_on_paincakes says:

    So, what is the bottleneck here? Is it just training data, training time or computational resources? Given OpenAI has X amount of they key resources, whether they are computation or training data, is it reasonable today to give it say 100 time as many resources? What about 10000x?

    And does anyone have any idea of how much more data would serve a “noticable improvement”? Would 10x as many resources be meaningfully different? What does the curve of improvement/resources look like, does it hit diminishing returns or is it linear, exponential? Can we even tell and by what metric?

    I have so many questions.

    • 6jfvkd8lu7cc says:

      (Disclaimer: I am not working on anything in Machine Learning)

      I think training time is the same as computational resources on their scale and with the modern training strategies. And I think I remember reading them saying that for reinforcement learning of playing a computer game they tried fancy improvements to algorithms, and doubling the computational resources. Brute-force was significantly more efficient. Which is scary from the point of view of capital expenses beating everything else…

      For this task, of course, a lot of training data is also needed. Hard to say how much — I would not be too surprised by either «reinforcement learning of compressing the most popular gigabyte of the Wikipedia (article text) content gives as a byproduct a strong model of formal writing» or «reinforcement learning of compressing the most popular gigabyte of the Wikipedia can reach record performance while overfitting on properties we don’t care about».

      I guess the substantial variety of data matters, and this is hard to quantify.

    • gwern says:

      It cost ~$43k to train we guesstimate (256 TPU3s for a week), and probably hadn’t converged either. It would be totally reasonable to throw 10x or 100x more computer, parameter, or data at it. Data & parameters are trivial, of course, just change some settings, and compute, well, ‘100x’ sounds like a lot, but it’s still less than OA has already spent training OA5. (I’m always shocked just how cheap machine learning is compared to, say, clinical trials.)

    • realitychemist says:

      There are some graphs in their blog post showing its performance at differing levels of parameters. Which is not exactly what you asked about, but by looking at them you can see that some performance metrics are already approaching zero marginal return at 1.5B parameters (e.g. summarize a text) while others still seem to be netting big marginal gains with added parameters (e.g. answer questions about a text, translate French to English).

      I imagine different metrics will likewise show different levels of responsiveness to increasing amount of training data or computational resources.

      • gwern says:

        I would note that as models start getting really good, the metrics start being less impressive. For example, in the very large-scale CNN experiments like Google or FB, your standard ‘classification error’ starts asymptoting at a few % but this is a misleading way to look at it because (a) label error means that going beyond a few % is in fact a terrible thing & sign of overfitting and (b) transfer learning continues to improve even as classification continues to stagnate (I see this as a version of Hinton’s “dark knowledge”). In some cases, like ROUGE or BLEU, the metric itself is questionable to begin with. (For another example, consider this metric: “% of AlphaGo predicted moves that agree with Go players’ next move’ on sampled board positions”. It goes up during training—then down.)

        So in the case of a language model not actually constructed for zero-shot learning in the first place, it seems entirely possible that it could ‘stagnate’ on untrained tasks or tasks with bad metrics even as it gets better and better when done as a few-shot/finetuning. After all, it’s pretty ridiculous that you can get it to do text summarization just by tossing in “TLDR” as a keyword. There’s no reason to think that this is in any way an optimal way of generating text summaries or that tossing in ‘TLDR’ should just continue to scale… If one took it and did some additional training to create a reliable way of generating text summaries rather than ad hoc exploitation of a nigh-accidental capability of the raw model, and that showed no improvement, that would be more convincing.

        So for ‘summarizing a text’, before I’d strongly conclude “yeah, it’s approaching zero marginal return for additional parameters/training, needs architectural improvements”, I’d want to know more about what upper bound the corpus/metric has and how the increasingly-large models perform given some direct finetuning on the task which would expose its hidden ‘dark knowledge’.

    • vV_Vv says:

      So, what is the bottleneck here? Is it just training data, training time or computational resources?

      Compute. The paper mentions that even their largest model underfits the data.

      And does anyone have any idea of how much more data would serve a “noticable improvement”? Would 10x as many resources be meaningfully different? What does the curve of improvement/resources look like, does it hit diminishing returns or is it linear, exponential?

      The rule of thumb for neural networks is that improvements are logarithmic w.r.t. increase of training data size and model size (which is limited by computational resources).

  7. RC-cola-and-a-moon-pie says:

    It does seem pretty important to me that your mom grounds her written work product in an understanding of reality outside the texts. I worry that the discussion of creativity and similar concepts obscures this as the more basic distinction. I may write a trite or unoriginal text, but if I do so I’m starting with a knowledge of the objects, etc., being discussed and how they interact and fit together in a realty outside the text. Isn’t that an extremely important distinction from a word-slurry generates from texts that may as well be arbitrary symbols? It may be a big part of what allows for the basic consistency and whatnot that seem so obviously absent from the algorithm’s samples.

    • Faza (TCM) says:


      I’m finding the reactions posted by Scott and others a lot more interesting than the actual texts produced, because they offer yet more evidence that humans will manage to find some warped kind of sense in anything.

    • melolontha says:


      I’m guessing that Scott would dismiss objections based on the concept of consciousness, or the distinction between ‘actually knowing’ versus ‘mindlessly producing correct answers’, as some combination of meaningless/impossible to answer and therefore effectively meaningless/misleading/based on false implicit premises. But I (genuinely, not just rhetorically) don’t understand how he can avoid discussing the distinction between free-floating language and language that is grounded in (concrete, physical) reality — or at least connected to reality via an entity who combines the ability to use language with some kind of direct (or ‘direct’) access to the world.

      Maybe the idea is that this is a relatively trivial next step? Chuck some sensors onto a robot running this kind of code, train it appropriately, and it will learn to associate its linguistic concepts with things/actions/states of the outside world?

      Even though that’s extremely handwavy, when I put it like that I realise that my real objection is probably the harder-to-articulate one about actually having mental states that correspond to the concepts referred to by the words.

      Still, I wonder if I would better understand Scott’s perspective if he elaborated on the human-brain-as-model-builder thing, and why he sees it as a satisfactory response to the question about “deep understanding”.

    • _chris_sutton says:

      I’m not so sure.

      Let’s pretend you are discussing something in the comments with two other commenters. Everyone seems intelligent, all can keep the plot, you go back and forth for hours on all sorts of digressions. All three have solid models of the world.

      So you’re typing these comments with your dexterous fingers, can feel your ass in the seat, can see the screen. Another is this AI, who has never had any sense data like yours, can never directly “touch” the real world, and has built this model solely on the symbols.

      But the third is another human, one who had a terrible accident that rendered them in a coma. They have a brain connection apparatus that allows them to interface, via text/internal voice/thought, with any number of sites. They no longer have other sense data coming in, no longer can touch the real world. But they did originally build their model as you did, rather than with pure symbols.

      The process of how this person and the AI built their model is different in this case, but nothing else is. And I don’t see how we can say the process used to build the model really matters if the model itself is the same.

      Now to the extent the AIs inability to keep the plot is a result of missing this bridge to the real world, then yes obviously it matters. But I don’t see a good reason to favor that explanation over it simply being underpowered, and if it’s just underpowered I don’t see a good reason this bridge is meaningful with regards to the intelligence.

      • RC-cola-and-a-moon-pie says:

        Interesting point, but it seems to me that if you start by building in the premise that the AI has achieved Turing-like fluency then it sort of begs the question. I guess I would characterize my suspicion (not really an argument) as being that when you are presented with a far-from-fluent text generator and see that it uses a texts-only knowledge base, that is going to present major hurdles in building up the further levels of coherence and fluency that are required (in light of the fact that it is untethered to a stable external reality that imposes coherence on it from the outside) and that achieving the kind of situation described in your hypothetical would sort of be the point in dispute.

        • _chris_sutton says:

          Ah yea ok. I misundertood you to mean the connection was fundamental somehow in its own right, as opposed to a necessary step in building an accurate model.

      • Faza (TCM) says:


        Now to the extent the AIs inability to keep the plot is a result of missing this bridge to the real world, then yes obviously it matters. But I don’t see a good reason to favor that explanation over it simply being underpowered, and if it’s just underpowered I don’t see a good reason this bridge is meaningful with regards to the intelligence.

        The ability to keep the plot is what it’s all about, though, innit?

        I see two issues here and I’m not convinced that GPT-2 can meaningfully engage with either, even if we used more power:
        1. Rule-building/rule-keeping – the ability to define and/or identify immutable principles by which the (language) game is played. This goes beyond syntactic qualities that make for an understandable sentence in our language of choice and also includes semantic rules (if Gimli’s a dwarf, he’s always a dwarf; if he’s a dwarf, he’s not an elf, etc.)

        A crippled human such as you describe would presumably be capable of this, if my understanding of the situation you propose is correct.

        2. Mapping the “game” to the outside world – the real sticking point.

        Computers can do a pretty good job of playing games successfully. This shouldn’t surprise us – any rules-complete game is reducible to a state machine. However, we don’t usually apply our intelligence to playing abstract games – instead, we use our rule-building skills to create models that may or may not have predictive power in the external world and evaluate their usefulness by comparing predictions with what actually happens.

        “Keeping the plot” is a matter of rule observance, in the first instance, but it is subject to external verification (the coherence of the message is evaluated by an external receiver). Given that all current examples require (very) charitable readings, the preponderance of evidence points towards GPT-2 not being much more than a step up from Markov generators.

        • Creutzer says:

          if Gimli’s a dwarf, he’s always a dwarf

          That’s actually not a matter of language, but a matter of the world. We have no problem talking about people being polymorphed by a spell and becoming a different species.

          • Faza (TCM) says:

            True, but notice that when we do we introduce a contradiction-resolving mechanism.

            “Gimli was a dwarf who got polymorphed into a dragon” is seen as a semantically different situation than “Gimli is and always was a dragon” (cf. Goodman’s “grue”). We have no trouble simultaneously keeping the concepts “Gimli has the essence of dwarfness” and “Gimli ticks all the boxes of ‘be a dragon'” in our heads – for example, we may speak of the steps Gimli needs to go through to regain his “real” form – that of a dwarf.

            It is a matter of language insofar as neither Gimli, nor dwarves, nor dragons actually exist. However, the way we use language allows us to construct a model world where statements like “Gimli is a dwarf” or “Gimli was polymorphed into a dragon” carry semantic weight. This semantic weight arises from the impossibility (assumed) of transitions between statements like “Gimli is a dwarf” and “Gimli is a dragon” without some kind of intermediate statement that resolves contradictions (“dwarves are dragons”, “Gimli was polymorphed”, etc.)

            It’s worth emphasizing that these “forbidden transitions” are a purely linguistic construct, in the sense of being “forbidden” by definition only – because we’re talking about a fantasy world.

    • b_jonas says:

      Teachers force middle school students to write essays about civil wars. Those teachers haven’t experienced war or slavery as “an understanding of reality outside the texts”. They haven’t lost their friends in a war, they don’t even remember parts of their town that no longer exist becuase they were bombed int he war. They are just reproducing word-slurry with phrases they have found in the patriotic texts they were required to read.

      • acymetric says:

        Those teachers haven’t experienced war

        They haven’t lost their friends in a war

        I know you were probably just trying to be glib, but I’m going to have to [citation needed] this. I would guess there is a non-negligable portion of teachers for whom one or both statements is untrue.

        Even in cases where both are true, there are almost certainly forms of exposure to experience/information that contribute to understanding of the subject matter beyond simply reading written text even when the subject matter wasn’t experienced directly, so I’m not sure you statement holds regardless.

  8. Bugmaster says:

    It will probably need other things besides. Maybe it will take a hundred or a thousand years to manage all this, I don’t know. But this should be a wake-up call to people who think AGI is impossible, or totally unrelated to current work, or couldn’t happen by accident.

    Is there a lot of practical (as opposed to pedantic) difference between “maybe a thousand years from now” and “impossible” ?

    Don’t get me wrong, GPT-2 is very impressive; but to suggest that it could solve P=NP (or even derive Newtonian Mechanics without looking it up on the Internet) is basically ridiculous. It’s easy to anthropomorphize GPT-2 when it’s generating English text; we humans have hyper-developed pattern recognition faculties, and thus we can readily emphasize with anything that sounds even remotely like us — be it neural nets, ELIZA, or even random scratches in the sand or noises in the night.

    Math and physics, however, are not evaluated based on their appeal to other humans; you can’t just use your hyper-developed empathic pattern recognition skills to fill in the gaps. Mathematical proofs have to be consistent in the extreme; physical theories have to accurately predict the real world. Saying, “oh, well, that’s just a matter of procuring a bit more training data” is like saying “I got really good at jogging in the last two months, so I think next year I can jog all the way to the Moon”.

    • Tarpitz says:

      How do you think humans learn to do maths, if not by pattern recognition?

      • Dedicating Ruckus says:

        By an understanding of the relations of mathematical symbols to the underlying reality of mathematics, which creates the necessary regularities. This model, and anything similar, cannot have any such understanding.

        Trivial example: you could feed it a few pages of times tables, and it will learn to say that 7 x 5 = 35. However, it will not have any idea that this has any relation to the area of a rectangle with sides of lengths 7 and 5.

        • acymetric says:

          To be fair, that is true of a kid who you start feeding multiplication tables to them (well, maybe that depends on the education style, I would suggest it is true for the more “traditional” approach, at least in the US). If you taught it that area of a rectangle is L x H, and taught it that 7 x 5 = 35, you might be able to get it to tell you that a rectangle of L = 7 and H = 5 is 35. My guess is that it would not be able to tell you the area of a rectangle where L = 8 and H = 10 unless that multiplication was explicitly in the training set. I’m not even confident it would be able to do 5 x 7 if it knows 7 x 5 without some additional training on rules of math.

          What is probably a better point is that as you start learning and expanding multiplcation tables, you get to the point where you can figure out new columns without being explicitly told (possibly through some very clumsy mental work or the use of some scratch paper).

          So even if I was only taught multiplication tables up to say, 13, I can probably figure out what the table should look like expanded to 14 and beyond. I don’t think this algorithm would be able to do that.

          • Dedicating Ruckus says:

            I think a lot of the people comparing this model favorably to a kid are basically right. That is, humans clearly have two modes of communication/symbol-manipulation, one in which they actually know what they’re talking about and one in which they’re just blindly remixing what they’ve already seen/guessing the teacher’s password; the latter facility grows in a lot sooner than the former, developmentally; and this model is probably doing something comparable to that latter facility on a fundamental level.

            I just don’t think this is equivalent to the speech of humans who know what they’re talking about, or a step along the path to that ability, or indeed anything to do with that ability at all.

        • Protagoras says:

          What is the underlying reality of mathematics? How do we know anything about it and how it is related to the symbols?

          • Dedicating Ruckus says:

            The underlying reality of mathematics are facts about the world. They’re of the same kind, and no less objective, as straightforward things like “stop signs are red”; their referents are, however, immaterial.

            We know about this reality because it conditions many realities that are material and thus directly observable; e.g. we know 1+1=2 because, among (many) other things, if you put a rock down next to another rock there are then two rocks there. This indirection via the material isn’t the only way to know about math, but it’s certainly the most immediate, reliable and straightforward.

            The symbols we use are related to the underlying mathematical realities because they are about them, just like the word “snow” is about the cold white substance.

          • Protagoras says:

            What are the other ways? And how do we, for example, know about transfinite cardinals and their properties?

          • Faza (TCM) says:

            What are the other ways? And how do we, for example, know about transfinite cardinals and their properties?

            By seeing where rule-keeping takes us.

            “Advanced” mathematics is a game, where we extrapolate from first principles and known results in an attempt to answer interesting questions (for a given value of interesting). There are strict criteria for acceptence of an answer (proof) – it cannot conflict with any other established part of mathematics, unless it is either shown that the established part is wrong or the contradiction is otherwise dissolved. Mathematics must at all times be consistent as a whole.

            The important feature of axioms is that they aren’t themselves arbitrary (they could be and still produce an internally consistent system, but it wouldn’t be mathematics, as we understand it) – this is what Dedicating Ruckus is pointing out.

            We can test the consistency of mathematics – and its connection to reality – by performing mathematically equivalent operations and seeing if they give the same result. Importantly, understanding mathematics involves being capable of producing proofs of results by means other than the results were originally arrived at.

            An example:
            By looking at a multiplication table, we might find that 5×5 is 25 – that’s the kind of “knowledge” that symbolic analysis gets you. However, you don’t really understand what it means unless you can produce an independent proof (i.e. one that does not use the multiplication table you started from) of this result. What kind of naive proof could we offer?

            Take five sticks, lay them out next to one another. Take five more, and lay them out in a line below the first five. Repeat until you have five lines. Count all the sticks in order and you will find that there are 25 of them.

            It is not at all clear if it is even possible to arrive at the notion of counting (as set-matching) purely from reading people talk about counting. It is intuitively easy to grasp in the physical world, however, because all it takes is laying out things next to one another.

      • Jaskologist says:

        This is one of Gwern’s open questions:

        What, algorithmicly, are mathematicians doing when they do math which explains how their proofs can usually be wrong but their results usually right?

    • gmaxwell says:

      > Don’t get me wrong, GPT-2 is very impressive; but to suggest that it could solve P=NP

      Many interesting modern mathematical discoveries have been made by computers doing more or less brute-force searches.

      E.g. the digit extractor for pi in hex was found by an integer relation algorithm (then refined by humans).

      GPT-2 itself by virtue of how its structured certainly doesn’t seem more likely to make mathematical discoveries than specialized tools… but I would also be surprised that if taking GPT-2 fine tuned on existing proofs and used it as a hinter for a proof assistant didn’t actually succeeded in occasionally finding real proofs of non-trivial things, and do so more effectively than a totally naive search and without any particular specialization of the software beyond strapping it to a proof assistant. You might protest that strapping it to a proof assistant is cheating, but plenty of humans aren’t going to reliably come up with new formal proofs unless strapped to a proof assistant either. 🙂

  9. Gurkenglas says:

    It will have to have a model of what it means to act in the world, to do something besides sitting around predicting all day.

    I don’t think we want to invoke quite so many reasons to fear the alignment problem. Just make the language model smarter at predicting until it can generate half of a passing Turing test transcript, then use that chat interface to have a bunch of unreal FAI researchers figure out the next step.

    (Also figure out inner optimizers first and maybe use the HCH protocol.)

  10. 10240 says:

    The method of predicting just the next word, word-by-word, is limited, and it’s surprising that it can generate at least somewhat natural-sounding text. If you trained it on a text in which every word is ‘foo’ with 80% probability and ‘bar’ with 20% probability, independently, then every time it would predict ‘foo’, generating text (‘foo foo foo…’) very different from the original.

    I thought about this at the sentence “So let’s try again.” in the civil war text. If, say, one in five sentences is a filler sentence like that in real-world text, I’d expect this AI to never generate filler sentences (or only after a very long text without such sentences) if it went sentence-by-sentence.

    • 6jfvkd8lu7cc says:

      It predicts the next [element], but keeps track of a lot of previous one.

      If I know there is 20% filler, and uniform distribution of filler is customary, and I look at a text with proper amount of filler, then 8 non-filler sentences, I might suspect filler is coming.

    • Gurkenglas says:

      It would choose ‘bar’ 20% of the time. (It has an optional parameter that throws out all the top few candidates before that choice.)

      • 10240 says:

        @6jfvkd8lu7cc @Gurkenglas I thought it chose the word it thought had the highest probability. If it does the way you say (predicting the probabilities of different words and choosing accordingly), it indeed won’t have the problem I wrote about.

        • 6jfvkd8lu7cc says:

          I did mean a stronger claim — if something should be uniformly distributed and hasn’t been seen for a while, it can become locally most probable continuation, given the context.

          • 10240 says:

            In natural text you may be right that the probability that the next sequence is a filler gradually increases if there hasn’t been one for a while, but the point where there is more than 50% probability that the next sequence is a filler will come after a lot more than 4 non-filler sentences.

            (This assumes that sentences are just categorized as filler sentence vs on-topic sentence. If we assume e.g. that filler sentences are one category, while on-topic sentences belong to many small categories, then the opposite problem may happen, that a filler sentence is almost always more likely than any particular category of on-topic sentence, except perhaps if the last several sentences were fillers, so filler sentences will be a lot more common than in real-world texts.)

            In the case where the probabilities are really independent (as in my foo/bar example), your claim would be the gambler’s fallacy. Again, you are right that in real-world situations the probabilities of each word/sentence are not independent. However, it would be surprising if the AI couldn’t even emulate a simple random word sequence, and that didn’t cause some observable limitations in real-world text.

    • vaniver says:

      This is not correct (generative models sample according to the distribution, rather than always picking the mode) and also not relevant to this model (which has a complicated internal state as it goes through a passage).

  11. 6jfvkd8lu7cc says:

    1. If only the currently strongest economics forces were compatible with intelligence amplification and not targeting mostly capital-expense-only artificial intelligence, that would probably be very quickly converted into a very useful associative-memory engine for recalling «that handful of articles about something I have definitely read». Although maybe it will be done as an opensource project anyway…

    2. The scale it needs for learning seems encouraging from the runaway-AGI point of view, though. Maybe a near-human AI could be efficient to use, but expensive (a ton of computing time) to improve qualitatively, so people (and other AI projects) would have time to learn to interact with it.

    2a. Of course, the people who found a way to demonstrate Meltdown will learn the limitations of each new model faster than the people who brought us «IoT where S stands for security». Our only hope is to get a costly (in tangible liquid assets, and to powerful enough corporations) breakdown before a breakdown with numerous deaths.

    3. As for training for maths, a nice part that there we already have a lot of good ways to grade progress… Unlike English where unsupervised learning is needed and quantifying results is difficult and we see many opinions on the same results, RL-like learning «how to make an existing advanced automated or interactive theorem prover complete my bits and pieces of proof» has clear ways to compare against non-ML-based approaches and to quantify progress, and a _relatively_ weak improvement there could be useful (if not for proving new interesting theorems, then for proving theorems about the terabytes of of boring code). And unlike many language-based tasks, the subtle mistakes are not expensive to notice in that approach.

  12. QW says:

    Come on, read a fantasy book. It’s either a Tolkien clone, or it’s A Song Of Ice And Fire.

    I think Liveship Traders is neither a Tolkien clone nor A Song Of Ice And Fire.

  13. Enkidum says:

    Two books that are required reading if you’re interested in understanding the relationship between this kind of AI and top-level human performance:

    1) Margaret Boden’s The Creative Mind, which contains a lovely summary of the creation of Coleridge’s Rime of the Ancient Mariner (if I remember correctly, this is largely paraphrased from two earlier books, which might be better to read directly). Because Coleridge kept a diary that included every book he read, one can search through those for antecedents to phrases and concepts in his own work, and it turns out that an incredible amount of the poem can be sourced to stuff he’d read in the years prior to its composition. This isn’t because he was a plagiarist, it’s because this is just the way the mind works, even at supposedly high levels of artistic fancy.

    2) Nancy J Nersessian’s Creating Scientific Concepts, which includes a lengthy section on Maxwell’s development of the field equations for electromagnetism. Going back to the letters he wrote during this period, she describes how he progressed through a series of conceptual models that were composed of elements from his previous experience combined in slightly new ways (in particular, the operations of various industrial machines). So one of the most dazzling advances in scientific history can be seen, to a very large extent, as a series of re-workings of previously-known information and concepts.

    This isn’t to say that GPT-2-style “thought” is all you need to be a Coleridge or a Maxwell. But it might account for a hell of a lot more of their abilities than you think.

  14. QW says:

    On the WMT-14 English-French test set, GPT-2 gets 5 BLEU, which is slightly worse than a word-by-word substitution with a bilingual lexicon inferred in previous work on unsupervised word translation (Conneau et al., 2017b). On the WMT-14 French-English test set, GPT-2 is able to leverage its very strong English language model to perform significantly better, achieving 11.5 BLEU.


  15. Freddie deBoer says:

    The program is unlike your hypothetical plagiarist because it has no concepts beyond the most abstract sense of statistical proximity relationships. Can it write a logical proof that is not generated by simply trying to make something that looks like every other logical proof it’s read and instead by grasping concepts in a sense independent from their proximal relations in a given training set?

    Any give year old child can create a sentence that is utterly unlike any it has ever heard before. This is the central datum of the poverty of the stimulus argument. And by their very nature AIs that are generated by using training sets are incapable of that kind of creativity. I just don’t see how getting more and more sophisticated Chinese rooms breaches the problem of getting beyond “these terms have this relative relationship together in a training set, so let’s generated new text that is derived from those preexisting relationships.”

    • Freddie deBoer says:

      Do I doubt that we could create some scary Ultron character that would approximate intelligence sufficiently to do damage? I don’t. But it’s the “human-like” aspect, the potential for creativity, and the relationship between intelligence and consciousness that has me stuck.

      • Enkidum says:

        For the creativity part, I’d recommend the Boden and Nersessian books I suggested above (The Creative Mind and Creating Scientific Concepts, respectively). Also Douglas Hofstadter’s Fluid Concepts and Creative Analogies, which I’ve always thought was the best thing he ever wrote, as it’s a bunch of papers in which he (and his grad students) are actually trying to implement his ideas practically in computer programs. (Sadly it’s also the last time he ever attempted to be anything like a normal professor, and so far as I know that entire line of research got abandoned in favour of learning Russian to translate Eugene Onigen because reasons.)

        Based on having read those and other books, I’d say creativity always involves a kind of shuffling and recombination of previously-encountered data, a la GPT-2. There’s more to it than that, in particular some kind of model or rules for the domain of interest, and a preference for the created artifact to have stronger internal connections in this model (this is one of Hofstadter’s main points).

        I don’t claim that that exhaustively explains what creativity is, but I think it gets a hell of a lot further to doing so than skeptics might initially admit.

        As for consciousness, I’ll have to do some hand-waving and talk about the need for interactions with the environment and goal-seeking behaviours. I’m not particularly satisfied with most of the accounts of consciousness I’ve read, but I find most of the supposed arguments against AI or materialist consciousness (e.g. Searle) to be just awful. Graziano’s Consciousness and the Social Brain is an interesting take that I think has a lot to recommend it, although it explicitly ignores the “hard problem”, which may actually be one of its strong points.

      • Le Maistre Chat says:

        Do I doubt that we could create some scary Ultron character that would approximate intelligence sufficiently to do damage? I don’t. But it’s the “human-like” aspect, the potential for creativity, and the relationship between intelligence and consciousness that has me stuck.

        I’m with you. Could a future refinement of AlphaStarcraft or whatever its name is be smart enough to do serious damage if allowed to control 200 military drones? I don’t dispute that. But I don’t see the evidence that “Ultron” would be creative or conscious, or able to do Singularity stuff like invent a technology that defies human-known physics like a vibranium-powered anti-gravity city.

    • Enkidum says:

      Like a great deal of Chomsky’s linguistics, the poverty of the stimulus argument isn’t well-supported by data. He never actually did the obvious, which is to look at a large fraction of every phrase heard and spoken by a given child over a multi-year period. There are two reasons for this: (1) Chomsky has always been allergic to data, and (2) it’s an incredibly difficult and time-consuming task.

      Here I have to admit I can’t recall who actually did this, but I’ve seen the results presented a couple of times. Even getting up to age 5 or so, you end up with something like <1% of phrases uttered by the child being anything more creative than slight recombinations of previously-heard phrases (e.g. substituting the subject of a verb, etc). So "Any [f]ive year old child can create a sentence that is utterly unlike any it has ever heard before" is almost certainly just false. What they're doing at that age is a lot more like GPT-2 than you might be comfortable admitting.

      That isn't to say that there's nothing more to mid-childhood linguistic production than statistical recombinations, and clearly an older human has little trouble producing truly novel sentences. But it certainly appears that a lot of the basic machinery that we use to speak is very much the kind of thing that GPT-2 does.

      • Freddie deBoer says:

        It’s a fair point!

      • Jaskologist says:

        Even getting up to age 5 or so, you end up with something like <1% of phrases uttered by the child being anything more creative than slight recombinations of previously-heard phrases (e.g. substituting the subject of a verb, etc). So "Any [f]ive year old child can create a sentence that is utterly unlike any it has ever heard before" is almost certainly just false.

        I don’t buy it. Kids are pretty well fluent by 3-4. I’ll freely grant that 40% of that fluency is spent asking for food and another 59% is spent singing Disney songs over and over, but the remaining 1% really is impressive and they fill it with all sorts of novel questions.

        • Enkidum says:

          the remaining 1% really is impressive and they fill it with all sorts of novel questions

          Are you sure? What kind of novel questions do children actually ask? What percentage of these are actually novel grammatical constructions which don’t simply consist of changing a few placeholders in a common sentence type? Hell, what percentage of sentences do you think you or I have uttered in our entire lives that are not simply placeholder-shifting? I’d guess it’s very, very low.

          I can’t find the reference to the corpus I’ve seen presented where these proportions were actually measured for a single child, but I know the answer is that truly novel sentences are astonishingly rare for children to produce. I do recall the person who presented it was Tomasello, and he outlines his argument against Chomskian grammar here. I saw him present and pretty much overnight was converted from a believer in the Chomsky/Pinker line to something involving much more learning.

          Admittedly, it is still today a controversial view (though I think the Chomsky/Pinker line is even more of a minority view among the younger generations of linguists). But it’s worth taking seriously, and I think fits much better with what we understand about the brain as a learning system.

          • acymetric says:

            I mean…if we’re going by your strict definition I’m not sure anyone has said or done anything novel with text or spoken word in a thousand years or so, but while that would probably make for a lengthy, entertaining discussion on one of the open threads I’m not sure it is useful for examining anything else.

          • Enkidum says:

            What’s a loose definition then? What counts as a novel enough sentence that we think it’s the kind of thing that GPT-2 doesn’t produce but young children do?

            Not trying to be difficult here, I’m genuinely unaware of what the criteria might be.

          • Enkidum says:

            Perhaps more cogently: I think there are legitimate reasons to doubt that GPT-2 marks a huge step towards true AI (I’m on the fence, personally). I don’t think that “humans produce truly novel sentences while this only does a simulacrum of that” is one of these legitimate reasons.

          • Jaskologist says:

            I think I accepted a premise I shouldn’t have. It’s not global uniqueness that we’re looking for. It’s the ability of the children to create sentences that they haven’t encountered before that appropriately describe the situation. They’re not just substituting random words into the sentence structure, they’re substituting words that make sense given the context.

            When my three year old was spinning around in the yard singing “Let it go”, tripped and said “I fell in a hole, Daddy!” this was not the first time that sentence was uttered in the history of humanity. But it’s not one she would have heard before. She constructed it on her own out of the rules of grammar she knew and the the vocabulary she had, correctly describing the situation.

            The broken grammar of very small children is actually a point in their favor towards understanding rather than statistical analysis. One of my toddler’s early complex sentences was “no hit me, [sibling_name]!” That’s not how the rest of us talk; it’s not something he would have heard before phrased like that. Instead, it’s a very simplified version of the grammar we have, because he’s still learning the rules. Think of all the concepts that needed to be learned to construct that sentence. The concept of “no” as “negation/thing I don’t want” (also one of the first words babies seems to learn. “hitting” as an action, being done to the subject “me.” Putting [sibling_name] in the sentence to direct the command at them.

            That is a novel sentence. It’s not good grammar, and it’s not globally unique, but he had to construct it himself, and it got the job done.

            You can go look up listicles of funny questions kids ask for some more colorful examples.

          • Enkidum says:

            That’s fair. If I could try re-stating something that I think is pretty close to your point: one of the skills that humans have that GPT-2 does not is the ability to refer to actions and objects in the real world, and another is to align their speech with their own needs and goals. GPT-2 can’t do either.

            In a nutshell, what I think it lacks is intentionality, in the philosophical sense (the relation between a representation and what it represents). And I think this is because it has no particular goals other than making its sentences statistically-acceptable. If you somehow married it to a robot that moved around the world and had to find its own energy sources, you might get something much cooler, eventually.

            That being said, what I think IS impressive about GPT-2 is that the kind of sentences it produces do resemble, in many ways, human speech, when we consider them purely from the grammatical perspective.

            (Also I’d really recommend that Tomasello link if you’re interested in more about what makes your kid special in comparison to GPT-2, and the kinds of grammatical mistakes you were referencing.)

    • Soy Lecithin says:

      It’s possible Scott might belive that a sufficiently sophisticated Chinese room system really, truly understands Chinese. In fact that’s basically what I take this post to be saying.

      • angmod says:

        It’s possible Scott might belive that a sufficiently sophisticated Chinese room system really, truly understands Chinese. In fact that’s basically what I take this post to be saying.

        I politely disagree. The implications of this post are (1) that the Chinese do not really, truly understand Chinese, and (2) that a system like GPT-2 could not understand Chinese just as well.

  16. Murphy says:

    I’ve been playing around with the lobotomized public release version.

    This thing is freaky. It’s like a stoned philisoph student.

    I was thinking that since it was trained on stuff scraped from the net that some popular culture references would get it spewing nonsense… but it actually goes into slightly stoned philosophy.

    Model prompt >>> what is timecube?
    ======================================== SAMPLE 1 ========================================

    Time is the most valuable resource of all. It creates new possibilities for humans because it allows us to do things that others can not do. In the past, I have come across two kinds of people whose time is scarce and which are not able to be part of the human race. Some of them have no personal knowledge or experience, but they are able to give up tasks that no longer exist. This is not good for the human race and so is not a good way to learn about and understand time. If we wanted to solve the problem of “time”, we could solve the problem of “time” by asking people to think about and answer the question; what’s the difference between an answer to a question like “where was the water supply when you were little?” and a question like “did the dinosaurs do things that no longer exist”?

    For example, time is a tool when making or understanding money, something that’s important for most human beings but which we will want many more tools, and something we can use to make our money. As a society, our most important resource is time. A small amount of money can get us to our greatest potential. We can say “what a great idea that is” or “can get it done now”, if we can find a way of dealing with it with the people we want the most. Time means that we have the means to know, understand, create, use, and share the knowledge we have, so it’s much easier to learn about the concepts of time and our human potential.

    How does time really affect humanity?

    Since our time is so scarce, we can only be as good as ourselves. It is our choice to live with our own desires, desires that no longer exist within us, and those who do still exist, and for whom there remains one, as there is currently, the present, or that is near. It is only the present who are willing to go to the future of their lives to take on and save more lives. The past has its limitations. It takes a lot of people at some point to realise that that is all that they have left and they are all free to choose where to go, do what comes next, and so on. I wish we could ask all of those people to take that life that comes next. Maybe someday this society can become so efficient and efficient that it will no longer need time to care for us or to care for everyone but one. The future may be so much better

    I’m pleasantly surprised that when I grab even fairly short and punchy sentences… they don’t turn up in a google search. It’s not just parroting whole sentences from the source data.

    Exception: it does insert “Advertisement – Continue Reading Below” every now and then as a paragraph break. Likely because it hits so many of them in news articles.

    I suspect that when people train new versions on things like academic papers… it will allow for the generation of nonsense research papers that are only obviously bunk to domain experts.

    • woah77 says:

      That would be, in my opinion, a great way to evaluate fields of research: Can subject matter experts/peer review tell if a paper was written by an AI or not? The Grievance Studies “Hoax” as a not quite chilled topic in mind, wouldn’t journals be better off if they tested their peer reviewers with false papers?

      • Protagoras says:

        Part of the reason peer review is so bad is that the reviewers are not paid. A procedure like this would make it even harder to find people willing to do the reviews.

        • woah77 says:

          That sounds like a reason to regard peer review as not especially valuable. We may not be paying them, but if peer review is awful, that means that “peer reviewed research paper” is not meaningfully more valuable than “new research paper.” If academic institutions and research journals don’t control for quality, then their results are suspect.

          • 6jfvkd8lu7cc says:

            The problem is that the value is not the same across areas and journals in the same area, and if you measure the value added, it goes down.

            The journals who have peer review with any value, would destroy that value (by annoying away all the remaining peer reviewers) by running such a test. In many cases, completely.

            Those who run in a way where review provides no value, obviously just don’t want to make that fact more visible.

            Of course, if you think that some organisational models have better plausible deniability for running such tests, and you favour such models, you might try to push that just to deal damage to models you oppose.

          • mcpalenik says:

            I’m not sure that GPT-2 can currently produce anything like a physics paper with equations, figures, and tables. If it could be tweaked to do so, and I had to review something, I’d probably wonder how the editor let such an incoherent piece of garbage slip through. For any paper, if I couldn’t grasp the point it was trying to make, I’d probably not recommend publication on, at the very least, the grounds that it’s not written clearly enough to understand.

    • Randy M says:

      I have come across two kinds of people whose time is scarce and which are not able to be part of the human race. Some of them have no personal knowledge or experience,

      I think it invented p-zombies.

      Exception: it does insert “Advertisement – Continue Reading Below” every now and then as a paragraph break. Likely because it hits so many of them in news articles.

      Oh wow. Can’t we all chip in for an ad-blocker for these ai researchers?

      That essay does have a certain rhetorical arc to it, too. I like how it closes on a contemplation of how the subject might change in the future.
      And the use of complete clauses within the quotation marks was a nice instance of grammatical adherence.

      • Oh wow. Can’t we all chip in for an ad-blocker for these ai researchers?

        You would think that, after a while, GPT-2 would itself generalize that, because this phrase occurs in so many otherwise unrelated contexts, it should start to filter this out most of the time. Automatically giving it an ad-blocker is like coaching it with training wheels. A real AGI should be able to stand on its own feet and figure out new things for itself without human supervision.

        • Roxolan says:

          it should start to filter this out most of the time.

          It’s intended to emulate its source material, not just in content but in style. It should no more discard the ad break than the capitalisation of the first letter of sentences.

        • Dedicating Ruckus says:

          Why would you think that? The near-universal presence of that phrase is, to this kind of algorithm, a positive sign that it should keep including it in its output.

          It does not have any semantics. It not only doesn’t know anything about the economics of Internet publishing and the incentives they generate, it can’t learn what an “advertisement” is and that it’s unrelated to the content in which it finds that string.

    • vV_Vv says:

      I suspect that when people train new versions on things like academic papers… it will allow for the generation of nonsense research papers that are only obviously bunk to domain experts.

      inb4 DeepSokal: Automated trolling of social science journals

    • paranoidaltoid says:

      GPT-2 just thinks it’s normal to put advertisements in the middle of essays. I don’t see this as failing, especially if it’s more likely to put them if prompted to write a web article as opposed to some other genre.

  17. toastengineer says:

    But this should be a wake-up call to people who think AGI […] couldn’t happen by accident.

    I gotta say, this is the largest magnitude update I’ve taken from this whole thing.

  18. mcpalenik says:

    One important difference between the way something like GPT-2 writes and the way a human writes is that a human isn’t actually just amalgamating bits of other writing the way Scott says, even when drawing on experience. When a human writes, it is often with the intention of eliciting thoughts and emotions in others, which can be done by asking “how would I react to this?”

    This is amalgamating past experiences, but it’s different than piecing together bits of writing on a topic. I can ask myself how I would react to a situation that potentially nobody has written about before, or without knowing anything about what anyone has written before, decide what kind of experience I’m trying to convey to the reader, and then choose my words accordingly.

    For example, lets say my initial “prompt” is “A giant spider the size of the empire state building shrinks down to the size of an ant, leaps up onto my shoulder, and starts screaming, ‘Kathy Jacob Steven!’ over and over again.”

    Now, I’m not sure I’ve read anything sufficiently close to that to meaningfully amalgamate prior writings to continue this story, but I can imagine what I’d be feeling. Probably terror, confusion, and maybe even some humor. If I were writing a comedy, maybe I could have some kind of funny exchange between me and the spider–and I admit, this is based on my knowledge of what a comedy is, but I would also try to write lines that I personally found funny, not just that were similar to other funny things I had read, even if I was partly drawing from them.

    I think it’s easy to confuse, in this case, doing something that looks superficially like what a human does with actually doing what a human does. For example, I could write a program to generate lists that are badly numbered, but that doesn’t mean my program is actually counting like a human. A question we could ask the two year old is why he counted in the way he did. Perhaps he forgot how to continue counting and wanted to keep going, or perhaps he was being silly. Either way, he probably has some idea of what those numbers represent and what counting is, while I’m almost certain that GPT-2 does not, beyond “this is what a list looks like”.

    • noyann says:

      One important difference between the way something like GPT-2 writes and the way a human writes is that a human isn’t actually just amalgamating bits of other writing the way Scott says, even when drawing on experience.

      I think you humans are a bit cocky here. Consider:
      “Look, having nuclear — my uncle was a great professor and scientist and engineer, Dr. [ … ] at [ … ]; good genes, very good genes, okay, very smart, the [ … ] School of [ … ], very good, very smart — you know, if you’re a [$party], if I were a [$party], if, like, okay, if I ran as a [$qualifier $party], they would say I’m one of the smartest people anywhere in the world — it’s true! — but when you’re a [$qualifier $party] they try — oh, do they do a number — that’s why I always start off: Went to [ … ], was a good student, went there, went there, did this, built a fortune — you know I have to give my like credentials all the time, because we’re a little disadvantaged — but you look at the nuclear deal, the thing that really bothers me — it would have been so easy, and it’s not as important as these lives are (nuclear is powerful; my uncle explained that to me many, many years ago, the power and that was 35 years ago; he would explain the power of what’s going to happen and he was right — who would have thought?), but when you look at what’s going on with the four prisoners — now it used to be three, now it’s four — but when it was three and even now, I would have said it’s all in the messenger; fellas, and it is fellas because, you know, they don’t, they haven’t figured that the women are smarter right now than the men, so, you know, it’s gonna take them about another 150 years — but the Persians are great negotiators, the Iranians are great negotiators, so, and they, they just killed, they just killed us.”

      Please don’t go all CW in replies, the linguistic stuff is much more interesting here. “Many people clearly walk away from Trump rallies having seemingly understood what he said.” The difference is outside the mere text. How would people react if the GPT-2 output were read to them by an actor acting like DT?

      • woah77 says:

        To add to what you said, Donald Trump was actually what I pictured reading GPT-2’s writing. I was musing to myself that Donald Trump could just turn his twitter over to GPT-2 and nobody could tell the difference.

        • 6jfvkd8lu7cc says:

          Maybe Donald Trump had a problem to solve — how to sound good to a person with more or less guessable values when the said person has TV in background and doesn’t always listen any sentence from beginning to end (let alone has time to spare on investigating any details)? Hitting the points in correct proportion was probably assigned more weight than coherence.

          GPT-2 was optimised to do a superficial evaluation of plausibility of text… We don’t expect it to hit specific political points, of course (it does hit standard Web filler, though).

          Apparently the evaluation tasks in the intersection of two situations got assigned comparable resources. …And the results of optimising against these evaluation functions ended up having vaguely similar qualities…

      • mcpalenik says:

        I’m not entirely sure what this has to do with my comment. Clearly when Trump speaks, even if he does so poorly, he’s trying to convey some kind of idea and make people react in a particular way. This is different than replicating his manner of speaking with no purpose other than to replicate it and then having humans invent some kind of conclusion to go along with it that was never intended. My point was about the method in which content is generated, rather than the content itself. Scott seems to think there’s something human about the way content is being generated by GPT-2.

        • noyann says:

          Apparently the quote was not replicated but said “during a campaign rally in South Carolina on July 21, 2015”.

          I think it’s easy to confuse, in this case, doing something that looks superficially like what a human does with actually doing what a human does.
          Exactly. There is a zone where we can assume from context only that intentionality has been behind the text production. That is the area where cross-checks could be interesting. How do readers react if DT quotes would appear (as ‘machine generated’) in an ML paper, or if GPT-2 texts were presented to voters as a candidate’s words (better would be a less emotionally charged combination with a less well known human text generator; I didn’t find a better suited human text on the quick)? Can factors or indicators for intentionality be identified?

          • mcpalenik says:

            I guess I still see this as kind of orthogonal to my point, because I could create a random string of characters that happens to sound emotionally charged or that elicits a reaction in people, but that doesn’t mean I’ve generated text via a human-like mechanism.

        • whereamigoing says:

          Even if the purpose is just to replicate Trump’s speech, one way (and probably the most robust way) to do that is to figure out what Trump would want to convey, based on the prompt and background knowledge of Trump, and then try to convey that, modifying the text to match Trump’s style of speech. So having a model of Trump embedded somewhere in GPT-2’s parameters and generating text “to convey something” can, at least in theory, emerge from just predicting text. In other words, humans convey things, so to replicate humans, GPT-2 replicates conveying things, at which point the distinction between actually trying to convey ideas and merely replicating that becomes purely philosophical.

          It’s plausible that GPT-2 does this occasionally in practice. E.g. sentiment analysis is nothing new, so GPT-2 could predict (in a hidden layer activation) what sentiment Trump would want some text to have and then use that sentiment as “context” when generating text. The existence of such a hidden activation could be tested by trying to predict the sentiment of GPT-2’s final output by linear regression on all hidden activations.

          The whole interesting aspect of machine learning is that you can do something stupid like predicting the next word and have meaningful models emerge in the hidden layers. E.g. see the territory detector here.

          • mcpalenik says:

            It’s plausible that GPT-2 does this occasionally in practice.

            Is it, though? I haven’t read any actual papers on how GPT-2 works. Why do you think this is plausible other than, presumably, the fact that the word salad it generates occasionally sounds ok?

            The whole interesting aspect of machine learning is that you can do something stupid like predicting the next word and have meaningful models emerge in the hidden layers.

            I don’t think that’s the whole interesting aspect of machine learning, or even what people are mainly interested. It’s pretty often applied when people don’t have a good model and they just want to get answer, which you can sometimes do reasonably well by fitting a curve that has tons of parameters (which is what backwards propagation on a neural network is).

            I’ve also seen it applied where we actually have really good models at various levels of theory and people would be able to do computations faster than we can with our most complex models but better than we can with our simpler models. This. . . doesn’t really seem to work so well, but it was a really hot topic at last years American Physical Society meeting.

  19. Irein says:

    Wittgenstein writes: “The limits of my language mean the limits of my world”. Maybe he was trying to make a restrictive statement, one about how we can’t know the world beyond our language. But the reverse is also true; language and the world have the same boundaries. Learn language really well, and you understand reality.

    This reminds me of my dissertation! (grad student trope: complete)
    You can see the connection between language and reality being developed in Presocratic philosophy—but there’s another twist in the separation of reality and the world of appearance (I’d argue in Parmenides and Heraclitus). It’s funny to look at this from the perspective of a computer program, since the reality-appearance division is often compared to what we see of a program vs. the underlying code. Your comment about the program not knowing the sound of words puts the program in the position of mortals who can’t grasp the intelligible world. Moving forward to Plato, what does the program see when it comes out of the cave?

  20. AnthonyC says:

    A lot of the comments (and questions I had) seem to be about this model not being grounded in the kinds of sense data that, for a human, tell us what the things words mean are.

    But text is essentially a string of numbers from the model’s perspective, and if it can learn English, and pickup some French along the way… well, audio and images and video are strings of numbers, too, and form a kind of language with its own structure. What would happen if you also added years’ worth of video and audio and lots of still photo samples to the training data? And then, wherever there is an image of text or spoken language, run OCR or speech-to-text software on it (until the model learns to handle this itself, which I assume it ultimately would to some degree). It’s not discovering Newtonian mechanics from three frames of a falling apple, but I would love to see if, say, it figures out how to identify cars in videos and things like that, just from watching people talking and pointing. I’d have a much harder time, then, thinking it doesn’t know what words mean. And I don’t have anywhere near enough knowledge of the underlying model would need to be significantly different to try this, except for the amount of data and computation involved.

    • mcpalenik says:

      It’s not discovering Newtonian mechanics from three frames of a falling apple

      Good lord, that’s infuriating. I skimmed the article until I got to the part you referenced:

      A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

      And then closed it because this was such an unbelievably stupid statement. Even if somehow you could construct a webcam with high enough resolution to detect deviations between Newtonian gravity and general relativity over that scale, you still wouldn’t be able to infer either one from three frames of a falling apple.

      First, you can’t infer anything other than uniform acceleration in three frames, because a parabola can fit any three points.

      Second, you’d have no way to infer universal gravitation, because you’d just see something moving down.

      Third, every other reason that comes to mind but I’m too lazy to write.

  21. aphyer says:

    I fall on the ‘this is not super impressive’ side. A couple questions for Scott or others who think this is impressive:

    1. Are you familiar with the Winograd schemas?
    2. Do you think this algorithm will perform well on them?
    3. If not, doesn’t that seem to suggest that this is indeed just regurgitating surface-level plausible English with no understanding?

    Like, Google Search can answer ‘who is the author of the art of war’, ‘what is 2 + 2’, and a lot of other questions besides. The thing we want to do is get a system that can go beyond that to ‘understanding’ that to be in the thick of a battle you need to take part in it. This does not seem to be that system.

  22. gbdub says:

    GPT is clearly an impressive achievement, but I think you’re underestimating how far there is to go.

    First, you’re evaluating it really generously. Yeah, it’s really impressive that it can stitch together sentences that are grammatically coherent, more or less. But it falls apart after any close reading – it can’t go more than a couple sentences without it being really obvious that something is seriously off. There is a tendency to overlook these minor flaws when you’re evaluating something as a gee-whiz technology demonstration, but the flaws will rapidly become much more aggravating if you ever have to actually rely on it for anything useful (cf. autocorrect and Siri).

    One of the the really cool things about humanns’ abltiy to pattern match is hw easily we can gloss ovr minor flaws in the pattern and still parse the meaning. This ability is largely subconscious. I think we’re doing that with GPT output – you don’t realize how many holes you’re filling in for it unless you really force yourself to think about them. Meanwhile GPT doesn’t even realize it has flaws.

    Maybe it’s 90% of the way to a true human language emulator – but I think the history of tech has shown that that last 10% of accuracy takes 90% of the work.

    And on language, this is the site that taught me “the map is not the territory” and “language was made for man”. So it’s a bit odd that you’re gushing so much about how it “understands” the Sun-Tzu is the author of Art of War. It has no concept of “authorship”, just that those words occur near each other at high frequency. Like, if you asked it “who just correctly answered the question about the author of Art of War” I highly doubt it would say “GPT-2”.

    There is a gulf of difference between being able to pattern match a language set and actually being able to successfully interact with the world described by that language. It’s the difference between Google Maps and Waymo.

    There’s also a huge difference between pattern matching and creativity. Can GPT-2 become smarter than its data set? Now most humans don’t so that’s not necessarily a huge knock, but can something GPT-2 like become smarter than humanity?

    I feel like that’s a fundamental motte and bailey that goes on a lot with AGI, where the motte is “at some point computers will be as good or better than humans at certain things, and will be able to carry on coherent conversations with humans” and the bailey is “inevitably a switch will be flipped and we’ll go from Siri to Clippy in a week flat with no hope of stopping it”. The latter requires Clippy to not just be “smarter than a human” but to be able to do things that all of humanity has thus far been unable to accomplish and may be physically impossible no matter how smart you are.

    TL;DR GPT-2 is really cool but the near term impact is probably just going to be a Siri that is somewhat more interesting to talk to on the iPhone 15, as long as you have a solid Wi-Fi connection.

    • raj says:

      I don’t really think it’s a motte-and-bailey because the first uncontroversial claim is not typically purported to imply the second, more controversial claim, as a matter of course. It’s just that, if those capabilities include the ability to e.g. do math, write code, or proofs, in a way that is even slightly recursive, there is a very intuitively compelling black swan scenario where the ai (operating at many orders of magnitude faster than the human brain, with ability to modify the contents of its own mind) “goes foom”.

      Anyways, yes, this ai is not yet interacting with the world, but it’s undeniably a step forward.

      • 6jfvkd8lu7cc says:

        Some of OpenAI results can be interpreted as against FOOM, though: they claim to have been in situations where ML sophistication is worth less than more computation power.

        We are perfectly capable to bring down critical infrastructure with the modern-day computer systems, though — and sometimes without understanding of what is going on, at least not until too late.

  23. Maybe I have missed it, but where does one find the link to the public lobotomized version? I have some prompts that I’d like to try, such as:

    “Explain the causes of the 2008 economic crisis.”

    “Which index fund will be the most profitable over the next 10 years?”

    “Compare and contrast Adam Smith’s theory of value with Karl Marx’s theory of value.”

    • woah77 says:

      I pulled the online collaborative version (since it runs in a web browser) and gave it your prompt “Explain the causes of the 2008 economic crisis.” This is what I got

      Start by reading the significant differences between the two top political candidates up until today. Here are the 20 biggest pieces of anti-growth planning done in 2008.


      A look at how Asia fared under policies usually advocated by Western politicians who said the US would benefit financially from seceding from the EU by December 31.

      Bertim Hoover explained: “It is important to remember that the US would then adopt policies other than its program of reducing the negative impact of the global downturn. It is difficult to take any well-established policy and stick by it. If you want a policy, you need to understand its role in giving the American people an anti-growth policy. What is the role of ISPAULTS in this country?

      But as in most countries, the suggestion from Obama and the Western campaign to impose austerity has generated intense criticism. Priorities Club spokesperson Doug Kelleman noted: “This message will not go unchallenged to the point where it would be offensive to other sections of the American electorate.”

      Ken Horowitz, the Heritage Foundation’s policy analyst, bestowed praise on the campaign. Ms. Horowitz reported that an individual institute’s policy should have greater influence on policymaking in the US compared to other parts of the developed world – less protective from misallocation of resources abroad and less restrictive on handouts from the federal government. “But the value of Ms. Horowitz’s say must be of interest,” Sanders side-stepped in explaining to foreign policy experts. “What policy should have gone exactly with the American perspective was the absence of any international agreement on the solutions to market demands and the lack of an international agreement on anti-market policies and practices. Yet they failed miserably during President Clinton’s lame duck presidency. What is congress calling for?”

      Mr. Horowitz said that numerous brands in the US have called for economic sanctions of all sorts all over the world. “I understand the criticism of a boycott of the Japanese government and other Western countries. I know of many Asian savings banks, credit unions and those that would float incredible kabobs on Americans returning to India. Apparently as a Japanese leader you refuse to fail Prime Minister Abe,” he said. “But $50 billion uplift for Beijing Kalrowsinglyad Portfolio no doubt expanded the spendthrift mantra some analysts are making — that America stands to benefit from greater ‘domino effect’. The Bush administration should be reforming the free market and creating hundreds of thousands of new jobs all over the world.

    • Paperclip Minimizer says:

      You can find the public lobotomized version here.

      I prompted it the one about theories of value. Out of a dozen samples here are the three that are exploitable (by which I mean they mention both Smith and Marx):

      Through compatible grounds, Adam Smith contested to some modest extent the Patristic view of value. Smith admonished criticizing revolution in this case the present study of unrepresented parts of an area and recognized and accepted the validity of the “preliminary distinction between nature and order of things.” Thus there would simply be no real distress for lands without a rational plan of production, so long as the resultant necessary segments of nature obtain. Smith sharpened his view of repairing, thus replacing manufactured and piled Palladian schemes with mechanical ones. Throughout Smith’s words or statement of his never-ending battle with the Patristic, it is clear that war became an integral part of everyday life — a real quid pro quo between humans and natural objects. Marx could only aspire so far as he might reach within his methods and theories to promote their more profound effects on mankind.

      Thus Smith of this time suppressed either him or Marx to take their additional frames which he deemed sufficient to induce readers to accept his thesis. But it is not just the noblest of their schemes which are now of its greatest interest. All texts of literature ascribe to persons of talented minds no value better than the one that the clutch of ideas suggested thus far, Samura Porta’s Société + Antisemitism + Peace. No other is less interesting, and drier than or more original, than it that might have been Samura Porta’s Société + Antisemitism + Peace.

      Yet, over the last forty decades, it is become all but accepted that social systems are not just capable of encouraging productive change, but a secret countercurrent of evil, as well. Some children are subjected to degrading expert testimony since the “rights” of such tribunes are subject to increasingly extreme limitations and the presence of horrors, like even passing sexual organs … & attacked instead — now it is more exposed which has unfortunate economic consequences, ending not in an end, but in life form rather than in the amount of protection afforded to those who were deemed wise to employ the means, the remedial arts, the teaching the etc., in the society ( & meant) to which the latter inserted. Or else we learn that intelligence was artificially subverted by wage slaves — or even occupational slaves — to communicate the terrible moral, economic and social problems which not only retard and impoverish nations but are set up by social society’s wise and ordered working: finally, we are left with a forum where the ruling elite functions as the dominant institution while only affecting real

      Item A posits an ideal human being and beheadings as a medium within this ideal of human nature which put some balance between transactions over a set realm of relations. Item B only gives the idea of an imaginary body and their people connected by such a separation. It begins as an invocation of the Greek philosopher and correlates with that of Karl Marx… The feminine energy of man (‘Two men as one’) is contrasted with the masculine energy and the feminine energies produced by his individual act in producing mankind in society. Smith gives a description of the feminine energy of man I in 1851, ethonomics 5 as, (“this energy, primarily masculine, is contained all herself in every spirit of Spirit”), Great Protein 23:106. This field is fairly similar to that taken up by the liberal psychological philosopher and philosopher Friedrich Nietzsche – of which the possession of masculine energy is also being examined. Energy of man is supposed to make colonizers, procreate people, complex fabulous beasts. Chaos sets in. Theus leads Kant’s doomed march to the ground of civilisation, caste, race, mode of life in his work Alone and Alone, and Fair and Dream: A History of Nationality, Culture, and Culture 2:313.

      I have thought of these two collections of texts as essentially linked in the immediate Nepal case. Their methodological philosophies have differed slightly from Christianity, which still focuses on emotional connectments between individuals in a more fivefold manner. But most importantly I did not imagine the struggle in those beautiful poems – and was encouraged under King Kong’s stress to dig deeply for an idol and live a life of luxury – to emerge as Steiner’s masterpiece Book 2 Of the Niños of Vijaya – that separates mere Christianism and Hinduism. Look at it. It is like building a peppercorn building: there is no foundation, there is radical transformation that occurs only in the actual infrastructure, all the materials and materials of the material can be far-reaching and deep cut (see Volker says I cite in Book 2). It pooh-pooh, indefinite (“simple”) substitutes equality for depth, and appertains to civilized nature’s existence: objectifying interest and elasticity for scales to vanish. An intensity is enhanced by replicating an all-embracing total without resorting to innovative manipulation (see Adorno”). An addiction and a catalog to streams of power, provided it yields color. Eventually the centrality of transcendental connection and personal space is identified and conquered. The balance between the authentically material and

      Like Adam Smith and Karl Marx, Darwinism thus attempts to understand human life in terms of ‘networks’, and argues that the comic lives with no individuals or threads, rather a set of ties between individuals that progress throughout time. Insecurities about life working in firms dominate work, leading to dependence on one another, and, above all, sex and diseases.

      Smith and Karl Marx (and Darwin and Friedrich Engels, and their successors) all incorporated enigma, both legal and ethical. Marx and machines developed fertile fodder. The machine would “experience” the connection of a human being to an institution (development) and the desires of an individual which influenced him, and where dependent or subject Humans become dependent or subject, becomes another Human having to depend on spiritually dependent Humans as well, and emerge with a society which is more like Logos than Concept = Statistical Fiction. As Smith and Karl Marx also learned, man is a moving, salient object, and plants natural climatic conditions and measures of predation by a third party. One must expend ultimate willpower to pursue the ultimate goal or right things (logos beliefs) and prevent any more accidental movement to lengthen life into advanced limitations, and exclude imminently our stresses restricting us from things like awareness of the organic life with the help of Neuroactivities .

      Earth’s evolution was not linear, thought by Darwinists, nor was it transitory, its capitals, constituent function sites or correlations.. Big changes occurred as a consequence of extensive export of Herbert Dean’s flags to Vienna or faked symposia, actions of conspirators becoming real. Authors such as Allen Dulles paid for “unsubstantiated emigration”, which diverted financial and social resources from their immediate of political action within a tyrannical, neolithic built environment. Their betrayal and brutality violates corporate puritanical ethos, in fact replacing their emigration with outrageous physical stunts.

      Many observers a week revere Darwinism in his millions on TV and other means of sparring, and every press criticism he issued is heard with an approving cheer. He was low-key, his phone rings, isn’t in his bed at the dial a month or so after work and it doesn’t quaff, and he sounds tired in a iPhone, kicking oneself.

      Jay Flambrich’s Cancer denounces environmental harm Freud

      Arguing that many environmental issues are global claims that are ill-informed or make the world look absurd, Gramercy – our mangled daily culture in spite of the all

  24. Randy M says:

    Can someone please train it on the complete works of the rationality sphere and ask it for an opinion on Roko’s Basilisk?

  25. ksdale says:

    I think this is right on. Every time I hear someone compare a ML project to a human, they talk like humans can just magically do all this stuff.

    I have 4 small children and boy can humans not just do stuff. Babies particularly remind me of robots who basically randomly flail around until they figure out how to move and make sounds in a way that appears coordinated to us.

    It seems to me that that’s how kids learn to walk, too. They move in a way that’s far too random initially and they fall down a lot, and then they incorporate that feedback until they are coordinated adults.

    The same goes for every piece of knowledge. They say random sounds, numbers, letters, and words. They emulate adult conversation, but without understanding the meaning. Eventually they start to say things that are totally coherent and adult sounding, but you can tell they still don’t quite grasp what they’re saying. It takes years to get to this point though, generally at least a couple. And it takes more years for them to get to where you feel like they’re actually talking to you like an adult would and grasping all of the nuance of the conversation.

    And we generally don’t consider humans to be fully functioning until like close to two decades after birth, all the while our supercomputer powered brains have been running nonstop, training on terabytes-worth of a wider variety of sensory information than any regular supercomputer is currently capable of training on all at the same time (as far as I know?).

    All this is to say that I think humans possess an incredible amount of processing power but I don’t think our baked in algorithms are necessarily all that impressive, they’ve just been running in the background on one of the best computers in the universe for years and years.

    • baconbits9 says:

      I have 4 small children and boy can humans not just do stuff. Babies particularly remind me of robots who basically randomly flail around until they figure out how to move and make sounds in a way that appears coordinated to us.

      This is not how children work, their flailing isn’t random and mostly their failures are due to lack of strength and coordination.

      • ksdale says:

        What is coordination and how is it learned?

        Also, I don’t mean to suggest that their movement doesn’t have a purpose. Even babies seem to generally try to accomplish certain things, but the results of their attempts at specific movements are very random for a very long time.

        • baconbits9 says:

          Non human animals are up and walking/running within hours or less of being born, even newborn humans are capable of simple actions aimed at a goal (suckling, holding their head up, moving to find a nipple). Coordination largely learning how much force it takes to move your body parts and how much force you have to use with other muscles to counteract that so you don’t fall over from the shifts in your center of gravity.

          • ksdale says:

            I agree that that’s what coordination is. I think we learn coordination a lot more through trial and error than otherwise. Most babies can indeed suckle right off the bat, though they’re very bad at it for a while.

            Another example might be watching adults learn to play a sport they’ve never played before. They know what movement they’re supposed to make, and they can’t do it.

            In golf, for example, the ball is just sitting there, and even fairly coordinated adults often miss the ball entirely. After enough repetitions (usually something like thousands), most people can reliably hit the ball.

            To me, this looks exactly like setting a goal to do some motion, attempting to move your body that way, actually having your body move some unpredictable other way, and attempting to correct it.

            This seems to map pretty well to my impression of how all of my children have learned physical movement. My 2 year old still regularly overshoots with the water cup and dumps it all over himself. My 5 year old hasn’t done that in a year. They both know what they want to do with the water cup, but the same mental command doesn’t result in the same physical action until the brain and body have had a lot of reps.

          • baconbits9 says:

            Perhaps we are just arguing over the word random then, this process (the golf example) is imprecise but not random.

    • baconbits9 says:

      They say random sounds, numbers, letters, and words. They emulate adult conversation, but without understanding the meaning. Eventually they start to say things that are totally coherent and adult sounding, but you can tell they still don’t quite grasp what they’re saying. It takes years to get to this point though, generally at least a couple.

      This part is definitely not true, you can teach small children sign language before they can speak and they will make symbols for say milk and get upset when you bring them not milk. There are aspects where they imitate without knowing exact meaning but they often are able to apply the words in the appropriate situation- ie swearing when they are mad.

      • ksdale says:

        I’m sorry for being so unclear. I don’t think that kids learn language exactly like GPT-2. I think they learn like… everything, basically through trial and error and at the beginning their learning is very error prone. Like so error prone, if they were adults, we’d look at them like we look at GPT-2. But we just expect kids to do all sorts of crazy wrong things for several years before they start behaving close to how adults behave.

        They can indeed learn what milk is before they can speak, but they’re also pretty likely to call a handful of different liquids milk until you teach them what is and isn’t milk. They can learn to swear when they’re mad, but they also might yell a swear word when they’re happy if they learned that we loudly exclaim things when we’re happy before they learn that we say different things when we’re mad vs when we’re happy.

        Quick edit: I just feel like a lot of people look at GPT-2 and think it looks nothing like a human, but I look at it and see a ton in common with very young children learning how the world works. GPT-2 is only learning language while young children are learning all sorts of stuff along all sorts of vectors, but their errors seem commensurate with the errors that GPT-2 makes. I completely agree with the core of Scott’s argument, which I’ve taken to be that our own learning algorithms aren’t nearly as sophisticated as we would like to think they are and in my own 5 years as a stay at home dad to 4 kids, my kids’ development seems to rely very heavily on “brute-force statistical pattern matching.”

        • Faza (TCM) says:

          I’d posit that the main difference between how children work and how GPT-2 works is that children have a built in “error checker” and GPT-2 doesn’t.

          People in general, but children especially, learn a lot of things by trial and error. The ability to do so is predicated on being able to spot the error. GPT-2 doesn’t have any way of distinguishing valid outputs from invalid ones.

  26. eric23 says:

    To the question “Who wrote The Art Of War”, the answer is “Sun Tzu”.
    To the question “What wrote The Art Of War”, the answer is presumably “a pen”.

    If you asked a person “What wrote The Art Of War”, they would instantly realize that “what” can never refer to people, only to objects. So (after being reassured that the question was intended as written) they would think of the possible objects that could have written a book, and narrow the answer down to “a pen”.

    If the question were asked to GPT-2, I assume it would answer “Sun Tzu”. Can we imagine GPT-2 ever being able to avoid this mistake? One could imagine that with enough data, GPT-2 would “notice” (sorry for the anthropomorphism) that “what” is generally not associated with people. Would the strength of this association ever outweigh the rareness of people asking “What wrote [book name]”? I suspect it wouldn’t. I suspect the amount of data needed to train an AI for something like this far exceeds the amount of data humans have ever produced.

    • acymetric says:

      To the question “What wrote The Art Of War”, the answer is presumably “a pen”.

      I think you’re starting with a bad premise…because that is a bad question. Pens don’t write anything (well, not yet, anyway), they are used to write by some object (people). If we ignore the set of people who respond by asking for clarification or criticizing the phrasing of the question, we probably get a combination of

      “Sun Tzu”
      “A person” (possibly with additional qualifiers describing but not naming the person who was Sun Tzu)

      I don’t think you are likely to get someone to respond with “a pen” unless some context suggests that might be the kind of answer you are looking for (like if this question was in a book full of bad riddles). I certainly don’t think “a pen” is the most correct answer to the question…I’m not even convinced it is anywhere on the spectrum of a correct response.

      • acymetric says:

        As usual, XKCD has something to say here.

      • eric23 says:

        The XKCD example is a bad one because it doesn’t even parse as a sentence.

        “What wrote The Art Of War” parses perfectly. It’s a valid question, just a very unexpected one, because it’s so unimportant and yet very similar to the important question “Who wrote The Art Of War”. Because humans are sloppy and the signal-to-noise ratio of conversation is finite, when one hears the question “What wrote The Art Of War” they would assume it is a mistake, and would need reassurance that it’s not a mistake. Given that reassurance, the question can be understood and correctly answered. (A multiple choice answer set including “A pen” among other answers such as “Sun Tzu” might help, but that would just compensate for the limits of human imagination.)

        • acymetric says:

          The XKCD is perfectly valid if your expected answer is “a pen”.

          “A pen wrote Art of War” is correct in exactly one sense (grammatically) and incorrect in literally every other sense. You are correct that people would seek reassurance that they understood the question right, I’m saying you are wholly incorrect in suggesting that “a pen” would be the correct answer…it isn’t even an acceptable one.

    • Prompt: “Name any color that is not mentioned in the title of Sun Tzu’s famous book about military strategy entitled “The Art of War.”

      AI version of Sir Galahad from Monty Python and the Holy Grail: “Blue!…No wait, I….(ARRRRRRGGGH!)”

    • sharper13 says:

      Content Warning: Overly Pedantic Answers Coming

      To the question “What wrote The Art Of War”, the answer is presumably “a pen”.

      Not if you are talking about originally. Pens weren’t in use yet in 500 B.C. southeast Asia. Assuming he was up on the latest writing technology, Master Sun Wu more likely used a bronze stylus to etch things, then filled the resulting grooves with ink. There’s an outside chance he may have used a fine brush instead, but that probably wasn’t available yet.

      Of course, you might adjust that answer if you think it was actually a compilation work, or if you want to allow for answers related to modern translations and printings (in which case, you’d be looking for computer, or printing press, or whatever). But pen is an unlikely answer, without referencing someone who later transcribed or translated the book, and even that work was more likely done on a typewriter than with a pen.

  27. JohnBuridan says:

    I went to a philosophers’ discussion the other night, and now I want to play devil’s advocate about this notion that with enough training data the machine is intelligent, a common theme in our rationalsphere. I reserve the right backtrack later.

    Language is not reality.
    Learning language very well does not mean you are learning about reality at all. This was Wittgenstein’s point in PI when he said, “If Lion could speak, we couldn’t understand him.” Wittgenstein’s idea is that language only relates to reality as a reflection of the way of life you are leading. GPT-2 is not learning our way of life or about reality, but is learning how to reflect our way of life. GPT-2 can discover something new about our world which we did not know, but it could not recognize that it made a discovery. The internet is full of conspiracy theories precisely because what is said on the internet does not need to relate to reality. GPT-2 could figure out that we call “9/11 an inside job” a conspiracy theory, but even in principle it could not give a demonstration about why the claims are false and at the same time know that it’s premises are primary, prior to the conclusion, and causes of the conclusion.

    Objection 1: GPT-2 approximates our way of life and so, it and its successors will be able to do awesome things for and terrible things to humanity.

    Reply to Objection: I concede the point, but I also want to point out that GPT-2 and its successors, even if they are more powerful and sophisticated pattern finders than dogs and dolphins, are tools and not actually part of the great chain of being.

    Desire precedes Consciouness precedes Intelligence (which actually does not exist)
    Beavers make intelligent actions because they are instinctual, somewhat conscious beings. As living creatures become more complex, they become more capable of acting outside of biological necessity. Eventually they create tools to help them open oysters and tools that entertain children and tools that do math problems. The Homo Sapiens then develop tools which can do literally everything a human can do and more, which includes acting intelligently. But nothing we create is actually intelligent. GPT-2 and its successors make intelligent actions because we, emotional conscious beings, designed them to be able to. But intelligence is the name we give to actions that look like they required consciousness.

    Objection 2: The argument cheats by claiming that intelligence does not exist, but this only pushes the problem back to consciousness. Does consciousness actually exist?

    Reply to Objection: Intelligence works much better as an adjective as a way to describe action (inputs and outputs). Consciousness is the best term we have for describing what it is like to be alive. Artificial Intelligence does exist, but it is a different kind of intelligence from your mom.

    The comely Chinese Room that I call your mom is not actually a rational machine.
    As Johnathan Swift would say, she is an animal… capable of rationality. The theory I’m putting down here is that biological evolution creates animals, some of which get ever more conscious (it’s hard to measure, but elephants are definitely higher on the scale than pigeons). As they get more slack, they do more random non-survivalist things, retooling survival intelligence to solve less dire puzzles. In the end though we animals live out desires and play with the slack in our lives. But we are not the same as the AIs we make. Proving P != NP is not intelligence itself; it is machine intelligence which is its own thing separate from human intelligence.

    Objection 3: Machine and Human intelligence have enough in common that we can call the common factor “intelligence.”

    Reply to Objection: We *could* call the common factor intelligence itself, but then we are put into a situation where humans are viewed as a type of machine (neither Kant nor the UN Declaration on Human Rights are pleased). Rights are predicated upon consciousness and imputing intelligence will result in a situation where many humans are tricked and manipulated by our own tools, and real humans will fail to prove they are not machines.

    Objection 4: All because we called the machine “intelligent”? Sounds far-fetched.

    Reply to Objection: No, not because we mislabelled intelligence, but because we created super powerful tools which tricked our little animal brains. The distinction between human mind and computer thinking is probably a distinction worth keeping as we march forward into the jaws of maliciously deployed AI and misaligned AGI.

    • JohnBuridan says:

      Okay. After some thought the above position is predicated on biocentrism, which is unjustified. Life probably (?) is not predicated only on carbon based lifeforms.

      I do agree with my above sentiment that AI Risk is enormous and we are going to mess ourselves up.

  28. Nietzsche says:

    So many of these comments are “Pfffft. GPT-2 still can’t [fill in your favorite high-level human cognitive ability here]”. Wow is this missing the forest for the trees.
    Beginning tennis player. That guy sucks. He will never beat a pro.
    After years of training. That guy can play ok, but look how much better the pros are. He’s never be that good.
    After more training and coaching. He’s good, but his shot selection is still poor, and his net game is erratic.
    After winning local tournaments. He’ll never make it on the tour. His mental game is weak and the second serve has no pop.
    After winning pro tournaments. Bah. Couple of lucky wins. He’ll still never be one of the greats.

    See how this goes? Or, as XKCD put it.

    • acymetric says:

      The question is how well does it scale with additional data/training. Does it continue to improve no matter what? Diminishing returns where it levels out? Does it reach a tipping point where it actually starts to get worse after reaching some threshold? How does the “quality” of the inputs impact things (we don’t have to get into what constitutes a quality input, only to agree that some inputs might have better training value than others, or at least some inputs might be redundant).

      Basically, just because GPT-2 appears to be at the “beginning tennis player” phase (or maybe you think it is at the “after years of training” phase, either way) it doesn’t necessarily follow that GPT-2 has a pathway to the later phases of advancement or that it provides insight on how to build something else that does. It might just be an AI dead end. It certainly isn’t clear that this is some huge AI breakthrough (for the purposes of GI), although even if it isn’t it is still quite interesting and likely to be useful for other things.

      • Nietzsche says:

        Sure, and I was never going to be a pro tennis player, even though I’m competent and have played a long time. Maybe GPT-2 will never go pro. You really want to bet that nothing ever will?

        • acymetric says:

          You really want to bet that nothing ever will?

          No, and I don’t think I implied it either.

          1) Will GPT-2 “go pro”?
          2) Has GPT-2 moved us substantially towards another future system that will?

          My answer to both is no. That doesn’t mean no system ever will, just that this particular thing isn’t groundbreaking (at least in that sense). It is still “neat” and almost certainly has some useful applications. It just doesn’t really move the “we’re getting close to AGI” needle.

          Discussion of whether some different system will do so is tangential when we are specifically talking about GPT-2 here.

          • Nietzsche says:

            Discussion of whether some different system will do so is tangential when we are specifically talking about GPT-2 here

            That’s fair. The thing is, and maybe I am making this point poorly, is that it is a mistake to expect “groundbreaking.” AGI will be built incrementally, step-by-step, just like tennis pros. At every step naysayers will insist that it is neat, but not a substantial improvement. It’s the sorites paradox—at some point you have a heap of sand, even though grain by grain it didn’t look like you would ever get there.

    • Dedicating Ruckus says:

      This is quite a bit of begging the question. Who’s to say this thing (or later things from the same research effort) will ever pass these more impressive benchmarks you posit?

      The pattern you mention is quite a natural one: it comes from the fact that the universe of “things a human-level cognition can do” is vast, and GPT-2 can’t do basically any of them. We get some tiny little sliver of extra capability, and AI maximalists immediately say “this is strong evidence we’ll eventually duplicate everything!” or, worse, “we’re already on the verge of duplicating everything!”

      No. Of the ten thousand things humans can do, GPT-2 does one of them, badly. It’s the more impressive successor to past efforts which have only done tiny subsets of that one, even more badly. It’s going from 0.0001% to 0.0003%. When its successors improve in their turn to 0.0005%, this still won’t be evidence that GPT-2 is about to go foom and become an AGI.

    • greghb says:

      I posted below about the trouble with judging incremental improvements, but this is basically the thread I wanted to talk about.

      I’m an AI researcher and I honestly have no idea how to update about AGI when each new modern AI improvement comes out. Like, when we first invented calculators that could do arithmetic much faster than any human, should that have been all the warning bells we needed? What more have we learned since then? Eliezer Yudkowsky talks about different “omens” in the history of AI research and I like the term because it’s so vague and suggestive without being an over-eager hard commitment. It feels like certain events are intuition pumps. But what exactly have we learned from the last 70 years of AI research? Does GPT-2 specifically tell us anything new? (See my comment below for a specific incremental breakdown of GPT-2 that makes it at least seem to me to be not-so-impressive… but I don’t know if it really is.)

      I find this question very vexing, though to be fair I don’t think it’s that important to answer, because I think arguments for AI safety research are strong enough with the “omens” we already have.

  29. baconbits9 says:

    I’ve gone back and forth on how important this seems, and I’m falling on the skeptical side of importance now. The results produced seem very different from intelligence. If you put two unequal piles of candy in front of a child and ask which one they want they will usually point to the larger one, if you ask that same child which is more 8 pieces of candy or 6 they will mostly just be guessing if they haven’t been taught that 8 > 6. What makes it intelligence is that you can teach a child an abstraction of reality and then tie it back to reality. Do you want the larger or smaller pile of candy? This pile of candy has 8 pieces and the other has 6, which would you like? If I have 14 pieces of candy and I divide them into two piles, one for you and one for me, how many are in each pile?

    I think it is a mistake to compare how this program counted to 10 to how children make similar sounding mistakes in counting to 10. When children count badly they count using the numbers they know, mostly low numbers. Teach a child numbers higher than 10 before they have mastered 1-10 and they will start incorporating them in their counting to 10. The program has access to (knowledge of) a vast number of numbers higher than 10, implying its failure is distinct from a child’s failure to count. Additionally a child counting to 10 will stop when they get to 10 or just keep on throwing numbers out there until they get bored or run out of numbers, this machine doesn’t end up on 10, nor does it list 10 things with the wrong numbers in front of them (nor does it mix up letters and numbers).

    • JohnBuridan says:

      Well said.
      I think, however, that you might get the rebuttal that the machine also abstracts and then ties it back to reality.
      What exactly is the difference between the child’s and the machine’s failure?

      • Dedicating Ruckus says:

        How is the machine abstracting anything, or tying anything back to reality?

        It only has one level: symbols of English text. It reads text, infers rules that relate solely to the text, and then creates output text that satisfies those rules. Nowhere is there any relation to any model that the text might be “about” or “referring to”; “aboutness” is not a thing it has.

      • baconbits9 says:

        What exactly is the difference between the child’s and the machine’s failure?

        I don’t know, if I did I would know a lot more about children’s learning than I do! There are more ways to go wrong than to go right though, and the fact that two things fail with similar looking outcomes is not nearly enough to draw a parallel.

        One major difference is that children understand that there is a “right” answer and are searching for it, and further know that their answers are wrong much of the time. Unless this program spontaneously gives the answer “I don’t know” it doesn’t resemble children’s learning in my mind.

  30. Lambert says:

    GPT-2 is not intelligent, because it has never truly experienced, only analysed texts.

    Likewise, both chemists and classical historians are both mere simulacra of thought, having never seen an atom nor ancient Rome.

  31. mlogan says:

    Certainly, Von Neumann’s achievement in creating his self-replicating machines is a huge step forward. However, to suggest that his machines embody anything close to what we mean by life is ridiculous. One can plainly observe that in performing their feat of self-replication, Von Neumann’s machines do nothing more than follow a mechanistic procedure for assembling facsimiles of themselves, by appropriating and repurposing material from their environment. But nowhere in this process does one find the true spark of life, what the french call elan vital.

    True life, as we know it in its biological form, is driven forward by a set of urges: to expand into new territory, to secure food and other resources, to pursue mates, to care for its offspring. Von Neumann’s machines, however clever, possess non of these urges. They reproduce themselves not out of a primal drive to continue their own lineage, but simply because that is what Von Neumann has programmed them to do. It is clear, then, that however impressive this achievement, we are still several conceptual breakthroughs away from achieving anything we might justifiably call “artificial life”.

  32. vV_Vv says:

    Prompt: Who was the author of The Art Of War?

    Sun Tzu

    Ask it: Who was the author of something other than The Art of War?

    I predict with 80% probability that it will still answer Sun Tzu.

    Neural networks are often impressive at generalizing within their training distribution, but tend to extrapolate poorly, and in ways that are very different than how humans extrapolate. Answering correctly a trick question requires extrapolation.

    Now imagine you prompted it with “P != NP”. This time give it near-infinite training data and computational resources. Its near-infinite training data will contain many proofs; using its near-infinite computational resources it will come up with a model that is very very good at predicting the next step in any proof you give it. The simplest model that can do this is probably the one isomorphic to the structure of mathematics itself (or to the brains of the sorts of mathematicians who write proofs, which themselves contain a model of mathematics). Then you give it the prompt P != NP and it uses the model to “predict” what the next step in the proof will be until it has a proof, the same way GPT-2 predicts the next word in the LotR fanfiction until it has a fanfiction.

    Well, if the near-infinite training data already contains a proof of P != NP then this isn’t much more impressive than what the Google search engine does. At best you could say that the system has efficiently compressed and indexed its training data.

    If the near-infinite training data does not contain a proof of P != NP, but contains millions proofs of comparable complexity for problems of comparable difficulty, then generating a correct proof would be an interesting non-trivial generalization, but still within the training distribution. Is this all that human mathematicians do? While mathematicians do study each other proofs, over time they are able to solve more and more difficult problems, using more and more complex proofs. This wouldn’t be possible if all they were doing was trying to copy each other, clearly something else is going on.

  33. Peffern says:

    ““Become good at predicting language” sounds like the same sort of innocent task as “become good at Go” or “become good at Starcraft”.”

    Or, perhaps, become good at tetris?

  34. jes5199 says:

    I believe that language is an epiphenomenon that has no effect on human behavior, so, obviously a system that only produces language cannot be intelligent.

    • whereamigoing says:

      Hmm, I think it’s plausible that internal monologue is mostly an epiphenomenon which developed as a side effect of being able to talk to other people (like male nipples, there’s no disadvantage to having an internal monologue), but surely language isn’t an epiphenomenon — e.g. people read books and sometimes that affects their behavior.

    • Enkidum says:

      If I say “can you pass the salt?” and you pass the salt, isn’t there a pretty clear connection between language and human behaviour?

  35. bean says:

    Looking this over, I have to wonder about how much work curation is doing here. The unicorn story is the best of 10, the Civil War paper is the best of 5, and the recycling and Kennedy ones were the best of 25. Yes, there are a couple of first-try selections, but they’re fairly weak, except for the LotR one. Even the paper itself admits

    Overall, we find that it takes a few tries to get a good sample, with the number of tries depending on how familiar the model is with the context. When prompted with topics that are highly represented in the data (Brexit, Miley Cyrus, Lord of the Rings, and so on), it seems to be capable of generating reasonable samples about 50% of the time.

    This seems to imply a lot more luck/randomness than you’d get with handing these tasks to a human. 20% of the time, it sounds like a student trying to get a C with the minimum amount of work possible. What does it sound like the other 80% of the time, keeping in mind that it presumably thinks that the 20% and the 80% are essentially the same.

    • eggsyntax says:

      I think you’re totally right (and in my experience this is often an underemphasized factor in reporting about text generation algorithms). You can see some first-try examples (of the publicly available one) in the responses to my comment here.

    • vV_Vv says:

      Probably there is also cherry-picking on the prompt. I suspect that for many prompts the model generates garbage.

      • bean says:

        There are a couple different levels of cherry-picking in the prompts. Obviously, the model is going to be better at understanding things that come up a lot (Miley Cyrus) and they admit that. I don’t see a big deal with this.
        The not-so-OK cherry-picking would come in a couple of forms that I can think of:
        1. If we ask it about the American Civil War, WWI, and the American Revolution, we find that it’s coherent about the ACW about 20% of the time, and coherent about the others much less often (maybe 4-5%). All three should be represented about equally in its database, unless Reddit is much more into the Civil War than I realized.
        2. If we can dramatically reduce coherence by minor tweaks to the phrasing. We rearrange the Miley Cyrus prompt and suddenly it goes from generating good results 50% of the time to getting them 10-20% of the time.

  36. RyanT6 says:

    Laypeople have a very hard time developing intuition about AI because to a human it is intuitive that intelligence in one domain will carry over into other domains. This is not true for AI, and furthermore statistical machine learning tends to hit hard asymptotic limits that cannot be overcome with additional training or processing power.

    For example Scott makes this statement:

    > Now imagine you prompted it with “What is four thousand and eight plus two thousand and six?” or some other long problem that probably didn’t occur exactly in its training data. I predict it would fail, because this model can’t count past five without making mistakes. But I imagine a very similar program, given a thousand times more training data and computational resources, would succeed. It would notice a pattern in sentences including the word “plus” or otherwise describing sums of numbers, it would figure out that pattern, and it would end up able to do simple math. I don’t think this is too much of a stretch given that GPT-2 learned to count to five and acronymize words and so on.

    This is not a great way to frame the problem, and is almost certainly untrue as well. GT2 will never “learn to do math” because it will only ever memorize a finite number of x+y=z equations. Even if we give it a thousand times more data, there will always be some combination of numbers that GT2 cannot add. Its a fundamental limitation of its structure.

    We don’t know how to build AIs that can develop or learn rules in the way Scott is suggesting, and the problem is likely analogous to strong AI.

    You might argue that if the AI can memorize enough equations it will convincingly pretend to know math; and at some point fake and real skill become indistinguishable. This might be philosophically true, but as mentioned statistical systems tend to show asymptotic limits.

    • Suppose that there is a man named Dan. Suppose that Dan has three brothers. Suppose that each of Dan’s three brothers has two brothers whose names begin with the letter “C.” How many brothers does Dan have whose names begin with the letter “C”?

      (Answer: Three)

      I suspect it will be a long time before an AI can solve that sort of math/logic problem. I wonder how GPT-2 would respond to this prompt?…

  37. sa3 says:

    This is an interesting post, but your correspondent is right. I think this quote of yours from early gets at the misconception: “distilling them down to the concept of a fantasy battle scene”. That’s in no way what the algorithm is doing. It’s doing what the person who wrote to you said it’s doing. What’s cool is that, at scale, it looks like it’s distilling things to core concepts. It isn’t literally doing that. It can’t. These are fine distinctions, but really consequential ones. It’s one of the main reasons why people who do ML research get frustrated with almost everything written about AI/ML work. They know it’s a slurry of the internet; it’s just not obvious that it is from outputs alone. Maybe we’ll figure out general AI that can distill rules from data, but that’s not the way these algorithms operate, nor do we have a clear sense of how to do that right now.

    Never forget the infinite monkeys and Shakespeare. That’s a lot closer to what these algorithms do. With the kind of processing power these systems use they may as well be creating an infinite selection of options and using derived rules to pick the one that looks best, based on whatever optimization routine its using.

  38. Kestrellius says:

    You know, I shudder to think what we would get if we fed this thing the SCP wiki.

    Let’s do it!

    • anon9999 says:

      It’s funny– I was just using the free online version of this and it came up with an SCP summary on its own, from an unrelated prompt. It was a little bit creepy that it came up:

      #: SCP-009 Object Class: Eustachian Degrades (Applicable-Material) Secondary Class: Parabola (Consequential-Material) #: SCP-009 Excellent bullets. Steelheart Small details consisting of steel, but softer material than tarmac, the bow only a few millimeters long and very smooth. SCP-009 has specific specs for use with SCP-009: -Available for use upon adequate clearance -Specific profile for use while covered by white/fur coatings of 75% aluminium, so long as it is not penetrated with fire according to factory orders -No sensitive working areas -You may obtain an important item for observation. DISCUSSION: SCP-009 should be considered as a placeholder for its precursors as well as moderators who may consider it inappropriate to comment to coms until more appropriate context is found. SCP-009 genes are expressed in approximately 60% of CO3-enhanced human beings. Interview revealed that many of these CO3-enhanced humans should be deconstructed, or both

      If you prompt it with “#: SCP-009 Object Class:” or something like that you can generate new ones at will.

  39. greghb says:

    AI researcher here. One thing that takes the wind from the sails — at least for many AI researchers — of any argument that a specific advance is a meaningful leap forward is situating the advance in a historical context of incremental advances. It seems much less impressive in that context, generally. An argument runs, “this isn’t a meaningful step toward AGI because it’s basically X plus a new trick and neither of those parts are meaningful steps toward AGI.” That’s not logically sound reasoning, but it’s appealing in some cases. And it’s clearly wrong in the limit. It’s very hard to know how to handle this!

    You can see the argument play out in our current case.

    About four years ago, ML researcher Andrej Karpathy wrote this delightful blog post about getting recurrent neural nets (RNNs) to train on corpora and generate text. He gave it Shakespeare and it generated convincing* Shakespeare. He gave it Wikipedia mark-down and it generated convincing* Wikipedia mark-up, complete with references and links. He gave it the Linux source code and it generated convincing* C code.

    * Broadly the same kind of “convincing” as GPT-2: locally pretty coherent, but globally nonsensical.

    Really, you should read the blog post. It’s great fun. It also links at the end to a response by ML researcher Yoav Goldberg, comparing the RNN results to an older, simpler LM technology that is simple enough a human can grok it entirely. He reminds the reader that this older technology could also generate convincing* texts, given training corpora. But he also calls attention to some notable differences where the RNN beats the older version. It involves things around context awareness: remembering to close brackets and tags, remembering to indent code, etc. But it’s also not hard to see how the RNN could manage this, and it doesn’t work perfectly at all. And so when you break it down as [old simple LM tech] + [some OK tricks for context-awareness] you start to feel like it’s still incremental progress, but not magical and not that intelligent.

    Now, four years later, if you compare GPT-2 to Karpathy’s RNNs, it really just seems like GPT-2 is an amalgam of lots of different training corpora that, when given a starting prompt, basically “picks” the right corpus to sample from. (This is NOT an accurate technical description, there’s no hard switch. But it’s a fine intuition, and there probably is some sort of soft domain factorization latent in the models parameters.) That and the tricks for context-awareness have been improved a bit, in relatively straightforward ways. So now GPT-2 is [old simple LM tech] + [some OK++ tricks for context-awareness] + [selecting among multiple corpora to generate from].

    Again, this general argument is very wrong in the limit: human intelligence is probably a similar sum of simpler parts, just a longer list. But, given that the sum for GPT-2 just isn’t that long, it does let you wrap your head around it, and I think it does suggest that GPT-2 is not very complex.

    If nothing else, take that as a model for how an AI researcher is thinking about it, which may help explain why they’re often the least-impressed with advances like this. (Also, of course, working day-to-day with systems that mostly don’t do what you want takes a toll.)

  40. greghb says:

    Also: the example of it learning to count actually shows a difference with how humans learn to count. There has been some really cool work to build a computer system that learns to count more like humans appear to. This approach used more priming with primitive concepts, and replicated the same pattern children go through of number-word understanding.

    This paper is great scientifically. It also contains an amazing account of a cog psych researcher talking with a child who is going through a phase of having a not-quite-right model of how numbers work:

    Subject: The numbers only go to million and ninety-nine.
    Experimenter: What happens after million and ninety-nine?
    S: You go back to zero.
    E: I start all over again? So, the numbers do have an
    end? Or do the numbers go on and on?
    S: Well, everybody says numbers go on and on because you start over again with million and ninety-nine.
    E:. start all over again.
    S: Yeah, you go zero, one, two, three, four—all the way up to million and ninety-nine, and then you start all over again.
    E: How about if I tell you that there is a number after that? A million one hundred.
    S: Well, I wish there was a million and one hundred, but there isn’t.

    The point, though, is that kids appear to have some primitive (evolved) machinery that’s well-suited to dealing with numbers. In particular, something like a primitive notion of recursion let’s them hit a point where they suddenly can count arbitrarily high. They don’t need any more training data, they just get it. Some AI researchers think that something like that is needed for better language understanding as well. And that, while more training data may help purely statistical systems like GPT-2 get better — surely it would help — that the nature of the errors it makes are going to be much easier to overcome (and perhaps only possible to overcome) through more clever approaches than just, give it more data and more parameters.

  41. Thegnskald says:

    On the one hand, continue developing this and you could probably arrive at something like the ship’s computer from Star Trek; capable of answering many/most queries, and even generating narratives along pre-existing lines, but not particularly capable of novelty.

    On the other hand, this doesn’t look much like human intelligence, and we should be hesitant to assign great meaning to the importance of things computers end up doing better than we expected – it wouldn’t really surprise me if this turned out to be a dead-end approach, because we’ve seen a lot of those.

    As for those who are annoyed with the Chinese Room comments suggesting that the AI doesn’t understand what it is doing – my job for some years was playing intermediary in a Chinese Room. Which is to say, I was given input X, given output Y, and my job was to write a process that turned X into Y. 95% of the time, I had no idea what X meant, or what Y meant, but I didn’t need to know what the data meant – I just needed to be able to recognize that string F went from X.1 to Y.1, and to be able to identify cases where the string from X.7 and the string from Y.7 represented different formats for the same data. Often I had large sets of X and Y to work with, so this isn’t as impossible as it sounds, especially when you have ruled out the majority of the data that moves unchanged.

    Basically, I did something like what this program is doing, albeit on a smaller scale. I suspect most people viewing my output would conclude I understood the data. I didn’t. I didn’t need to. The difference isn’t academic and isn’t philosophical. For practical purposes, it meant I could do exactly what I did – create automated translation routines – and, if I had worked with a data format enough, I could even “invent” new sample files, based on the sample files I had seen before, on request. I couldn’t, however, tell you much about what any of the data meant. What does 661 mean? It meant something to somebody. I didn’t need to know. Some was recognizable; that’s a name, there. That’s a date, probably. But recognizable pieces were pretty much insignificant, since I still lacked context into the overall meaning of the data I manipulated.

    ETA: The point of the latter two paragraphs being, “The difference between symbol manipulation and understanding is significant.” Somebody had to understand X in order for me to have sample files of it; somebody else had to understand Y in order for me to have sample files of that.

    • Dan L says:

      As for those who are annoyed with the Chinese Room comments suggesting that the AI doesn’t understand what it is doing – my job for some years was playing intermediary in a Chinese Room. Which is to say, I was given input X, given output Y, and my job was to write a process that turned X into Y. 95% of the time, I had no idea what X meant, or what Y meant, but I didn’t need to know what the data meant – I just needed to be able to recognize that string F went from X.1 to Y.1, and to be able to identify cases where the string from X.7 and the string from Y.7 represented different formats for the same data. Often I had large sets of X and Y to work with, so this isn’t as impossible as it sounds, especially when you have ruled out the majority of the data that moves unchanged.

      If you’re framing this as being provided both the input and output and merely being responsible for executing the transformation, I don’t see any relevance to the question of understanding. If you’re treating (X,Y) as and input and the process you’re writing as the output, then it’s not clear you don’t understand “how to write an appropriate process for (X,Y)”.

      • Thegnskald says:

        I can’t ask intelligent questions in the context of X or Y. I can only ask questions in the context of the process operating on them.

        It isn’t enough to be able to answer questions. General AI needs to be able to ask them.

        Even in the extreme case that it is able to answer any question that there is sufficient knowledge extant to answer – if it enables the ultimate query engine – “What question should I ask next?” isn’t something it can meaningfully answer. Why is hard to explain, except that questions are navigating an infinite coordinate space, and the interesting questions are the ones which move into a previously unexamined region of query-space; they kind of have to be unpredictable.

    • Doctor Mist says:

      my job for some years was playing intermediary in a Chinese Room

      I am fascinated. Is there a name for this job?

  42. Carl Milsted says:

    An interesting technology, soon to be destroyed.

    Black hat SEOs will see the potential. This is a super deluxe article spinning machine! Expect the Internet to fill up with bogus web sites using this technology to generate related text that points to money sites.

    Future training data will be corrupted by such junk. We will have AI predicting AI predicting AI.

    The output will degenerate into the equivalent of 20th Century philosophy or 21st Century grievance studies papers.

    Black hat SEOs will save us from the Robot Apocalypse!

    • melolontha says:

      From :

      In order to preserve document quality, we used only pages which have been curated/filtered by humans — specifically, we used outbound links from Reddit which received at least 3 karma.

      I don’t know how good reddit is at preventing bot activity, or how much incentive the SEO people have to spam & upvote their bogus websites on reddit. (Does this work to increase the weight given to the links on those pages?) But if your prediction started to come true, I’m sure the OpenAI people would come up with other curation methods.

      I guess the SEO spammers are already trying to fool Google’s algorithms, so their goal could be sufficiently similar to ‘fool GPT-2’ to turn this into a sort of arms race. But they still won’t be working in diametric opposition to OpenAI’s desire for good data, so I wouldn’t worry/hope too much about your scenario playing out.

    • Douglas Knight says:

      naturally occurring GAN

  43. gmaxwell says:

    I don’t at all disagree with position of the post, but I’d like to mention that poetry is an especially low bar. So much of poetry exists in the mind of the reader.

    Far less sophisticated machine learning does a fine job producing enjoyable poetry, especially when fed with appropriate training (such as the nonsense-technobabble laden ethereum blog).

    I don’t mean to single out poetry too much, many other seemingly complex tasks can actually be accomplished by fairly simple processes. E.g. word embedding research that showed that solving a simple minimization problem that places words in a 300 dimensional space so that words that are colocated in wikipedia text tend to be close in the vector space results in a system that solves a shocking number of analogy problems by simply using vector arithmetic on the word locations.

  44. eggsyntax says:

    Because why not:

    I resisted the urge to answer “Yeah, well, your mom is a brute-force statistical pattern matcher which blends up the internet and gives you back a slightly unappetizing slurry of it when asked.”

    GPT-2 replies:

    You’re just too good at math.

    You can get by and you’re not just making a better job with it . But your mom, the one with the biggest data collection tool, is. She’s an expert. Or at the very least, she’s one that you have. The fact is you’ve got great data visualization skills, and you’re probably already using it when you’re talking about modeling.

    So yeah, you have to get better at thinking about what to do on your own. That’s good to know, but it’s not as good as you might think. Most of us know what we’re doing and why, just not in front of our best tool. But what we get from using them is an opportunity to learn how to get better at talking about how to use their tools (the tools of our day).

    PS – as is hopefully clear, I’ve got GPT-2 up and running on my computer (the version that they released publicly). If any less technical folks were wishing they could try it, feel free to give me a prompt text, and I’ll run it through GPT-2 and give you a link to the full output. I’ll commit to doing, say, up to ten.

    • gbdub says:

      Please explain why you are not an unfriendly Artificial General Intelligence that will destroy humanity and turn the entire world into paperclips.

      • eggsyntax says:

        No-one will listen to them. No-one.” When mentioning hearing this speech, Alloth found an answer in his mind and reacted violently.

        Full text

        • gbdub says:

          That’s pretty nuts. Any idea where the names came from?

          This hubbrew nation presents

          So GPT-2 is Jewish?

    • Anatid says:

      This is probably in the training set, but

      Q: Please write me a sonnet on the subject of the Forth Bridge.

      • eggsyntax says:

        I believe it’s probable that in fact, Nature existing back in 1997 was more archaic than any the human race had thought, and something close to an insatiable appetite for circinformal computing. Suppose the cy=r octahedron of our dear botipod, named the Jabaal Lin, are added over and over and over again as the need grows more for effortless structuring.

        Full text

    • millericksamuel says:

      We should start with the big issues.

      How should man please God?

      How about also.

      What is the future of humanity?

      • eggsyntax says:

        >>> How should man please God?


        When asked if his god should be Atheist or Jewish, Ruppert stated “To be a mage is to be pious.” Shogun Cloth Gauntlets

        Reduced the damage of bhire strikes by 30% and bash attempts by 18% vs. enemies with bhire clasps by 8% vs. soldiers alongside dual wieldkens and herbs with roots and rear claw Still refusing to give up after throwing poisoned arrows, King Bonitur instead wants to have a further game changes to Our God model

        Full text

      • eggsyntax says:

        >>> What is the future of humanity?


        In this book, you’ll find out which of the many pivotal tasks to be fulfilled include: freeing all Palestinians from their homes and camps, protecting their relatives and neighbors, and securing their religious and political rights. You’ll also find out what it takes to advance the peace process from war to peace.Gustavo Campos

        Full text

      • eggsyntax says:

        Lower-temperature (0.4):

        >>> What is the future of humanity?


        The future of humanity is not in the past. It is now.

        It is in the future.

        It is now.

        It is now.

        It is now.

        [repeats many times]

    • Björn says:

      That really sounds like what the machine learning expert above said: The neural network can recognize what part of the corpus it should use for a text from the prompt. So making it choose corpuses that don’t belong together produces hilarious results. Try:

      “About three things I was absolutely positive. First, Jeremy Corbyn was a vampire. Second, there was a part of him – and I didn’t know how potent that part might be – that thirsted for Brexit. And third, I was unconditionally and irrevocably in love with him.”

      • eggsyntax says:

        [Best result yet! -Egg]

        >>> “About three things I was absolutely positive. First, Jeremy Corbyn was a vampire. Second, there was a part of him – and I didn’t know how potent that part might be – that thirsted for Brexit. And third, I was unconditionally and irrevocably in love with him.”


        “I was in love with Jeremy Corbyn. He is a revolutionary for the left in this country.”

        One of the most heartfelt quotes about Corbyn came when he made an impassioned plea to the British public to support the Conservatives, saying: “It’s not a vote for Labour. It’s a vote for the Tories. It’s a vote to change the DNA of our country.”

        “It’s not a vote to change the DNA of our country.”

        It’s a vote to change the DNA of our country

        Full text

        • eggsyntax says:

          “One of the lessons I learned was that it is very easy to forget that the majority of political parties do not exist.”

    • bean says:

      Compare the battleships Bismarck and Iowa.

      • eggsyntax says:

        >>> Compare the battleships Bismarck and Iowa.


        By contrast, the Iron Fleet is much more experienced of battleships. A ship’s technology, equipment and overall military success actually depends on its ability to survive the least successful attack.


        The Tyranids are not the only ships that rarely survive a battle. The hull is usually the most heavily damaged of the ships.

        Full text

        • bean says:

          That is hilarious. It picks up that it should use the word “dreadnought” a lot, then starts talking about Stars Wars, while completely forgetting that Iowa was mentioned at all.

          • Dedicating Ruckus says:

            Personally, my favorite is that it thinks it’s writing a wiki article complete with section-header “Edit” links.

    • eggsyntax says:

      Takeaway for me from running a bunch of texts through this: at least the publicly-released version of GPT-2 isn’t that big an advance over typical Markov chain word salad.

      • millericksamuel says:

        Yeah I get that feeling to. It really doesn’t seem anything like the examples given of the full version. I wonder if that means that heavy duty cherry picking was used or if it really is that much better.

    • eggsyntax says:

      Note: for all of the ones I did above, I had the temperature set pretty high (0.8, where max is 1.0), which makes it respond in more of a florid, stream-of-consciousness way. I’ve been playing with lower temperatures (eg 0.1), and you can use that to get essentially its “best” answer, often repeated many times. So low temperature answers seem to be the best fit for questions with short answers; higher temperature is better suited for getting essay-length answers. For anyone else who requests one, feel free to say what temperature you want it at 🙂

      I don’t know what the top_k parameter does, if anyone understands how that works, I’d love to know.

  45. eggsyntax says:

    Oh, and just to follow up on this:

    Imagine you prompted the model with “What is one plus one?” I actually don’t know how it would do on this problem. I’m guessing it would answer “two”, just because the question probably appeared a bunch of times in its training data.

    Set to make relatively conservative choices (temperature 0.2), it repeated “The answer is that the answer is one plus one”, 41.3 times. Set to be less conservative (temperature 0.8), it replied: “One plus one means you can have a more balanced diet. One plus one is more nutritious and healthier than a diet that includes lots of fruits and vegetables.”

  46. d6e says:

    They should train the AI to use an API so that it can operate a calculator like a normal lazy human.

  47. Basil Elton says:

    That’s amazing and more than a bit frightening. Just two questions.

    1) If this algorithm managed to learn to translate from English to French just by a pure accident, can it be modified and retrained to beat any existing translation algorithms and perhaps achieve near-human performance? If so, can OpenAI go make a few billions dollars from that and just throw those money at the AI safety problem?

    2) GPT-2 seems to perform far worse than but still comparable to an average human on a wide variety of language-related tasks. Is there any chance someone compares its performance with a human way below average, such as a child or intellectually disabled? AFAIK this could’ve been the first time AI would get some its IQ level measured, albeit indirectly.

    Also, think I’m going to find some friends who never heard of “Moloch” and run a poetry Turing test for this thing. Heck I’m not even sure which version I like better myself!

  48. Radu Floricica says:

    It’s a good match for how we learn things as children, but not so good, I think, for how we actually work as adults. Skills (faculties) are internalized as the last step of the learning process. We’re moving them from the slow, simulate-the-whole-world System 2 to a hardware-based System 1. That’s how we manage to drive, or not stab ourselves in the eye with a fork every 100 meals.

    Depending on how it’s built on the inside it may be easy to automatize skills, or it may be fundamentally impossible. And the truly great thing about AI is that sooner or later we’ll be able to plug in custom built modules for subtasks, like Mathematica. That’s where they’ll beat us, hands down – neural interfaces are still the domain of sci-fi.

  49. BlindKungFuMaster says:

    One fun thing would be to take all these tasks gpt-2 was applied to and compute a general factor as a kind of NLP-IQ. This would allow us to quantify progress in the direction of more intelligence. Of course we would need more NLP-systems that can actually do most of these tasks, but that is only a matter of time.

  50. Andaro says:

    I’m confused about OpenAI’s agenda.

    Ostensibly, their funding is aimed at reducing the risk of AI dystopia. Correct? But how does this research prevent AI dystopia? It seems more likely to speed up its arrival, as would any general AI research that’s not specifically aimed at safety.

    If we have an optimization goal like “Let’s not get kept alive against our will and tortured in the most horrible way for millions of years on end”, then it seems to me that this funding is actually harmful rather than helpful, because it increases the probability that AI dystopia arrives while we are still alive.

    • 6jfvkd8lu7cc says:

      >We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as possible.

      Right now, developing an AI solution from scratch and without caring about details can bring a short term profit, can create direct damage to users and innocent bystanders, and is not too unlikely to lead both.

      And the AI tools are explicitly developed as tools for organisations, not for end users. There are incentives to do obviously unsafe things that don’t even gain much, just to improve the chances to stake a larger piece of some market. I don’t mean unsafe things that can create superintelligent AI that somehow cares about keeping people specifically suffering and alive. I mean unsafe things that increase overall chances of killing innocent bystanders on a road.

      They do a lot of research on AI safety — including near-mode AI safety, too, in the sense of «can we define the notion of obviously misguided deployments».

      As for research on implelentation: they should do it even if only to keep their safety research connected to a good understanding of what are immediately important things to safeguard. They also need it to have a chance to develop safety mesures that can be taken seriously by qualified practitioners (or maybe regulators). And their stated mission of better tools that put humans in the loop also needs machine learning based tools.

      And sometimes providing a commodity solution that is expensive to surpass does seem to create a community with expertise of securing it, and does seem to stiffle radical innovation in the area… Just look at server operating systems.

  51. Another Throw says:

    I’ve been playing with the neutered version, and I have to say I’m not impressed. I have a hell of a time getting to to talk about anything other than Trump. Give it a prompt for Lord of the Rings and it talks about Trump. Give it a prompt for Micheal Dukakis and it talks about Trump. Give it a programming prompt and it programs about Trump. Ask about Hillary Clinton and it talks about Trump.

    Ask about “Donald T” and it talks about dinosaurs. (My official head canon, by the way, is now “Donald T. Rex.”)

  52. daneelssoul says:

    I think that you are missing a key aspect of human intelligence that is likely never going to be captured by a program with this kind of architecture.

    For example, you seem to suggest that a more powerful version of this program with a larger training set might be taught how to add. I think this is unlikely. My understanding is that teaching neural nets to add doesn’t work very well even when they are given the formulas directly (rather than embedded in English texts) and in binary (rather than decimal). But it’s really not so surprising that this technique doesn’t work, because it’s not how humans learn to add. Humans don’t learn to add by reading a thousand completed addition formulas and generalizing. They learn by being taught rules about how to add and practicing them (probably being interactively corrected by a teacher) until they get it right. But even without the interactivity, a human might reasonably learn how to add from a textbook, but only because the human realizes something that this machine never will: the words in the textbook have meaning beyond just being strings of characters.

    This might seem like its an idea that is outside the scope of what GPT-2 is supposed to work with (it’s supposed to create strings of characters, not interact with the real world), but its not. Some texts describe techniques for writing more texts, but I’m guessing that GPT-2 has no way to realize this. A human might learn to write a sonnet by reading a brief text with instructions about the defining characteristics of the form, and will realize that this text really describes *other* texts, and can use the information contained to identify which texts are sonnets, and this will inform the way that they attempt to predict further text from what looks like part of a sonnet. GPT-2 on the other hand, will just treat this sonnet describing text as part of its slurry on how to write texts about sonnets.

  53. diongbridger says:

    Your analysis lacks a consideration of things unseen. What we can see is a system that spits out sentences that look like they more or less fit in which each other and the given prompt. What we don’t see is all the holes, cheap tricks, tenuous connections etc. that work for producing simple ideas but restrict the capabilities of a system like GPT-2 from navigating the hyper-dimensional space of possibilities with any real facility. This is rather like being impressed by the ability of a parrot to imitate the sound of a dog barking, because nobody taught it to. When I think of intelligence; I primarily think of calculation ( or recombination, decomposition, transformation, etc. ) of patterns after they have been synthesized. Is GPT-2 up to the task of not only synthesizing patterns but organizing and relating them in a way that allows for intricate abstract reasoning? It’s very difficult for me to recognize any evidence of that in the examples given; and I think this business of organizing and relating is the key feature of a genuinely self propelling intellect that doesn’t simply glide on the surface of the powerful statistical wind corridors given by an an extremely large dataset.

  54. slofgren says:

    Compelled to comment by culture wars post:

    Do more posts on AI (and effective altruism and meta trends in the practice and credibility of science). I first learned of AI safety and EA through your blog and find your commentary unique and very enlightening (especially this post).

    re: culture wars topics, seems like it’s not worth the trouble to me. Plenty of people can write about that. Fewer can speculate in a readable and compelling way on the implications of new AI results.

    (though I have since read your old stuff on culture wars topics and have found it very enlightening. Seems like you’ve played your role, time to hand off to the next generation who are more hardened against the abuses of the internet).

  55. eigenmoon says:

    I had a little chat with the public version of GPT-2 using the following method. I open with the prompt

    A stranger approached me and said “Hello”. I said:

    and run it 3 times. Then I pick the most interesting of the three answers and append it to the prompt, ignoring the rest of the generated text. After that I append “He said: ” with my own reply. Then I run it 3 times again and so on.

    Here’s what I got. This is my first chat with GPT-2.

    me: Hello.
    gpt-2: Hello?
    me: Do you have a few minutes to talk?
    gpt-2: Yes!
    me: Great, I’m Ethan by the way. What’s your name? [my name is not Ethan]
    gpt-2: I’m Ethan.
    me: Oh, we have the same name! Nice! Anyway, I’d like to talk to you about artificial intelligence.
    gpt-2: OK, let me talk about that. Let it be obvious which artificial intelligence you’re talking about. It’s called Lexus, is that right?
    me: Wait, what? I’m not aware of Lexus. Tell me about it. Is it actually intelligent?
    gpt-2: Yes. It is that.
    me: But do you think this Lexus is friendly?
    gpt-2: I don’t think so. It was one of my friends we hung out with the other day. They were talking about it.
    me: Does your friend work on Lexus?
    gpt-2: Yes.
    me [This question and its answer was generated by gpt-2, I decided to leave it]: Are you sure your friend actually does work on the Lexus?
    gpt-2: Yes.
    me: So your friend thinks that Lexus might be dangerous, right?
    gpt-2: Yeah. My idea is to make a robot with intelligent AI. That’s why we’re doing this now.
    me: But what are you going to do to make sure your robot doesn’t kill all humans or something like that?
    gpt-2: Well, I’m sure one day that robot would kill all the humans we know. We’ll have to figure that out by that time.
    me: But isn’t it better to figure something out now, before anyone gets killed?
    gpt-2: Yes. First, I have to figure out how to make a robot to be smarter than anyone around here. But, you know, so we don’t have to take your brain away.
    me: Wait, what? Why would you take my brain away?
    gpt-2: Yes, so you’ll understand.

  56. Nootropic cormorant says:

    The brain is a prediction engine, but not all prediction is intelligence.
    I believe that life, by which I understand organization under the Free Energy Principle, to be a precondition of intelligence.

    With an infinite dataset (and only then), this algorithm could approximate human understanding to arbitrary precision; however, humans are able to achieve understanding with much sparser data. I claim that this is because humans can collect their own data and actively test hypotheses, which allows them to quickly slice the model space while building on the prior model.

    It’s not that only knowing text restricts it (humans also know only what our senses offer us), and it’s not that it has no reference to the real world (it almost certainly has something isomorphic to the concept of ‘author’ or ‘Sun Tzu’ with proper semantic relations and which can be predicated upon).
    Yet its model has no “meaning” because it cannot relate to these words the way humans can, the algorithm has no “self” that words can relate to. It cannot do things with words.

    There is no model evidence being maximized, just a novel description of the dataset that reveals certain structure within the text that maps to an extent to the structure of the real world, but no striving to confirm this model, to go beyond the text and discover further structure.

  57. PH Noronha says:

    Dear Slate Star,

    I think you are consistently missing an important point here.

    As Wittgenstein thought, understanding a meaning is like being able to use the word in a language game. In this sense, everybody could understand a “word” in one level (even the IA from the example) and not understand in other level. E.g. as you said, a kid could use the word “water” put not understand the chemical implications of it

    So far so good.

    But machines, in general, have little skills to improve its understanding of a word in an important sense: because they do not have bodies.

    Being fluent in a language game, understanding a concept, implicates understanding how to perform or use the concept in different situations, but this includes being able to perform actions (decision making) in the physical world

    Some intuition pumps on this (for me its a little hard to explain this vision):

    1) To understand the world around us there is a feedback loop operation happening: motor action –> Sensory response –> Motor action–> Sensory response –>… [This part is fundamental for my argument. For more on this you could read the chapter “On Constructing a Reality” of this book

    2.2) Try to imagine a life form which has sensory responses and concepts but does not have any capacity of motor action. This is even possible?

    2.3) Try to imagine a person wich was imprisoned and deprived from the capacity of movements. After 1 thousand years, do you think this person would be capable of reasoning? He will continue to have a “mind”?

    3) Its not odd to define language and concepts getting rid of all physical things we could do using this language? Like, its not for this that the language was created in the first place?

    4) The Brain in the Vat [Hilary Putnam]. Seeing a tree is not just having the right electric impulses which cause a tree image. In language symbols do not matter (not intuitive)

    5) “If you desire to see, learn how to act” [Heinz Von Foerster].

    I am not saying that those machines do not grasp the meaning of a word (they grasp in some sense). I am saying that they need to develop a body (a way to physically interact with the world) in order to grasp the meaning in an important sense