OpenAI code generation breakthrough

[dmm]

In this video Microsoft CTO is interviewing OpenAI CEO starting from 25:00 mark (right before this mark he is talking about a huge computer system Microsoft created for OpenAI; the style of this overall Microsoft video does feel quite weird to my taste, but this fragment with Sam Altman is good):

twitter.com/matvelloso/status/1263193089310461952

At about 29:00 mark OpenAI demos their new transformer-based code-generating system trained on a large subset of GitHub. I'd say, it's quite impressive, it does feel like a breakthrough in coding-assisting tools. Some discussion here:

news.ycombinator.com/item?id=23250379

Generally speaking, people are saying lately that large modern transformer models only pretend to be sequence-to-sequence, but in reality they learn tons of structured linguistic information, see e.g. this informal essay-style paper and references therein:

arxiv.org/abs/2005.06420 "The Unstoppable Rise of Computational Linguistics in Deep Learning"

(This is not yet a artificial junior software engineer one can hire, but this OpenAI prototype is a considerable step in that direction. May 20, 2020 will be remembered as an important milestone.)

Comments

[dmm]

Speaking about structured linguistic information captured by transformers, prominent papers include:

https://arxiv.org/abs/1905.09418 "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned" (this paper is led by people from Yandex)

https://arxiv.org/abs/1905.05950 "BERT Rediscovers the Classical NLP Pipeline" (I've seen this one earlier)

https://arxiv.org/abs/1906.02715 "Visualizing and Measuring the Geometry of BERT" (this looks very interesting)

There are also dozens of references to these papers at this point (58, 75, and 19 references respectively, counted by Google Scholar); so there must be further analysis in the literature.

[dmm]

7 weeks later: 95, 144, and 32 references respectively.

[dmm]

Two and a half months further down the road: 137, 202, and 41 references respectively.

[dmm]

4 more months: 196, 297, and 58 references.

The ""Language Models are Few-Shot Learners" itself: 591 references (I expected more at this point).