Two events next weekend

[dmm]

An online talk on efforts towards practical use of AI for assistance in software engineering at Google (this is California time):

www.meetup.com/Scala-Bay/events/271129752/

("Towards an ML-augmented Programming Stack" by Eugene Kirpichov. A slide deck is attached to the event page, the key slides are slides 11-13.)

An online demoparty (this is Boston time):

atparty-demoscene.net/

One can just watch, or one can take part in various competitions.

Comments

[dmm]

Speaking of the demoparty competitions, http://atparty-demoscene.net/competitions/ , the rules for the classical Demo Compo allow the following in additional to the more traditional self-contained directory with an executable: "Javascript / WebGL demos running in a fresh install of Chrome or Firefox with no add-ons are also acceptable".

So people are free to try various Javascript / WebGL tools, such as cables.gl:

https://github.com/cables-gl
https://cables.gl/
https://docs.cables.gl/
https://cdm.link/2019/01/magical-3d-visuals-patched-together-with-wires-in-browser-cables-gl/

[dmm]

I want to meditate a bit about machine learning tools for software engineering.

First of all the message of the upcoming talk: stop focusing on fragile "state-of-the-art benchmarks", focus on simple scalable solutions which "just work" for many problems across the board, reliably, and without manual tweaking and fine-tuning, which don't need much training data, and can deal on their own with drift in data characteristics (that's what "nonstationary" means in that slide deck).

So, basically, start bringing all this down from "research quality" to the level of normal, robust traditional software technology.

[dmm]

Then there is another angle. On one hand, a purely linguistic "Transformer" model (which does not know anything about programming or about what's going on inside a computer) was making very nice guesses in the recent milestone OpenAI demo ("AI breakthrough of the year" so far, I would say): https://dmm.dreamwidth.org/27542.html

Then there is a new MIT "DreamCoder" work, which knows much more about programming and about what's going on inside a computer:

https://web.mit.edu/ellisk/www/documents/dreamcoder_with_supplement.pdf

https://github.com/ellisk42/ec

Still, because this is "programming by example" (inductive inference of a program from specified test results), one can only think in terms of probabilities that the solution will be good (and that's what "DreamCoder" does). Unless one wants to generate programs according to formal specifications, a degree of uncertainty will remain. And writing formal specification is essentially programming in a slightly different very formal declarative language (so if one wants to automate programming from examples and/or from linguistic hints, a degree of uncertainty seems to be inevitable; in the case of manual software engineering or manual writing of formal specs this uncertainly is also present ("how one would know that a person wrote the formal specs 'correctly'?")).

[dmm]

So, even in the case of purely linguistic OpenAI model, this should be somehow grounded inside a system which knows about computers and which can compare that different things yield results consistent to each other.

For an OpenAI system, the way this is done, first of all it is used to generate various unit tests and other tests (which are prime candidates for automated code-generation anyway, since no algorithmic difficulties are involved). And then one can actually run tests to validate the generated software or to find bugs.

***

On the other hand, none of these systems come close to solving problems of finding non-trivial algorithms (of learning to worry about algorithmic complexity).

But then, it is not so frequent that people specify algorithmic complexity in the comments or in the tests (as long as it's inside an engineer's mind, but not written down, and tested only to the extent of "being not too slow", we would not even have data to feed into a computer system for it to learn; alternatively, the computer systems would need to learn to understand mathematical human-written texts on a more serious level than a Transformer does; we read a textbook, and that's how we can learn to reason about algorithmic complexity; we are not yet close to a computer system which would be able to read a textbook and learn to reason about algorithmic complexity from that).

[dmm]

> we are not yet close to a computer system which would be able to read a textbook and learn to reason about algorithmic complexity from that

is that correct, actually?

Look at this paper from Allen Institute for AI:

https://arxiv.org/abs/2002.05867

"Transformers as Soft Reasoners over Language"

'This paper investigates a modern approach to this problem where the facts and rules are provided as natural language sentences, thus bypassing a formal representation. We train transformers to reason (or emulate reasoning) over these sentences using synthetically generated data. Our models, that we call RuleTakers, provide the first empirical demonstration that this kind of soft reasoning over language is learnable, can achieve high (99%) accuracy, and generalizes to test data requiring substantially deeper chaining than seen during training (95%+ scores). We also demonstrate that the models transfer well to two hand-authored rulebases, and to rulebases paraphrased into more natural language. These findings are significant as it suggests a new role for transformers, namely as limited "soft theorem provers" operating over explicit theories in language.'

perhaps, we might be closer than it seems...