OpenAI: Improvements in algorithmic efficiency of neural networks

openai.com/blog/ai-and-efficiency/

arxiv.org/abs/2005.04305

"Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both."

Mathematical Foundations of Programming Semantics: June 2-6

Online, free: www.monoidal.net/paris2020/mfps/

Jointly with "Quantum and Physics Logic" conference.

***

The last time I attended MFPS was, I think, in May 1998 (London) www.eecs.qmul.ac.uk/~edmundr/mfps/, where I gave a talk on our joint work with Sveta Shorina, "Relaxed Metrics, Maximal Points, and Negative Information". The feedback from that conference led to our follow-up joint paper, "On a Smyth Conjecture", topology.auburn.edu/tp/reprints/v24/tp24203.pdf

These results became Chapter 12 of my 2002 dissertation, "Mathematics of Domains", arxiv.org/abs/1512.03868 ; and a few years later I made dark T-shirts with Figure 6.2 (page 107 (page 122 of the PDF file); this T-shirt can still be printed from cafepress site; there were two reasons to design and print this T-shirt: to establish that I know how to print a dark T-shirt (it's a bit tricky, one needs transparent background, otherwise quality would suck) and to remind myself of the part of this work which was most difficult to keep understanding (section 12.4.3 explaining how Stone duality manifests in this case)). This still tends to be my most standard T-shirt (I printed a lot of them over the years, in various colors, but mostly in black).