Entry tags:
MSML21: Mathematical and Scientific Machine Learning
Starts in 5 minutes:
msml21.github.io/
No registration is needed - they are just handling it in a relaxed fashion
msml21.github.io/
No registration is needed - they are just handling it in a relaxed fashion
no subject
A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms, Chao Ma (Princeton University), Lei Wu (Princeton University), Weinan E (Princeton University)
Paper Highlight, by Pankaj Mehta
The paper connects the continue-time limits of adaptive gradient descent methods, RMSProp and Adam, to the sign gradient descent algorithm and explores three types of typical phenomena in these adaptive algorithms’ training processes. By analyzing the signGD flow, this paper explains the fast initial convergence of these adaptive gradient algorithms with a learning rate approximating 0 and fixed momentum parameters. The connection, the convergence analysis, and experiments on verifying the three qualitative patterns are original and technically sound.