![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
"Differentiable Finite State Machines"
Jun. 8th, 2022 12:37 pmAlex Mordvintsev (known for DeepDream and more recently for beautiful Neural Cellular Automata and Self-Organizing Textures) created this cool tutorial:
google-research.github.io/self-organising-systems/2022/diff-fsm/
"how differentiable optimization can be used to learn Finite State Machines (FSM) for solving toy string processing tasks"
"how simple regularization and initialization techniques can steer continuous optimization towards finding discrete deterministic solutions"
"experiments shown here may have some educational value, e.g. in demonstrating less conventional (and perhaps unexpected) uses of differentiable programming and some elegant JAX tricks."
He introduces two techniques to "sparsify" the system (to reduce the size of the state machine): penalty for entropy and addition of identity transform.
:-) I quote-retweeted a summary of this, and Alex retweeted my tweet (I felt honored by that), and that tweet of mine went "semi-viral" as a result :-)
google-research.github.io/self-organising-systems/2022/diff-fsm/
"how differentiable optimization can be used to learn Finite State Machines (FSM) for solving toy string processing tasks"
"how simple regularization and initialization techniques can steer continuous optimization towards finding discrete deterministic solutions"
"experiments shown here may have some educational value, e.g. in demonstrating less conventional (and perhaps unexpected) uses of differentiable programming and some elegant JAX tricks."
He introduces two techniques to "sparsify" the system (to reduce the size of the state machine): penalty for entropy and addition of identity transform.
:-) I quote-retweeted a summary of this, and Alex retweeted my tweet (I felt honored by that), and that tweet of mine went "semi-viral" as a result :-)
