"Differentiable Finite State Machines"
Jun. 8th, 2022 12:37 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
dmmAlex Mordvintsev (known for DeepDream and more recently for beautiful Neural Cellular Automata and Self-Organizing Textures) created this cool tutorial:
google-research.github.io/self-organising-systems/2022/diff-fsm/
"how differentiable optimization can be used to learn Finite State Machines (FSM) for solving toy string processing tasks"
"how simple regularization and initialization techniques can steer continuous optimization towards finding discrete deterministic solutions"
"experiments shown here may have some educational value, e.g. in demonstrating less conventional (and perhaps unexpected) uses of differentiable programming and some elegant JAX tricks."
He introduces two techniques to "sparsify" the system (to reduce the size of the state machine): penalty for entropy and addition of identity transform.
:-) I quote-retweeted a summary of this, and Alex retweeted my tweet (I felt honored by that), and that tweet of mine went "semi-viral" as a result :-)
google-research.github.io/self-organising-systems/2022/diff-fsm/
"how differentiable optimization can be used to learn Finite State Machines (FSM) for solving toy string processing tasks"
"how simple regularization and initialization techniques can steer continuous optimization towards finding discrete deterministic solutions"
"experiments shown here may have some educational value, e.g. in demonstrating less conventional (and perhaps unexpected) uses of differentiable programming and some elegant JAX tricks."
He introduces two techniques to "sparsify" the system (to reduce the size of the state machine): penalty for entropy and addition of identity transform.
:-) I quote-retweeted a summary of this, and Alex retweeted my tweet (I felt honored by that), and that tweet of mine went "semi-viral" as a result :-)
no subject
Date: 2022-06-08 04:54 pm (UTC):-) I am sure all those likes, retweets, and follows resulting from this are there mainly because Alex has retweeted this :-)
:-) I guess I have written a good "tweet summary" there :-)
no subject
Date: 2022-06-08 05:01 pm (UTC)https://twitter.com/zzznah/status/1532395703527415809
no subject
Date: 2022-06-08 08:08 pm (UTC)Oh how interesting!