ScheduleFree optimizer wrapper

[Vectorrent]

The Road Less Scheduled

It would be nice to see a generic version of the Schedule-Free wrapper, which can be used in-lieu of a cosine annealing strategy. Facebook offers some nice reference implementations, though the generic one requires some coordinated calls to train() and eval(), which can make it a bit tricky to implement. There are closure-based versions to circumvent this limitation, but sadly - nobody has integrated that into the generic wrapper version, yet.

Maybe we could do that, over here? 😉