Save training state every N epochs

[williamkmlau]

Hi, right now we can save training state, but it forces it to save every epoch.
These training states are big, it consumes a lot of time to save and wastes a lot of space.
But for high epochs it likely requires resuming, but saving every epoch is way too inefficient.
Can we get a feature to save only every N epochs?

[DKnight54]

Actually, the training state is saved at the same frequency as the safetensor files are generated.
For example, here I have set it to save the safetensor every 2 epochs, and if I select the save training state, it'd save the training states at every N epoch as well.

However, I also wonder if there is additional benefits from doing this? as during training of Loras, I found that I could resume training by selecting the previously generated loras (a couple of hundred megabytes instead of 4+gb!). That being said, do take the precaution of moving your loras/training states out of the output folder before resuming, since the script will always restart numbering from 1 and try to overwrite (and fail if the model/training state is loaded).