Support for BTX (Branch-Train-MiX) merge method? #432
MoonRide303
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Few days ago I've noticed a paper from FAIR / Meta describing method of merging multiple fine-tuned models to achieve superior cumulative performance: Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM (aka BTX).
Results look pretty good:
Formula for mixing is described in part "3.2 MiX: Combining Separate Experts to be a Mixture-of-Experts". This method (or some variation of it) might be compatible with SD / SDXL / SD3 models, and if so - it could be a nice feature in supermerger.
Beta Was this translation helpful? Give feedback.
All reactions