Add scale map to apply_max_norm_regularization

[rockerBOO]

Allows setting the scale_weight_norms via

toml

scale_weight_norms = 2.5
scale_weight_norms_map = { "*ff_net_*_proj*" = 5.0 }

CLI (formats it as a string and we convert it into a dict)

--scale_weight_norms 2.5
--scale_weight_norms_map "{ '*ff_net_*_proj*' = 5.0 }"

Set --scale_weight_norms 5.0 to some value for the other values and the map would set those values for matching module keys.

This is is just an idea to allow us to scale the weight norms differently for different modules. The implementation details are up in the air because I'm not sure what would be best here.

Could be that we want to use regex for the module name matching. Or make the map work exclusively to scale_weight_norms.

See https://github.com/rockerBOO/sd-ext/blob/main/scale_norms.py which emulates the same behavior but on the static weights. You can try different strategies with the map.

[recris]

This is something I wanted to implement for some time.

Maybe match the names using fnmatch style patterns? Slightly simpler than regex.

[rockerBOO]

Yeah fnmatch seems to be a good fit. I will give it a try.

[rockerBOO]

Added fnmatch. It is only fnmatch so will need the * where you want it to be matching. Note it will be including .lora_down in the name so if it's at the end of the base name you'll want to add the * at the end.

toml

scale_weight_norms = 2.5
scale_weight_norms_map = { "*ff_net_*_proj*" = 5.0 }

CLI (formats it as a string and we convert it into a dict)

--scale_weight_norms 2.5
--scale_weight_norms_map "{ '*ff_net_*_proj*' = 5.0 }"

Also only supports vanilla LoRA. I am wanting to add more logging/reporting to better get what keys are being scaled and to what, and what ratio. Lots of metrics to help showcase what it's doing as it is a little buggy with reporting for some reason.