Release v1.0.12

Nov 28, 2024

• More optimizers
  • Add MARS optimizer (https://arxiv.org/abs/2411.10438, https://github.com/AGI-Arena/MARS)
  • Add LaProp optimizer (https://arxiv.org/abs/2002.04839, https://github.com/Z-T-WANG/LaProp-Optimizer)
  • Add masking from 'Cautious Optimizers' (https://arxiv.org/abs/2411.16085, https://github.com/kyleliang919/C-Optim) to Adafactor, Adafactor Big Vision, AdamW (legacy), Adopt, Lamb, LaProp, Lion, NadamW, RMSPropTF, SGDW
  • Cleanup some docstrings and type annotations re optimizers and factory
• Add MobileNet-V4 Conv Medium models pretrained on in12k and fine-tuned in1k @ 384x384
  • https://huggingface.co/timm/mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k
  • https://huggingface.co/timm/mobilenetv4_conv_medium.e250_r384_in12k
  • https://huggingface.co/timm/mobilenetv4_conv_medium.e180_ad_r384_in12k
  • https://huggingface.co/timm/mobilenetv4_conv_medium.e180_r384_in12k
• Add small cs3darknet, quite good for the speed
  • https://huggingface.co/timm/cs3darknet_focus_s.ra4_e3600_r256_in1k

Nov 12, 2024

• Optimizer factory refactor
  • New factory works by registering optimizers using an OptimInfo dataclass w/ some key traits
  • Add list_optimizers, get_optimizer_class, get_optimizer_info to reworked create_optimizer_v2 fn to explore optimizers, get info or class
  • deprecate optim.optim_factory, move fns to optim/_optim_factory.py and optim/_param_groups.py and encourage import via timm.optim
• Add Adopt (https://github.com/iShohei220/adopt) optimizer
• Add 'Big Vision' variant of Adafactor (https://github.com/google-research/big_vision/blob/main/big_vision/optax.py) optimizer
• Fix original Adafactor to pick better factorization dims for convolutions
• Tweak LAMB optimizer with some improvements in torch.where functionality since original, refactor clipping a bit
• dynamic img size support in vit, deit, eva improved to support resize from non-square patch grids, thanks https://github.com/wojtke

Oct 31, 2024

Add a set of new very well trained ResNet & ResNet-V2 18/34 (basic block) weights. See https://huggingface.co/blog/rwightman/resnet-trick-or-treat

Oct 19, 2024

• Cleanup torch amp usage to avoid cuda specific calls, merge support for Ascend (NPU) devices from MengqingCao that should work now in PyTorch 2.5 w/ new device extension autoloading feature. Tested Intel Arc (XPU) in Pytorch 2.5 too and it (mostly) worked.

What's Changed

• mambaout.py: fixed bug by @NightMachinery in #2305
• Cleanup some amp related behaviour to better support different (non-cuda) devices by @rwightman in #2308
• Add NPU backend support for val and inference by @MengqingCao in #2109
• Update some clip pretrained weights to point to new hub locations by @rwightman in #2311
• ResNet vs MNV4 v1/v2 18 & 34 weights by @rwightman in #2316
• Replace deprecated positional argument with --data-dir by @JosuaRieder in #2322
• Fix typo in train.py: bathes > batches by @JosuaRieder in #2321
• Fix positional embedding resampling for non-square inputs in ViT by @wojtke in #2317
• Add trust_remote_code argument to ReaderHfds by @grodino in #2326
• Extend train epoch schedule by warmup_epochs if warmup_prefix enabled by @rwightman in #2325
• Extend existing unit tests using Cover-Agent by @mrT23 in #2331
• An impl of adafactor as per big vision (scaling vit) changes by @rwightman in #2320
• Add py.typed file as recommended by PEP 561 by @antoinebrl in #2252
• Add CODE_OF_CONDUCT.md and CITATION.cff files by @AlinaImtiaz018 in #2333
• Add some 384x384 small model weights by @rwightman in #2334
• In dist training, update loss running avg every step, sync on log by @rwightman in #2340
• Improve WandB logging by @sinahmr in #2341
• A few weights to merge Friday by @rwightman in #2343
• Update timm torchvision resnet weight urls to the updated urls in torchvision by @JohannesTheo in #2346
• More optimizer updates, add MARS, LaProp, add Adopt fix and more by @rwightman in #2347
• Cautious optimizer impl plus some typing cleanup. by @rwightman in #2349
• Add cautious mars, improve test reliability by skipping grad diff for… by @rwightman in #2351
• See if we can avoid some model / layer pickle issues with the aa attr in ConvNormAct by @rwightman in #2353

New Contributors

• @MengqingCao made their first contribution in #2109
• @JosuaRieder made their first contribution in #2322
• @wojtke made their first contribution in #2317
• @grodino made their first contribution in #2326
• @AlinaImtiaz018 made their first contribution in #2333
• @sinahmr made their first contribution in #2341
• @JohannesTheo made their first contribution in #2346

Full Changelog: v1.0.11...v1.0.12