@mehtadushy thanks, I'll take a look at this next week

@mehtadushy Looks good, compelling GPU resource utilization for accuracy levels. I'm going to merge as is and then tweak a few style/consistency things myself (lower case model strings/entry point fns, etc).

I downloaded your weights files, I plan to add the hash to the filename and host a copy in a separate GitHub release within this repo that mentions the origin so that it works with the model zoo downloader. Just like HRNet, Res2Net (https://github.com/rwightman/pytorch-image-models/releases) Is that okay?

Yes, that would be ok as long as a note refers back to the original repository (https://github.com/mehtadushy/SelecSLS-Pytorch) for license terms, and to the paper details (https://arxiv.org/abs/1907.00837) for citation.

@mehtadushy okay, done... it's on the master now and created a release with the requested links and info for the weights. I made a few changes to bring in line with some naming prefs. Checkpoint compatibility maintained.

https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-selecsls

Thanks!

@mehtadushy Do you happen to have any of the hparams used for training the SelecSLS? I was going to run some experiments with them for some new augmentations since they're faster throughput/bigger batch then the my ResNet usuals, but my first pass with my usual LR and hparams for ResNet wasn't quite as good.

I trained them a long time ago, and did not seem to have saved the exact hyperparameters used, but the following ones which I have been using for recent related experiments are very close.

First train with RMSProp for around 100 epochs
CUDA_VISIBLE_DEVICES=0,1 ./distributed_train.sh 2 <dataset/path> --model <model_name> --sched step --epochs 150 --warmup-epochs 7 --lr 0.011 --opt rmsproptf --opt-eps 0.001 --decay-rate 0.92 --decay-epochs 3 --min-lr 5e-5 --batch-size 256 -j 16 --reprob 0.4 --remode pixel --amp --output ../Pytorch/logs/imagenet_selesls_fancy/ --model-ema --model-ema-force-cpu --color-jitter 0.2 (I initialized this with a checkpoint from one of my previous attempts with SGD, but I think should be ok without an initial checkpoint. If there is an initial checkpoint, then --warmup-epochs 0 --lr 0.01)

I don't let the RMSProp training run through the entire epoch range, and stop it close to 100, and have SGD with a smaller batch size do the tail end of training. I let this run for around 30 epochs
CUDA_VISIBLE_DEVICES=1 ./distributed_train.sh 1 <dataset/path> --model --sched cosine --epochs 40 --warmup-epochs 0 --lr 1e-3 --min-lr 5e-5 --batch-size 256 -j 16 --reprob 0.4 --remode pixel --amp --output . ./Pytorch/logs/imagenet_selecsls_fancy/ --model-ema --model-ema-force-cpu --initial-checkpoint <path_to_checkpoint_from_rmsproptf_run> --color-jitter 0.1

Then for the EMA part to have a slightly more diverse history, I lower the batch size further, and run for around 10 epochs (I stop at 10 even though this runs for 20)
CUDA_VISIBLE_DEVICES=0 ./distributed_train.sh 1 <dataset/path> --model --sched cosine --epochs 10 --warmup-epochs 0 --lr 1e-4 --min-lr 5e-5 --batch-size 128 -j 16 --reprob 0.2 --remode pixel --amp --output ../Pytorch/logs/imagenet_selecsls_fancy/ --model-ema --initial-checkpoint <path_to_checkpoint_from_sgd_run> --color-jitter 0.05

In addition to EMA, I also used SWA with a couple of runs with adam, starting from the EMA and non EMA weights of the previous run, but I don't remember the exact details of that, and have not been using it for my recent experiments. Without this, you should be able to get within 0.2 of the reported top-1 performance.

@mehtadushy thanks for the details, I try something along those lines and see what I get.