RuntimeError when resuming training

[piedrahitacarol]

Hi, I was training on my own dataset, but when I run the code train.py with --resume option I got this error:

Traceback (most recent call last):
File "tools/train.py", line 239, in
trainer.train()
File "tools/train.py", line 147, in train
self.optimizer.step()
File "/home/cpiedrahita/anaconda3/envs/segmentron/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 66, in wrapper
return wrapped(*args, **kwargs)
File "/home/cpiedrahita/anaconda3/envs/segmentron/lib/python3.6/site-packages/torch/optim/sgd.py", line 106, in step
p.data.add_(-group['lr'], d_p)
RuntimeError: value cannot be converted to type float without overflow: (6.33039e-07,-2.05687e-07)

My environment: python 3.6, pytorch 1.4, cuda 10.1

Thanks!

[lhy118118]

Hello, I have met the same problem and do you solve it? Thank you very much!

[piedrahitacarol]

Hi, I didn't solve it sorry