I wanted functionality where GPU VRAM isn't constantly hogged while I'm serving a PyTorch model, so that I could simultaneously train stuff.
I wanted a solution which was agnostic to the type of the model, with respect to loading and inferring.
So I made this AutoUnloadModel class that unloads the model if it hasn't been used for some period. I used __init_subclass__ to ensure that all the details regarding timers, locks etc are hidden from the subclass.
I found __init_subclass__ very cool for this job, which is the reason I'm sharing this. Thanks!
[+] [-] matroid|2 years ago|reply
I wanted a solution which was agnostic to the type of the model, with respect to loading and inferring.
So I made this AutoUnloadModel class that unloads the model if it hasn't been used for some period. I used __init_subclass__ to ensure that all the details regarding timers, locks etc are hidden from the subclass.
I found __init_subclass__ very cool for this job, which is the reason I'm sharing this. Thanks!