CUDA out of memory - minimum gpu size #167

Activity

1. BIGBALLON commented on May 13, 2024

   

   BIGBALLON

   on May 13, 2024

   More actions

   @yscoffee The total number of model parameters is 25.5B, consisting of the architecture of InternViT-6B-448px-V1-5 + MLP + InternLM2-Chat-20B.

   • For bf16, at least 52GB of GPU memory is required.
   • For int8, at least 26GB of GPU memory is needed. There is also some additional overhead. so 32GB memory is needed.

   Examples, if you can understand Chinese, you can refer to this article:

   • 1 GPU for int8 model:
   • 2 GPU for bf16 model:
   • 4 GPU for bf16 model:
2. yscoffee commented on May 13, 2024

   

   yscoffee

   on May 13, 2024

   Author

   More actions

   thank you for the detail explanation!
3. yscoffee

   closed this as completedon May 13, 2024