[Question] What is the memory footprint of te.Linear() weights ?

[vince62s]

Does each parameter take 1 byte, 2 bytes, 4 bytes ?

it does not seem so clear in terms of memory efficiency.

[ptrendx]

Currently the FP8 weights are only internal and so the actual model weights take the same amount of memory as without FP8 execution (e.g. 2B for FP16+FP8 training). We are working together with Meta on exposing FP8 tensors in pyTorch, which will enable storing only the FP8 weights, resulting in memory savings over the base model as well as e.g. faster communication in FSDP, but it is currently in the PoC stage.