To understand the Project DIGITS desktop (128 GB for 3k), look at the existing Grace CPU systems

There seems to be a lot of confusion about how Nvidia could be selling their 5090 with 32GB of VRAM, but their Project Digits desktop has 128 GB of VRAM.

Typical desktop GPUs have GDDR which is faster, and server GPUs have HBM which is even faster than that, but the Grace CPUs use LPDDR (https://www.nvidia.com/en-us/data-center/grace-cpu/), which is generally cheaper but slower.

For example, the H200 GPU by itself only has 96/144GB of HBM, but the Grace-Hopper Superchip (GH200) adds in an additional 480 GB of LPDDR.

The memory bandwidth to this LPDDR from the GPU is also quite fast! For example, the GH200 HBM bandwidth is 4.9 TB/s, but the memory bandwidth from the CPU to the GPU and from the RAM to the CPU are both around 500 GB/s still.

It's a bit harder to predict what's going on with the GB10 Superchip in Project Digits, since unlike the GH200 superchips it doesn't have any HBM (and it only has 20 cores). But if you look at the Grace CPU C1 chip (https://resources.nvidia.com/en-us-grace-cpu/data-center-datasheet?ncid=no-ncid), there's a configuration with 120 GB of LPDDR RAM + 512 GB/s of memory bandwidth. And the NVLink C2C bandwidth has a 450GB/s unidirectional bandwidth to the GPU.

TL;DR: Pure speculation, but it's possible that the Project Digits desktop will come in at around 500 GB/s memory-bandwidth, which would be quite good! Good for ~7 tok/s for Llama-70B at 8-bits.