Page attention - applicable only the first transformer

[etam89]

KV values are cached and looked up by token in page attention. Input to the first transformer are words. But input to other transformers are not vocabulary related. Those vectors are not limited in numbers. wouldn't page attention technique can be applied to the first transformer?

thanks!