The Price of Token Boundaries: Compression Certificates and Prediction
arXiv:2609.35869v1 Announce Type: new Abstract: Pre-tokenisation restricts which text fragments can become prediction units, but its compression cost is obscured when tokenisers are compared only under the same boundaries. We measure this...
Read more: https://arxiv.org/abs/2609.35869
Scan with your phone
Scan with your phone