Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

While its more token costly, the tokenization of numbers gets weird.

Taking "123" it becomes [123]. "1234" becomes [12][34]. "12345" becomes [123][45].

Consider using spaces between each digit to force them to be distinct tokens.

"1 2 3 4 5" is tokenized as [1][ 2][ 3][ 4][ 5]

It may then be possible to work with the digits more easily because they're not getting tokenized in potentially odd ways that are harder to manipulate.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: