Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
H8crilA
on April 6, 2023
|
parent
|
context
|
favorite
| on:
OpenAI Tokenizer
This works really poorly in non-latin scripts. Try pasting "Україна" (Ukraine) or "北京是中国的首都" (Beijing is the capital of China). I'm a little surprised that nobody optimized that, there must be enough training data to warrant this effort.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: