Tokenizer: watch text become tokens
Type any text below. Switch between character-level and word-level tokenization and watch how the pieces — and the token count — change.
0 tokens (words)
characters: 0
words: 0
ratio: – tokens/word
word / character
space
has punctuation