Implement a **basic whitespace tokenizer** — the foundation of NLP preprocessing.
1. Lowercase the entire string
2. Remove punctuation: `. , ! ? ; : " '`
3. Split on whitespace
4. Remove empty strings
tokenize("Hello, World!") → ["hello", "world"]
tokenize("The quick brown fox") → ["the", "quick", "brown", "fox"]
tokenize("AI is great!") → ["ai", "is", "great"]Similar Problems
Test Cases (3 visible · 1 hidden)
Case 1: Basic
Input: tokenize("Hello, World!")
Expected: ['hello', 'world']
Case 2: No punctuation
Input: tokenize("The quick brown fox")
Expected: ['the', 'quick', 'brown', 'fox']
Case 3: Exclamation
Input: tokenize("AI is great!")
Expected: ['ai', 'is', 'great']
⌘↵ Run · ⌘⇧↵ Submit