tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Growing steadily this week

7d +0.17% +19 stars in 7d
30d +0.63% +70 stars in 30d
Sep 13, 2026: 11,034 starsSep 14, 2026: 11,038 starsSep 15, 2026: 11,036 starsSep 16, 2026: 11,039 starsSep 17, 2026: 11,040 starsSep 18, 2026: 11,047 starsSep 19, 2026: 11,047 starsSep 20, 2026: 11,050 starsSep 21, 2026: 11,052 starsSep 22, 2026: 11,068 starsSep 23, 2026: 11,100 starsSep 24, 2026: 11,108 starsSep 25, 2026: 11,112 starsSep 26, 2026: 11,117 starsSep 27, 2026: 11,123 starsSep 28, 2026: 11,134 starsSep 29, 2026: 11,138 starsSep 30, 2026: 11,142 starsOct 1, 2026: 11,145 starsOct 2, 2026: 11,147 starsOct 3, 2026: 11,150 starsOct 4, 2026: 11,151 starsOct 5, 2026: 11,152 starsOct 6, 2026: 11,156 starsOct 7, 2026: 11,158 starsOct 8, 2026: 11,162 starsOct 9, 2026: 11,162 starsOct 10, 2026: 11,163 starsOct 11, 2026: 11,166 starsOct 12, 2026: 11,167 starsOct 13, 2026: 11,167 starsOct 14, 2026: 11,169 starsOct 15, 2026: 11,171 starsOct 16, 2026: 11,177 starsOct 17, 2026: 11,181 starsOct 18, 2026: 11,183 starsOct 19, 2026: 11,185 starsOct 20, 2026: 11,187 starsOct 21, 2026: 11,188 starsOct 22, 2026: 11,190 starsOct 23, 2026: 11,192 starsOct 24, 2026: 11,193 starsOct 25, 2026: 11,198 starsOct 26, 2026: 11,203 starsOct 27, 2026: 11,204 starsOct 28, 2026: 11,205 starsOct 29, 2026: 11,211 starsOct 30, 2026: 11,215 starsOct 31, 2026: 11,216 starsNov 1, 2026: 11,217 starsNov 2, 2026: 11,220 starsNov 3, 2026: 11,217 starsNov 4, 2026: 11,219 starsNov 5, 2026: 11,222 starsNov 6, 2026: 11,224 starsNov 7, 2026: 11,226 starsNov 8, 2026: 11,228 starsNov 9, 2026: 11,230 starsNov 10, 2026: 11,231 starsNov 11, 2026: 11,243 starsNov 12, 2026: 11,253 starsNov 13, 2026: 11,260 starsNov 14, 2026: 11,244 starsNov 15, 2026: 11,247 stars 11,03411,14711,260 Sep 13Nov 15

11,034 → 11,247 stars since Sep 13

11,030 Stars 1,259 Forks 184 Open issues

Timeline

💬 1 Hacker News discussion

bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers

Don't lose track of what's rising

The map refreshes every week. Follow @awesomemapdev for the week's biggest risers, or star the repo so you can find your way back.