Repository navigation
Enable custom token counting #7263
Description
Currently, Tokenizer returns counts strictly based on tokens.
However, there are scenarios where library authors may want / need to implement their own custom token counting function.
One such scenario is providing token counts for image inputs. In such cases, AI service providers provide an arbitrary way of calculating cost based on a fixed token count.
| Provider | Cost calculations | Does tokenization | Link |
|---|---|---|---|
| OpenAI | Fixed | No | https://platform.openai.com/docs/guides/vision/calculating-costs |
| Claude | Fixed | No | https://docs.anthropic.com/en/docs/build-with-claude/vision#calculate-image-costs |
| Gemini | Fixed | No | https://ai.google.dev/gemini-api/docs/tokens?lang=python#multimodal-tokens |
| Cohere | N/A | N/A | https://docs.cohere.com/docs/tokens |
| Mistral | N/A | N/A | https://docs.mistral.ai/guides/tokenization/#tokens-count |
Activity
rosebyte commentedon Sep 23, 2026
@luisquintanilla, could you please elaborate the request and ideally provide a use case?
dotnet-policy-service commentedon Sep 23, 2026
This issue has been marked
needs-author-actionand may be missing some important information.dotnet-policy-service commentedon Oct 7, 2026
This issue has been automatically marked
no-recent-activitybecause it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will removeno-recent-activity.dotnet-policy-service commentedon Oct 21, 2026
This issue will now be closed since it had been marked
no-recent-activitybut received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.