Microsoft.ML.Tokenizers.Data.Cl100kBase 1.0.3
About
The Microsoft.ML.Tokenizers.Data.Cl100kBase includes the Tiktoken tokenizer data file cl100k_base.tiktoken, which is utilized by models such as GPT-4.
Key Features
- This package mainly contains the cl100k_base.tiktoken file, which is used by the Tiktoken tokenizer. This data file is used by the following models: 1. gpt-4 2. gpt-3.5-turbo 3. gpt-3.5-turbo-16k 4. gpt-35 5. gpt-35-turbo 6. gpt-35-turbo-16k 7. text-embedding-ada-002 8. text-embedding-3-small 9. text-embedding-3-large
How to Use
Reference this package in your project to use the Tiktoken tokenizer with the specified models.
// Create a tokenizer for the specified model or any other listed model name
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4");
// Create a tokenizer for the specified encoding
Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("cl100k_base");
Main Types
Users shouldn't use any types exposed by this package directly. This package is intended to provide tokenizer data files.
Additional Documentation
Related Packages
Microsoft.ML.Tokenizers
Feedback & Contributing
Microsoft.ML.Tokenizers.Data.Cl100kBase is released as open source under the MIT license. Bug reports and contributions are welcome at the GitHub repository.
Showing the top 20 packages that depend on Microsoft.ML.Tokenizers.Data.Cl100kBase.
| Packages | Downloads |
|---|---|
|
Aspose.PDF
Aspose.PDF for .NET is a PDF document creation and manipulation component that enables your .NET applications to read, write and manipulate existing PDF documents without using Adobe Acrobat. It also allows you to create forms and manage form fields embedded in a PDF document. This component is written in managed C# and it allows developers to add PDF creation and manipulation functionality to their Microsoft .NET applications (WinForms, ASP.NET and .NET Compact Framework).
Aspose.PDF for .NET is affordable and offers an incredible wealth of features including PDF compression options; table creation and manipulation; support for graph objects; extensive hyperlink functionality; extended security controls; custom font handling; integration with data sources; add or remove bookmarks; create table of contents; add, update, delete attachments and annotations; import or export PDF form data; add, replace or remove text and images; split, concatenate, extract or insert pages; transform pages to image; print PDF documents and much more.
|
2 |
https://aka.ms/mlnetreleasenotes
.NET Standard 2.0
- Microsoft.ML.Tokenizers (>= 1.0.3)
| Version | Downloads | Last updated |
|---|---|---|
| 3.0.0-preview.26457.2 | 1 | 2026/9/17 |
| 3.0.0-preview.26160.2 | 1 | 2026/9/17 |
| 2.0.0 | 1 | 2026/9/17 |
| 2.0.0-preview.25527.5 | 1 | 2026/9/17 |
| 2.0.0-preview.25503.2 | 1 | 2026/9/17 |
| 2.0.0-preview.25373.1 | 1 | 2026/9/17 |
| 2.0.0-preview.1.25127.4 | 1 | 2026/9/17 |
| 2.0.0-preview.1.25125.4 | 1 | 2026/9/18 |
| 1.0.3 | 1 | 2026/9/17 |
| 1.0.2 | 1 | 2026/9/17 |
| 1.0.1 | 1 | 2026/9/17 |
| 1.0.0 | 1 | 2026/9/17 |
| 0.22.0 | 1 | 2026/9/16 |
| 0.22.0-preview.24526.1 | 1 | 2026/9/17 |
| 0.22.0-preview.24522.7 | 1 | 2026/9/17 |