This module contains the NLP support with Huggingface tokenizers implementation.
This is an implementation from Huggingface tokenizers RUST API.
You can build the latest javadocs locally using the following command:
The javadocs output is built in the
You can pull the module from the central Maven repository by including the following dependency in your
<dependency> <groupId>ai.djl.huggingface</groupId> <artifactId>tokenizers</artifactId> <version>0.17.0</version> </dependency>