This module contains the NLP support with Huggingface tokenizers implementation.
This is an implementation from Huggingface tokenizers RUST API.
You can build the latest javadocs locally using the following command:
./gradlew javadoc
The javadocs output is built in the build/doc/javadoc folder.
You can pull the module from the central Maven repository by including the following dependency in your pom.xml file:
<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
<version>0.17.0</version>
</dependency>