This document outlines the procedure to add new datasets to DJL.
test/resources/mlrepo/dataset
folder and create a folder in it to store your dataset based on its category.
For example, cv/ai/djl/basicdataset/mnist
.0.0.1
). The version should match your dataset version.metadata.json
fileYou need to create a metadata.json
file for the repository to load the dataset. You can refer to the format in the metadata.json
files for existing datasets to create your own.
Note: You need to update the sha1 hash of each file in your metadata.json
file. Use the following command to get the sha1Hash value:
$ shasum -a 1 <file_name>
Create a class that implements the dataset and loads it.
For more details on creating datasets, see the dataset creation guide.
You should also look at examples of official DJL datasets such as AmesRandomAccess
or Cifar10
.
Then, add some tests for the dataset. For testing, you can use a local repository such as:
Repository repository = Repository.newInstace("testRepository", Paths.get("/test/resources/mlrepo"));
Add your dataset to the list of built-in datasets.
The official DJL ML repository is located on an S3 bucket managed by the AWS DJL team. You have to add the metadata and any dataset files to the repository.
For non-AWS team members, go ahead straight to Step 6 and open a pull request. Within the pull request, you can coordinate with an AWS member to add the necessary files.
For AWS team members, run the following command to upload your model to the S3 bucket:
$ ./gradlew syncS3
The metadata.json
in DJL is mainly a repository of metadata.
Within the metadata is typically contains only links indicating where the actual data would be found.
However, some datasets can be distributed by DJL depending on whether it makes it easier to use and the dataset permits redistribution. In that case, coordinate with an AWS team member in your pull request.
Note: Avoid checking in binary files to git. Binary files should only be uploaded to the S3 bucket.
If you are relying on an AWS team member, you should leave your code with the local test repositories. If you try to use the official repository before it contains your metadata, the tests will not pass the CI.
Once an AWS team member adds your metadata, they will prompt you to update your PR to the official repository.