While DJL is designed to be engine-agnostic and can easily integrated with any engines, we currently support the following models:
Yes. DJL does support inference on GPU. If GPUs are available, DJL automatically detects the GPU, and runs inference on a single GPU by default.
OnnxRuntime engine by default depends on com.microsoft.onnxruntime:onnxruntime
CPU package.
You need install com.microsoft.onnxruntime:onnxruntime_gpu
to enable GPU for OnnxRuntime.
See: Install Onnxruntime GPU package
Yes. DJL offers high performance multi-threaded inference. For more information, see the inference_performance_optimization.
Yes. DJL offers multi-GPU support. DJL can automatically detect if GPUs are available. If GPUs are available, it will run on a single GPU by default, unless the user specifies otherwise.
During training, if you wish to train on multiple GPUs or if you wish to limit the number of GPUs to be used (you may want to limit the number of GPU for smaller datasets), you have to configure the TrainingConfig
to do so by
setting the devices. For example, if you have 7 GPUs available, and you want the Trainer
to train on 5 GPUs, you can configure it as follows.
int maxNumberOfGpus = 5;
TrainingConfig config = new DefaultTrainingConfig(initializer, loss)
.setOptimizer(optimizer)
.addEvaluator(accuracy)
.setBatchSize(batchSize)
// Set the devices to run on multi-GPU
.setDevices(Engine.getInstance().getDevices(numberOfGpus));
All of the examples in the example folder can be run on multiple GPUs with the appropriate arguments. Follow the steps in the example to train a ResNet50 model on CIFAR-10 dataset on a GPU.
DJL does not currently support distributed training.
Yes. Each DJL release has a range of PyTorch version.
You can set PYTORCH_VERSION
environment variable or Java System properties to choose
a different version of PyTorch.
If you want to load your custom build PyTorch you can follow this instruction.
DJL uses NDList
as a standard data type to pass to the model. NDList
is a flat list of tensor.
A typical PyTorch model can accept a Map, List or Tuple of tensor. DJL provides the following way
to automatically map NDList
to PyTorch’s IValue
:
NDArray
name with suffix []
to group them into a list[Tensor]
, see: exampleNDArray
name with suffix ()
to group them into a tuple[Tensor]
, see: exampleNDArray
name with suffix group.key
to group them into a dict(str, Tensor)
, see: exampleIf your model requires non-tensor input or complex IValue
, you have to use IValue
class directly
(This makes your code bound to PyTorch engine). See this example.
DJL provides a hybrid engine design that allows you to leverage PyTorch/MXNet/TensorFlow
to performance NDArray
operations for OnnxRuntime.
Yes, DJL has Python engine that allows you run inference with Python code. The Python engine provides the same code experience as other engines, and makes it easy for you to migrate to native Java model easier in future.
Sometime, it’s hard to port python data processing code into java due to lack of equivalent java library or implementing it in Java is time consuming. In this case, you can package your Python code as a processing model. See How to run python pre/post processing
You can check out our troubleshooting document, discussions, issues.
You can also join our slack channel to get in touch with the development team, for questions and discussions.