Add the ability to manage settings located with the model. #1344
Closed
Eugene-M
started this conversation in
New features / APIs
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
First of all, thanks for the library.
I have an Intel Core Ultra 7 265K (with NPU) and two NVIDIA GeForce 4070 Ti Super GPUs on board.
Currently, as far as I understood, it's impossible to configure library to run model with NPU enabled mode.
At the same time I can't select GPU1 to load the model:
My attempt to do this was not successful:
const string cuda = "cuda";
var config = new Config(gpuModelPath);
config.ClearProviders();
config.AppendProvider(cuda);
config.SetProviderOption(cuda, "device_id", "0");
config.SetProviderOption(cuda, "enable_cuda_graph", "1");
var model = Model(config);
If I change "device_id" from 0 to 1 I get the following error:
Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: 'Exception during initialization:
D:\a_work\1\s\onnxruntime\core\providers\cuda\cuda_call.cc:129 onnxruntime::CudaCall
D:\a_work\1\s\onnxruntime\core\providers\cuda\cuda_call.cc:121 onnxruntime::CudaCall CUDNN failure 2007:
CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH ; GPU=1 ; hostname=<MY_HOST> ;
file=D:\a_work\1\s\onnxruntime\core\providers\cuda\cuda_execution_provider.cc ; line=183 ; expr=cudnnSetStream(cudnn_handle_,
stream);
Along with the model itself, there are many files: general_config.json, merges.txt , tokenizer_config.json, vocab.json. It would be nice if you specified a way to transfer filePaths to the Config object, as well as the model name itself. Currently, we only define a folder, and somewhere in the library you have filenames (filePaths) hardcode. This leads to a lack of control and forces me to abandon the library and use Microsoft.ML.OnnxRuntime instead. But there aren't many good examples for the latter either, especially if I want to use a generative model.
Beta Was this translation helpful? Give feedback.
All reactions