Setting session options

[benHeid]

I have a question regarding how I can set session options.

E.g., I would like to set sess_options.graph_optimization_level = rt.GraphOptimizationLevel, however, when using onnxruntime-genai, I haven't found any possibilities to so. Is this possible at all and if yes I would appreciate an hint how to set these options

[kunal-vaishnavi]

Here is how you can do that.

[benHeid]

Thanks for hinting me to this comment.

Unfortunately, when trying to set it in the config

            "session_options": {
                "log_id": "onnxruntime-genai",
                "provider_options": [],
                "graph_optimization_level": 0
            },

I will get the following RuntimeError:

RuntimeError: Error encountered while parsing './onnx/genai_config.json' JSON Error: model:decoder:session_options: Unknown value "graph_optimization_level" at line 9 index 46

So I guess that this option is not supported yet. Is this correct?

Also the SessionOptions of onyxruntime-genai, seems to not have graph_optimization_level

[kunal-vaishnavi]

Yes, it appears that graph_optimization_level has not been added. There are three places to update for this to be enabled.

1. When reading the value from the genai_config.json file
   

   onnxruntime-genai/src/config.cpp

   Lines 68 to 98 in fe3604a

   	void OnValue(std::string_view name, JSON::Value value) override {
   	if (name == "log_id")
   	v_.log_id = JSON::Get<std::string_view>(value);
   	else if (name == "enable_profiling")
   	v_.enable_profiling = JSON::Get<std::string_view>(value);
   	else if (name == "ep_context_embed_mode")
   	v_.ep_context_embed_mode = JSON::Get<std::string_view>(value);
   	else if (name == "ep_context_file_path")
   	v_.ep_context_file_path = JSON::Get<std::string_view>(value);
   	else if (name == "intra_op_num_threads")
   	v_.intra_op_num_threads = static_cast<int>(JSON::Get<double>(value));
   	else if (name == "inter_op_num_threads")
   	v_.inter_op_num_threads = static_cast<int>(JSON::Get<double>(value));
   	else if (name == "log_severity_level")
   	v_.log_severity_level = static_cast<int>(JSON::Get<double>(value));
   	else if (name == "enable_cpu_mem_arena")
   	v_.enable_cpu_mem_arena = JSON::Get<bool>(value);
   	else if (name == "enable_mem_pattern")
   	v_.enable_mem_pattern = JSON::Get<bool>(value);
   	else if (name == "disable_cpu_ep_fallback")
   	v_.disable_cpu_ep_fallback = JSON::Get<bool>(value);
   	else if (name == "disable_quant_qdq")
   	v_.disable_quant_qdq = JSON::Get<bool>(value);
   	else if (name == "enable_quant_qdq_cleanup")
   	v_.enable_quant_qdq_cleanup = JSON::Get<bool>(value);
   	else if (name == "ep_context_enable")
   	v_.ep_context_enable = JSON::Get<bool>(value);
   	else if (name == "use_env_allocators")
   	v_.use_env_allocators = JSON::Get<bool>(value);
   	else
   	throw JSON::unknown_value_error{};

2. When setting the value inside ONNX Runtime GenAI (i.e. at the location in your screenshot above)
   

   onnxruntime-genai/src/config.h

   Lines 32 to 51 in fe3604a

   	struct SessionOptions {
   	std::optional<int> intra_op_num_threads;
   	std::optional<int> inter_op_num_threads;
   	std::optional<bool> enable_cpu_mem_arena;
   	std::optional<bool> enable_mem_pattern;
   	std::optional<bool> disable_cpu_ep_fallback;
   	std::optional<bool> disable_quant_qdq;
   	std::optional<bool> enable_quant_qdq_cleanup;
   	std::optional<bool> ep_context_enable;
   	std::optional<std::string> ep_context_embed_mode;
   	std::optional<std::string> ep_context_file_path;
   	std::optional<std::string> log_id;
   	std::optional<int> log_severity_level;
   	std::optional<std::string> enable_profiling;
   	// TODO(baijumeswani): Sharing env allocators across sessions leads to crashes on windows and iOS.
   	// Identify the reason for the crash to enable allocator sharing by default.
   	bool use_env_allocators{};
   	std::vector<ProviderOptions> provider_options;
   	};

3. When setting all of the session options inside ONNX Runtime GenAI by calling the method in ONNX Runtime that sets the graph optimization level
   

   onnxruntime-genai/src/models/model.cpp

   Lines 326 to 342 in fe3604a

   	void Model::CreateSessionOptionsFromConfig(const Config::SessionOptions& config_session_options,
   	OrtSessionOptions& session_options,
   	bool is_primary_session_options,
   	bool disable_graph_capture) {
   	// Default to a limit of 16 threads to optimize performance
   	constexpr int min_thread_nums = 1;
   	constexpr int max_thread_nums = 16;
   	int num_of_cores = std::max(min_thread_nums, static_cast<int>(std::thread::hardware_concurrency() / 2));
   	session_options.SetIntraOpNumThreads(std::min(num_of_cores, max_thread_nums));
   	if (config_session_options.intra_op_num_threads.has_value()) {
   	session_options.SetIntraOpNumThreads(config_session_options.intra_op_num_threads.value());
   	}
   	if (config_session_options.inter_op_num_threads.has_value()) {
   	session_options.SetInterOpNumThreads(config_session_options.inter_op_num_threads.value());
   	}

Please feel free to open a PR to contribute this change.

[RyanUnderhill]

@benHeid We can make this change if you'd like, just say the word.

[benHeid]

I already created a pull request.