Skip to content

Commit cfb2cc0

Browse files
xaduprexadupre
authored
Update CI with latext onnxruntime, xgboost (#602)
* Update CI with latext onnxruntime, xgboost Signed-off-by: xadupre <[email protected]> * add one unit test for issue 601 Signed-off-by: xadupre <[email protected]> * fix iterable Signed-off-by: xadupre <[email protected]> * fix .str_ Signed-off-by: xadupre <[email protected]> * hinge Signed-off-by: xadupre <[email protected]> * fix hinge Signed-off-by: xadupre <[email protected]> * fix str Signed-off-by: xadupre <[email protected]> * update CI Signed-off-by: xadupre <[email protected]> * fix yml Signed-off-by: xadupre <[email protected]> * fix coreml issue Signed-off-by: xadupre <[email protected]> * update CI Signed-off-by: xadupre <[email protected]> * update CI Signed-off-by: xadupre <[email protected]> * fix CI Signed-off-by: xadupre <[email protected]> * fix CI Signed-off-by: xadupre <[email protected]> * update CI Signed-off-by: xadupre <[email protected]> * update CI Signed-off-by: xadupre <[email protected]> * update CI Signed-off-by: xadupre <[email protected]> * update CI Signed-off-by: xadupre <[email protected]> Signed-off-by: xadupre <[email protected]>
1 parent d9a6108 commit cfb2cc0

File tree

13 files changed

+247
-51
lines changed

13 files changed

+247
-51
lines changed

‎.azure-pipelines/linux-conda-CI.yml‎

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,53 +14,68 @@ jobs:
1414
vmImage: 'ubuntu-latest'
1515
strategy:
1616
matrix:
17+
Python310-1130-RT1131-xgb173:
18+
python.version: '3.10'
19+
ONNX_PATH: 'onnx==1.13.0' #'-i https://test.pypi.org/simple/ onnx==1.12.0rc4'
20+
ONNXRT_PATH: onnxruntime==1.13.1 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
21+
COREML_PATH: NONE
22+
xgboost.version: '>=1.7.3'
23+
numpy.version: ''
1724
Python310-1120-RT1121-xgb161:
18-
python.version: '3.9'
25+
python.version: '3.10'
1926
ONNX_PATH: 'onnx==1.12.0' #'-i https://test.pypi.org/simple/ onnx==1.12.0rc4'
2027
ONNXRT_PATH: onnxruntime==1.12.1 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
21-
COREML_PATH: git+https://github.com/apple/[email protected]
22-
xgboost.version: '>=1.6.1'
28+
COREML_PATH: NONE
29+
xgboost.version: '==1.6.1'
30+
numpy.version: ''
2331
Python39-1120-RT1110-xgb161:
2432
python.version: '3.9'
2533
ONNX_PATH: 'onnx==1.12.0' #'-i https://test.pypi.org/simple/ onnx==1.12.0rc4'
2634
ONNXRT_PATH: onnxruntime==1.11.0 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
2735
COREML_PATH: git+https://github.com/apple/[email protected]
2836
xgboost.version: '>=1.6.1'
37+
numpy.version: ''
2938
Python39-1120-RT1110-xgb142:
3039
python.version: '3.9'
3140
ONNX_PATH: 'onnx==1.12.0' #'-i https://test.pypi.org/simple/ onnx==1.12.0rc4'
3241
ONNXRT_PATH: onnxruntime==1.11.0 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
3342
COREML_PATH: git+https://github.com/apple/[email protected]
3443
xgboost.version: '==1.4.2'
44+
numpy.version: ''
3545
Python39-1110-RT1110:
3646
python.version: '3.9'
3747
ONNX_PATH: onnx==1.11.0 # '-i https://test.pypi.org/simple/ onnx==1.9.101'
3848
ONNXRT_PATH: onnxruntime==1.11.0 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
3949
COREML_PATH: git+https://github.com/apple/[email protected]
50+
numpy.version: ''
4051
Python39-1110-RT1100:
4152
python.version: '3.9'
4253
ONNX_PATH: onnx==1.11.0 # '-i https://test.pypi.org/simple/ onnx==1.9.101'
4354
ONNXRT_PATH: onnxruntime==1.10.0
4455
COREML_PATH: git+https://github.com/apple/[email protected]
4556
xgboost.version: '>=1.2'
57+
numpy.version: ''
4658
Python39-1101-RT190:
4759
python.version: '3.9'
4860
ONNX_PATH: onnx==1.10.1
4961
ONNXRT_PATH: onnxruntime==1.9.0
5062
COREML_PATH: git+https://github.com/apple/[email protected]
5163
xgboost.version: '>=1.2'
64+
numpy.version: '<=1.23.5'
5265
Python39-190-RT180-xgb11:
5366
python.version: '3.9'
5467
ONNX_PATH: onnx==1.9.0
5568
ONNXRT_PATH: onnxruntime==1.8.0
5669
COREML_PATH: git+https://github.com/apple/[email protected]
5770
xgboost.version: '>=1.2'
71+
numpy.version: '<=1.23.5'
5872
Python38-181-RT170-xgb11:
5973
python.version: '3.8'
6074
ONNX_PATH: onnx==1.8.1
6175
ONNXRT_PATH: onnxruntime==1.7.0
6276
COREML_PATH: git+https://github.com/apple/[email protected]
6377
xgboost.version: '>=1.2'
78+
numpy.version: '<=1.23.5'
6479

6580
maxParallel: 3
6681

@@ -85,7 +100,7 @@ jobs:
85100
pip install xgboost$(xgboost.version)
86101
pip install $(ONNX_PATH)
87102
pip install $(ONNXRT_PATH)
88-
pip install $(COREML_PATH)
103+
pip install "numpy$(numpy.version)"
89104
displayName: 'Install xgboost, onnxruntime'
90105
91106
- script: |
@@ -149,7 +164,7 @@ jobs:
149164
displayName: 'pytest - h2o'
150165
151166
- script: |
152-
pip install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
167+
pip install torch --extra-index-url https://download.pytorch.org/whl/cpu
153168
pip install hummingbird-ml --no-deps
154169
displayName: 'Install hummingbird-ml'
155170
@@ -159,9 +174,16 @@ jobs:
159174
displayName: 'pytest - hummingbirdml'
160175
161176
- script: |
162-
export PYTHONPATH=.
163-
pytest tests/coreml --durations=0
164-
displayName: 'pytest - coreml'
177+
if [ '$(COREML_PATH)' == 'NONE' ]
178+
then
179+
echo "required version of coremltools does not work on python 3.10"
180+
else
181+
export PYTHONPATH=.
182+
pip install $(COREML_PATH)
183+
pytest tests/coreml --durations=0
184+
fi
185+
displayName: 'pytest - coreml [$(COREML_PATH)]'
186+
# condition: ne('$(COREML_PATH)', 'NONE')
165187
166188
- task: PublishTestResults@2
167189
inputs:

‎.azure-pipelines/win32-conda-CI.yml‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,47 +15,61 @@ jobs:
1515
strategy:
1616
matrix:
1717

18+
Python310-1130-RT1131:
19+
python.version: '3.10'
20+
ONNX_PATH: 'onnx==1.13.0' # '-i https://test.pypi.org/simple/ onnx==1.12.0rc4'
21+
ONNXRT_PATH: onnxruntime==1.13.1 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
22+
COREML_PATH: NONE
23+
numpy.version: ''
24+
1825
Python39-1120-RT1110:
1926
python.version: '3.9'
2027
ONNX_PATH: 'onnx==1.12.0' # '-i https://test.pypi.org/simple/ onnx==1.12.0rc4'
2128
ONNXRT_PATH: onnxruntime==1.11.0 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
2229
COREML_PATH: git+https://github.com/apple/[email protected]
30+
numpy.version: ''
2331

2432
Python39-1110-RT1110:
2533
python.version: '3.9'
2634
ONNX_PATH: onnx==1.11.0 # '-i https://test.pypi.org/simple/ onnx==1.9.101'
2735
ONNXRT_PATH: onnxruntime==1.11.0 #'-i https://test.pypi.org/simple/ ort-nightly==1.11.0.dev20220311003'
2836
COREML_PATH: git+https://github.com/apple/[email protected]
37+
numpy.version: ''
2938

3039
Python39-1110-RT190:
3140
python.version: '3.9'
3241
ONNX_PATH: 'onnx==1.11.0' # '-i https://test.pypi.org/simple/ onnx==1.9.101'
3342
ONNXRT_PATH: onnxruntime==1.10.0
3443
COREML_PATH: git+https://github.com/apple/[email protected]
44+
numpy.version: ''
3545

3646
Python39-1102-RT190:
3747
python.version: '3.9'
3848
ONNX_PATH: 'onnx==1.10.2' # '-i https://test.pypi.org/simple/ onnx==1.9.101'
3949
ONNXRT_PATH: onnxruntime==1.9.0
4050
COREML_PATH: git+https://github.com/apple/[email protected]
51+
numpy.version: '<=1.23.5'
4152

4253
Python39-190-RT181:
4354
python.version: '3.9'
4455
ONNX_PATH: 'onnx==1.9.0'
4556
ONNXRT_PATH: onnxruntime==1.8.1
4657
COREML_PATH: git+https://github.com/apple/[email protected]
58+
numpy.version: '<=1.23.5'
4759

4860
Python39-190-RT180:
4961
python.version: '3.9'
5062
ONNX_PATH: onnx==1.9.0
5163
ONNXRT_PATH: onnxruntime==1.8.0
5264
COREML_PATH: git+https://github.com/apple/[email protected]
65+
numpy.version: '<=1.23.5'
5366

5467
Python38-181-RT170:
5568
python.version: '3.8'
5669
ONNX_PATH: onnx==1.8.1
5770
ONNXRT_PATH: onnxruntime==1.7.0
5871
COREML_PATH: git+https://github.com/apple/[email protected]
72+
numpy.version: '<=1.23.5'
5973

6074
maxParallel: 3
6175

@@ -83,15 +97,11 @@ jobs:
8397
python -m pip install --upgrade scikit-learn
8498
displayName: 'Install scikit-learn'
8599
86-
- script: |
87-
call activate py$(python.version)
88-
python -m pip install %COREML_PATH%
89-
displayName: 'Install coremltools'
90-
91100
- script: |
92101
call activate py$(python.version)
93102
python -m pip install %ONNX_PATH%
94103
python -m pip install %ONNXRT_PATH%
104+
python -m pip install "numpy$(numpy.version)"
95105
displayName: 'Install onnxruntime'
96106
97107
- script: |
@@ -124,9 +134,11 @@ jobs:
124134
125135
- script: |
126136
call activate py$(python.version)
127-
export PYTHONPATH=.
128-
python -m pytest tests/coreml --durations=0
129-
displayName: 'pytest coreml'
137+
set PYTHONPATH=.
138+
if "$(COREML_PATH)" neq "NONE" python -m pip install %COREML_PATH%
139+
if "$(COREML_PATH)" neq "NONE" python -m pytest tests/coreml --durations=0
140+
displayName: 'pytest coreml - [$(COREML_PATH)]'
141+
#condition: ne('$(COREML_PATH)', 'NONE')
130142
131143
- script: |
132144
call activate py$(python.version)
@@ -160,7 +172,7 @@ jobs:
160172
161173
- script: |
162174
call activate py$(python.version)
163-
python -m pip install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
175+
python -m pip install torch
164176
python -m pip install hummingbird-ml --no-deps
165177
displayName: 'Install hummingbird-ml'
166178

‎onnxmltools/convert/sparkml/operator_converters/aft_survival_regression.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@ def convert_aft_survival_regression(scope, operator, container):
1515
coefficients = op.coefficients.toArray().astype(float)
1616
coefficients_tensor = scope.get_unique_variable_name('coefficients_tensor')
1717
container.add_initializer(coefficients_tensor, onnx_proto.TensorProto.FLOAT, [1, len(coefficients)], coefficients)
18-
intercepts = op.intercept.astype(float) if isinstance(op.intercept, collections.Iterable) else [float(op.intercept)]
18+
intercepts = (
19+
op.intercept.astype(float)
20+
if isinstance(op.intercept, collections.abc.Iterable)
21+
else [float(op.intercept)])
1922
intercepts_tensor = scope.get_unique_variable_name('intercepts_tensor')
2023
container.add_initializer(intercepts_tensor, onnx_proto.TensorProto.FLOAT, [len(intercepts)], intercepts)
2124

‎onnxmltools/convert/sparkml/operator_converters/linear_regressor.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,10 @@ def convert_sparkml_linear_regressor(scope, operator, container):
1313
attrs = {
1414
'name': scope.get_unique_operator_name(op_type),
1515
'coefficients': op.coefficients.astype(float),
16-
'intercepts': op.intercept.astype(float) if isinstance(op.intercept, collections.Iterable) else [
17-
float(op.intercept)]
16+
'intercepts': (
17+
op.intercept.astype(float)
18+
if isinstance(op.intercept, collections.abc.Iterable)
19+
else [float(op.intercept)])
1820
}
1921
container.add_node(op_type, operator.input_full_names, operator.output_full_names, op_domain='ai.onnx.ml', **attrs)
2022

‎onnxmltools/convert/xgboost/convert.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,4 +41,13 @@ def convert(model, name=None, initial_types=None, doc_string='', target_opset=No
4141
topology = parse_xgboost(model, initial_types, target_opset, custom_conversion_functions, custom_shape_calculators)
4242
topology.compile()
4343
onnx_model = convert_topology(topology, name, doc_string, target_opset, targeted_onnx)
44+
opsets = {d.domain: d.version for d in onnx_model.opset_import}
45+
if '' in opsets and opsets[''] < 9 and target_opset >= 9:
46+
# a bug?
47+
opsets[''] = 9
48+
del onnx_model.opset_import[:]
49+
for k, v in opsets.items():
50+
opset = onnx_model.opset_import.add()
51+
opset.domain = k
52+
opset.version = v
4453
return onnx_model

‎onnxmltools/convert/xgboost/operator_converters/XGBoost.py‎

Lines changed: 35 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import json
44
import numpy as np
5+
from onnx import TensorProto
56
from xgboost import XGBClassifier
67
from ...common._registration import register_converter
78
from ..common import get_xgb_params
@@ -241,14 +242,17 @@ def convert(scope, operator, container):
241242
raise RuntimeError("XGBoost model is empty.")
242243
if ncl <= 1:
243244
ncl = 2
244-
# See https://github.com/dmlc/xgboost/blob/master/src/common/math.h#L23.
245-
attr_pairs['post_transform'] = "LOGISTIC"
246-
attr_pairs['class_ids'] = [0 for v in attr_pairs['class_treeids']]
247-
if js_trees[0].get('leaf', None) == 0:
248-
attr_pairs['base_values'] = [0.5]
249-
elif base_score != 0.5:
250-
cst = - np.log(1 / np.float32(base_score) - 1.)
251-
attr_pairs['base_values'] = [cst]
245+
if objective != 'binary:hinge':
246+
# See https://github.com/dmlc/xgboost/blob/master/src/common/math.h#L23.
247+
attr_pairs['post_transform'] = "LOGISTIC"
248+
attr_pairs['class_ids'] = [0 for v in attr_pairs['class_treeids']]
249+
if js_trees[0].get('leaf', None) == 0:
250+
attr_pairs['base_values'] = [0.5]
251+
elif base_score != 0.5:
252+
cst = - np.log(1 / np.float32(base_score) - 1.)
253+
attr_pairs['base_values'] = [cst]
254+
else:
255+
attr_pairs['base_values'] = [base_score]
252256
else:
253257
# See https://github.com/dmlc/xgboost/blob/master/src/common/math.h#L35.
254258
attr_pairs['post_transform'] = "SOFTMAX"
@@ -264,13 +268,33 @@ def convert(scope, operator, container):
264268
attr_pairs['classlabels_strings'] = classes
265269

266270
# add nodes
267-
if objective == "binary:logistic":
271+
if objective in ("binary:logistic", "binary:hinge"):
268272
ncl = 2
269-
container.add_node('TreeEnsembleClassifier', operator.input_full_names,
270-
operator.output_full_names,
273+
if objective == "binary:hinge":
274+
attr_pairs['post_transform'] = 'NONE'
275+
output_names = [operator.output_full_names[0],
276+
scope.get_unique_variable_name("output_prob")]
277+
else:
278+
output_names = operator.output_full_names
279+
container.add_node('TreeEnsembleClassifier',
280+
operator.input_full_names,
281+
output_names,
271282
op_domain='ai.onnx.ml',
272283
name=scope.get_unique_operator_name('TreeEnsembleClassifier'),
273284
**attr_pairs)
285+
if objective == "binary:hinge":
286+
if container.target_opset < 9:
287+
raise RuntimeError(
288+
f"hinge function cannot be implemented because "
289+
f"opset={container.target_opset}<9.")
290+
zero = scope.get_unique_variable_name("zero")
291+
one = scope.get_unique_variable_name("one")
292+
container.add_initializer(zero, TensorProto.FLOAT, [1], [0.])
293+
container.add_initializer(one, TensorProto.FLOAT, [1], [1.])
294+
greater = scope.get_unique_variable_name("output_prob")
295+
container.add_node("Greater", [output_names[1], zero], [greater])
296+
container.add_node('Where', [greater, one, zero],
297+
operator.output_full_names[1])
274298
elif objective in ("multi:softprob", "multi:softmax"):
275299
ncl = len(js_trees) // params['n_estimators']
276300
if objective == 'multi:softmax':

‎onnxmltools/proto/__init__.py‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ def _check_onnx_version():
2121
_check_onnx_version()
2222

2323

24-
def _make_tensor_fixed(name, data_type, dims, vals, raw=False):
24+
def make_tensor_fixed(name, data_type, dims, vals, raw=False):
2525
'''
2626
Make a TensorProto with specified arguments. If raw is False, this
2727
function will choose the corresponding proto field to store the
@@ -45,6 +45,3 @@ def _make_tensor_fixed(name, data_type, dims, vals, raw=False):
4545

4646
tensor.dims.extend(dims)
4747
return tensor
48-
49-
50-
helper.make_tensor = _make_tensor_fixed

‎onnxmltools/utils/utils_backend_onnxruntime.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ def compare_runtime(test, decimal=5, options=None, verbose=False, context=None):
5252
return
5353

5454
try:
55-
sess = onnxruntime.InferenceSession(onx)
55+
sess = onnxruntime.InferenceSession(onx, providers=["CPUExecutionProvider"])
5656
except ExpectedAssertionError as expe:
5757
raise expe
5858
except Exception as e:
@@ -312,7 +312,7 @@ def run_with_runtime(inputs, model_path):
312312
'''
313313
try:
314314
import onnxruntime
315-
session = onnxruntime.InferenceSession(model_path)
315+
session = onnxruntime.InferenceSession(model_path, providers=["CPUExecutionProvider"])
316316
output = session.run(None, inputs)
317317
return (output, session)
318318
except Exception as e:

‎tests/baseline/test_convert_baseline.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,10 @@
99
from onnx.defs import onnx_opset_version
1010
from onnxmltools.convert import convert_coreml
1111
from onnxconverter_common.onnx_ex import DEFAULT_OPSET_NUMBER
12-
import coremltools
12+
try:
13+
import coremltools
14+
except ImportError:
15+
coremltools = None
1316

1417

1518
TARGET_OPSET = min(DEFAULT_OPSET_NUMBER, onnx_opset_version())
@@ -24,6 +27,8 @@ def check_baseline(self, input_file, ref_file):
2427
def get_diff(self, input_file, ref_file):
2528
this = os.path.dirname(__file__)
2629
coreml_file = os.path.join(this, "models", input_file)
30+
if coremltools is None:
31+
return []
2732
cml = coremltools.utils.load_spec(coreml_file)
2833
onnx_model = convert_coreml(cml, target_opset=TARGET_OPSET)
2934
output_dir = os.path.join(this, "outmodels")

‎tests/h2o/test_h2o_converters.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ def _prepare_one_hot(file, y, exclude_cols=None):
9191

9292

9393
def _train_test_split_as_frames(x, y, is_str=False, is_classifier=False):
94-
y = y.astype(np.str) if is_str else y.astype(np.int64)
94+
y = y.astype(np.str_) if is_str else y.astype(np.int64)
9595
x_train, x_test, y_train, _ = train_test_split(x, y, test_size=0.3, random_state=42)
9696
f_train_x = H2OFrame(x_train)
9797
f_train_y = H2OFrame(y_train)
@@ -138,7 +138,7 @@ def predict_with_probabilities(self, data):
138138
return [preds.to_numpy()]
139139
else:
140140
return [
141-
preds.iloc[:, 0].to_numpy().astype(np.str),
141+
preds.iloc[:, 0].to_numpy().astype(np.str_),
142142
preds.iloc[:, 1:].to_numpy()
143143
]
144144

0 commit comments

Comments
 (0)