Skip to content

[GenAI] Use BitsAndBytes for 4bit quantization. #7406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LittleLittleCloud wants to merge 8 commits into
base: main
Choose a base branch
from

Merge branch 'u/0228' of https://github.com/LittleLittleCloud/machine…

62ffa45
Select commit
Loading
Failed to load commit list.
Open

[GenAI] Use BitsAndBytes for 4bit quantization. #7406

Merge branch 'u/0228' of https://github.com/LittleLittleCloud/machine…
62ffa45
Select commit
Loading
Failed to load commit list.
Azure Pipelines / MachineLearning-CodeCoverage failed Mar 11, 2025 in 1h 25m 23s

Build #20250311.1 had test failures

1 errors / 5 warnings

Details

Tests

  • Failed: 8 (0.40%)
  • Passed: 1,861 (93.71%)
  • Other: 117 (5.89%)
  • Total: 1,986

Annotations

Check failure on line 599 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Build log #L599 

Cmd.exe exited with code '1'.

Check failure on line 1 in Microsoft.ML.GenAI.Core.Tests.QuantizedLinearTests.Int8QuantizeForwardTest

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.GenAI.Core.Tests.QuantizedLinearTests.Int8QuantizeForwardTest 

System.DllNotFoundException : Unable to load DLL 'LibTorchSharp' or one of its dependencies: The specified module could not be found. (0x8007007E)
Raw output 
   at TorchSharp.PInvoke.NativeMethods.THSNN_custom_module(String name, ForwardFunctionC forward, IntPtr& pBoxedModule)
   at TorchSharp.torch.nn.Module..ctor(String name)
   at TorchSharp.torch.nn.HookableModule`2..ctor(String name)
   at TorchSharp.torch.nn.Module`2..ctor(String name)
   at Microsoft.ML.GenAI.Core.GenAILinear..ctor(Int32 inFeatures, Int32 outFeatures, Boolean hasBias, ScalarType dtype, String device) in D:\a\_work\1\s\src\Microsoft.ML.GenAI.Core\Module\GenAILinear.cs:line 19
   at Microsoft.ML.GenAI.Core.QuantizedLinear..ctor(Int32 inFeatures, Int32 outFeatures, Boolean hasBias, ScalarType dtype, String device) in D:\a\_work\1\s\src\Microsoft.ML.GenAI.Core\Module\QuantizedLinear.cs:line 22
   at Microsoft.ML.GenAI.Core.Tests.QuantizedLinearTests.Int8QuantizeForwardTest() in D:\a\_work\1\s\test\Microsoft.ML.GenAI.Core.Tests\QuantizedLinearTests.cs:line 41
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Check failure on line 1 in Microsoft.ML.TorchSharp.Tests.ObjectDetectionTests.SimpleObjDetectionTest

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.TorchSharp.Tests.ObjectDetectionTests.SimpleObjDetectionTest 

System.TypeInitializationException : The type initializer for 'TorchSharp.torch' threw an exception.\r\n---- System.NotSupportedException : The libtorch-cpu-win-x64 package version 2.5.1.0 is not restored on this system. If using F# Interactive or .NET Interactive you may need to add a reference to this package, e.g. \n    #r \"nuget: libtorch-cpu-win-x64, 2.5.1.0\". Trace from LoadNativeBackend:\n\r\nTorchSharp: LoadNativeBackend: Initialising native backend, useCudaBackend = False\r\n\r\nStep 1 - First try regular load of native libtorch binaries.\r\n\r\n    Trying to load native component torch_cpu relative to C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\\lib\\net6.0\\TorchSharp.dll\r\n    Failed to load native component torch_cpu relative to C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\\lib\\net6.0\\TorchSharp.dll\r\n    Trying to load native component LibTorchSharp relative to C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\\lib\\net6.0\\TorchSharp.dll\r\n    Failed to load native component LibTorchSharp relative to C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\\lib\\net6.0\\TorchSharp.dll\r\n    Result from regular native load of LibTorchSharp is False\r\n\r\nStep 3 - Alternative load from consolidated directory of native binaries from nuget packages\r\n\r\n    torchsharpLoc = C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\\lib\\net6.0\r\n    packagesDir = C:\\Users\\cloudtest\\.nuget\\packages\r\n    torchsharpHome = C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\r\n    Trying dynamic load for .NET/F# Interactive by consolidating native libtorch-cpu-win-x64-* binaries to C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\\lib\\net6.0\\cpu...\r\n    Consolidating native binaries, packagesDir=C:\\Users\\cloudtest\\.nuget\\packages, packagePattern=libtorch-cpu-win-x64, packageVersion=2.5.1.0 to target=C:\\Users\\cloudtest\\.nuget\\packages\\torchsharp\\0.105.0\\lib\\net6.0\\cpu...\r\n
Raw output 
   at Microsoft.ML.TorchSharp.Utils.TorchUtils.InitializeDevice(IHostEnvironment env) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\Utils\TorchUtils.cs:line 39
   at Microsoft.ML.TorchSharp.AutoFormerV2.ObjectDetectionTrainer.Trainer..ctor(ObjectDetectionTrainer parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\AutoFormerV2\ObjectDetectionTrainer.cs:line 213
   at Microsoft.ML.TorchSharp.AutoFormerV2.ObjectDetectionTrainer.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\AutoFormerV2\ObjectDetectionTrainer.cs:line 166
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.Data\DataLoadSave\EstimatorChain.cs:line 68
   at Microsoft.ML.TorchSharp.Tests.ObjectDetectionTests.SimpleObjDetectionTest() in D:\a\_work\1\s\test\Microsoft.ML.TorchSharp.Tests\ObjectDetectionTests.cs:line 81
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
----- Inner Stack Trace -----
   at TorchSharp.torch.LoadNativeBackend(Boolean useCudaBackend, StringBuilder& trace)
   at TorchSharp.torch.InitializeDeviceType(DeviceType deviceType)
   at TorchSharp.torch.InitializeDevice(Device device)
   at TorchSharp.torch..cctor()

Check failure on line 1 in Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNer

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNer 

System.DllNotFoundException : Unable to load DLL 'LibTorchSharp' or one of its dependencies: The specified module could not be found. (0x8007007E)
Raw output 
   at TorchSharp.PInvoke.NativeMethods.THSNN_custom_module(String name, ForwardFunctionC forward, IntPtr& pBoxedModule)
   at TorchSharp.torch.nn.Module..ctor(String name)
   at TorchSharp.torch.nn.HookableModule`2..ctor(String name)
   at TorchSharp.torch.nn.Module`3..ctor(String name)
   at Microsoft.ML.TorchSharp.NasBert.Models.BaseModel..ctor(NasBertOptions options) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\BaseModel.cs:line 25
   at Microsoft.ML.TorchSharp.NasBert.Models.NasBertModel..ctor(NasBertOptions options, Int32 padIndex, Int32 symbolsCount) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\NasBertModel.cs:line 20
   at Microsoft.ML.TorchSharp.NasBert.Models.NerModel..ctor(NasBertOptions options, Int32 padIndex, Int32 symbolsCount, Int32 numLabels) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\NerModel.cs:line 19
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase.CreateModule(IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 208
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.TrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 156
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 185
   at Microsoft.ML.TorchSharp.NasBert.NerTrainer.Trainer..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NerTrainer.cs:line 121
   at Microsoft.ML.TorchSharp.NasBert.NerTrainer.CreateTrainer(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NerTrainer.cs:line 106
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 107
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.Data\DataLoadSave\EstimatorChain.cs:line 68
   at Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNer() in D:\a\_work\1\s\test\Microsoft.ML.TorchSharp.Tests\NerTests.cs:line 79
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Check failure on line 1 in Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNerOptions

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNerOptions 

System.DllNotFoundException : Unable to load DLL 'LibTorchSharp' or one of its dependencies: The specified module could not be found. (0x8007007E)
Raw output 
   at TorchSharp.PInvoke.NativeMethods.THSNN_custom_module(String name, ForwardFunctionC forward, IntPtr& pBoxedModule)
   at TorchSharp.torch.nn.Module..ctor(String name)
   at TorchSharp.torch.nn.HookableModule`2..ctor(String name)
   at TorchSharp.torch.nn.Module`3..ctor(String name)
   at Microsoft.ML.TorchSharp.NasBert.Models.BaseModel..ctor(NasBertOptions options) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\BaseModel.cs:line 25
   at Microsoft.ML.TorchSharp.NasBert.Models.NasBertModel..ctor(NasBertOptions options, Int32 padIndex, Int32 symbolsCount) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\NasBertModel.cs:line 20
   at Microsoft.ML.TorchSharp.NasBert.Models.NerModel..ctor(NasBertOptions options, Int32 padIndex, Int32 symbolsCount, Int32 numLabels) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\NerModel.cs:line 19
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase.CreateModule(IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 208
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.TrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 156
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 185
   at Microsoft.ML.TorchSharp.NasBert.NerTrainer.Trainer..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NerTrainer.cs:line 121
   at Microsoft.ML.TorchSharp.NasBert.NerTrainer.CreateTrainer(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NerTrainer.cs:line 106
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 107
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.Data\DataLoadSave\EstimatorChain.cs:line 68
   at Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNerOptions() in D:\a\_work\1\s\test\Microsoft.ML.TorchSharp.Tests\NerTests.cs:line 158
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
View more details on Azure Pipelines