Skip to content

[GenAI] Use BitsAndBytes for 4bit quantization. #7406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LittleLittleCloud wants to merge 8 commits into
base: main
Choose a base branch
from

Merge branch 'u/0228' of https://github.com/LittleLittleCloud/machine…

62ffa45
Select commit
Loading
Failed to load commit list.
Open

[GenAI] Use BitsAndBytes for 4bit quantization. #7406

Merge branch 'u/0228' of https://github.com/LittleLittleCloud/machine…
62ffa45
Select commit
Loading
Failed to load commit list.
Build Analysis / Build Analysis failed Mar 11, 2025 in 0s

.NET Result Analysis

Details

Tip

To unconditionally bypass the build analysis check (turn it green), you can use the escape mechanism feature. The completion time may vary, potentially taking several minutes, depending on the build analysis workload at the moment.

Build Failures

MachineLearning-CodeCoverage / Windows_x64 / Windows_x64 Build_Debug / Run All Tests.

[ 🚧 Report infrastructure issue] [ 📄 Report repository issue]
    ❌Cmd.exe exited with code '1'.

MachineLearning-CI / Centos_x64 / Centos_x64 Debug_Build / Run Helix Tests

[ 🚧 Report infrastructure issue] [ 📄 Report repository issue]
    ❌Bash exited with code '1'.

MachineLearning-CI / Centos_x64 / Centos_x64 Release_Build / Run Helix Tests

[ 🚧 Report infrastructure issue] [ 📄 Report repository issue]
    ❌Bash exited with code '1'.

10 other jobs are failing and were not displayed due to space limitations.
Builds with failures: [MachineLearning-CodeCoverage] [MachineLearning-CI]

Test Failures (61 tests failed)

🔹 [All failing tests from MachineLearning-CI]

    ⚠️ 48 failing tests not displayed for this pipeline due to space limitations

    ❌ Microsoft.ML.AutoML.Test.AutoMLExperimentTests.AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async [Console] [Details] [Artifacts] [1.02% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]

    Failing Configuration
    Exception Message
    Expected value to be less than or equal to 2500L, but found 2501L (difference of 1).
    CallStack
       at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
   at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
   at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args)
   at FluentAssertions.Numeric.NumericAssertions`2.BeLessThanOrEqualTo(T expected, String because, Object[] becauseArgs)
   at FluentAssertions.Numeric.NumericAssertions`2.BeLessOrEqualTo(T expected, String because, Object[] becauseArgs)
   at Microsoft.ML.AutoML.Test.AutoMLExperimentTests.AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async() in /__w/1/s/test/Microsoft.ML.AutoML.Tests/AutoMLExperimentTests.cs:line 146
--- End of stack trace from previous location ---
    ❌ Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNer [Console] [Details] [Artifacts] [3.32% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]

    Failing Configurations (8)
    Exception Message
    System.DllNotFoundException : Unable to load DLL 'LibTorchSharp': The specified procedure could not be found. (Exception from HRESULT: 0x8007007F)
    CallStack
       at TorchSharp.PInvoke.NativeMethods.THSNN_custom_module(String name, ForwardFunctionC forward, IntPtr& pBoxedModule)
   at TorchSharp.torch.nn.Module..ctor(String name)
   at Microsoft.ML.TorchSharp.NasBert.Models.BaseModel..ctor(NasBertOptions options) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\BaseModel.cs:line 25
   at Microsoft.ML.TorchSharp.NasBert.Models.NasBertModel..ctor(NasBertOptions options, Int32 padIndex, Int32 symbolsCount) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\NasBertModel.cs:line 22
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase.CreateModule(IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 204
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.TrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 156
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 188
   at Microsoft.ML.TorchSharp.NasBert.NerTrainer.CreateTrainer(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NerTrainer.cs:line 106
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 106
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.Data\DataLoadSave\EstimatorChain.cs:line 68
   at Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNer() in D:\a\_work\1\s\test\Microsoft.ML.TorchSharp.Tests\NerTests.cs:line 79
    ❌ Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNerOptions [Console] [Details] [Artifacts] [3.32% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]

    Failing Configurations (8)
    Exception Message
    System.DllNotFoundException : Unable to load DLL 'LibTorchSharp': The specified procedure could not be found. (Exception from HRESULT: 0x8007007F)
    CallStack
       at TorchSharp.PInvoke.NativeMethods.THSNN_custom_module(String name, ForwardFunctionC forward, IntPtr& pBoxedModule)
   at TorchSharp.torch.nn.Module..ctor(String name)
   at Microsoft.ML.TorchSharp.NasBert.Models.BaseModel..ctor(NasBertOptions options) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\BaseModel.cs:line 25
   at Microsoft.ML.TorchSharp.NasBert.Models.NasBertModel..ctor(NasBertOptions options, Int32 padIndex, Int32 symbolsCount) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\NasBertModel.cs:line 22
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase.CreateModule(IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 204
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.TrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 156
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 188
   at Microsoft.ML.TorchSharp.NasBert.NerTrainer.CreateTrainer(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NerTrainer.cs:line 106
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 106
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.Data\DataLoadSave\EstimatorChain.cs:line 68
   at Microsoft.ML.TorchSharp.Tests.NerTests.TestSimpleNerOptions() in D:\a\_work\1\s\test\Microsoft.ML.TorchSharp.Tests\NerTests.cs:line 158
    ❌ Microsoft.ML.TorchSharp.Tests.ObjectDetectionTests.SimpleObjDetectionTest [Console] [Details] [Artifacts] [1.99% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]

    Failing Configurations (4)
    Exception Message
    System.TypeInitializationException : The type initializer for 'TorchSharp.torch' threw an exception.
---- System.NotSupportedException : This application or script uses TorchSharp but doesn't contain a reference to libtorch-cpu-win-x64, Version=2.5.1.0.

    Consider referencing one of the combination packages TorchSharp-cpu, TorchSharp-cuda-linux, TorchSharp-cuda-windows or call System.Runtime.InteropServices.NativeLibrary.Load(path-to-LibTorchSharp.dll) explicitly for a Python install of pytorch. See dotnet/TorchSharp#169;.
    

    For CUDA, you may need to call 'TorchSharp.torch.InitializeDeviceType(TorchSharp.DeviceType.CUDA)' before any use of TorchSharp CUDA packages from scripts or notebooks.
    

    Trace from LoadNativeBackend:
    

    TorchSharp: LoadNativeBackend: Initialising native backend, useCudaBackend = False
    

    Step 1 - First try regular load of native libtorch binaries.
    

    Trying to load native component torch_cpu relative to C:\h\w\A6B308BC\w\A91909AA\e\TorchSharp.dll
Failed to load native component torch_cpu relative to C:\h\w\A6B308BC\w\A91909AA\e\TorchSharp.dll
Trying to load native component LibTorchSharp relative to C:\h\w\A6B308BC\w\A91909AA\e\TorchSharp.dll
Failed to load native component LibTorchSharp relative to C:\h\w\A6B308BC\w\A91909AA\e\TorchSharp.dll
Result from regular native load of LibTorchSharp is False

    

    Step 3 - Alternative load from consolidated directory of native binaries from nuget packages
    

    torchsharpLoc = C:\h\w\A6B308BC\w\A91909AA\e
packagesDir = C:\h\w
torchsharpHome = C:\h\w\A6B308BC\w
Giving up, TorchSharp.dll does not appear to have been loaded from package directories

    CallStack
       at Microsoft.ML.TorchSharp.Utils.TorchUtils.InitializeDevice(IHostEnvironment env) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\Utils\TorchUtils.cs:line 39
   at Microsoft.ML.TorchSharp.AutoFormerV2.ObjectDetectionTrainer.Trainer..ctor(ObjectDetectionTrainer parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\AutoFormerV2\ObjectDetectionTrainer.cs:line 212
   at Microsoft.ML.TorchSharp.AutoFormerV2.ObjectDetectionTrainer.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\AutoFormerV2\ObjectDetectionTrainer.cs:line 166
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.Data\DataLoadSave\EstimatorChain.cs:line 68
   at Microsoft.ML.TorchSharp.Tests.ObjectDetectionTests.SimpleObjDetectionTest() in D:\a\_work\1\s\test\Microsoft.ML.TorchSharp.Tests\ObjectDetectionTests.cs:line 81
----- Inner Stack Trace -----
   at TorchSharp.torch.LoadNativeBackend(Boolean useCudaBackend, StringBuilder& trace)
   at TorchSharp.torch.InitializeDeviceType(DeviceType deviceType)
   at TorchSharp.torch.InitializeDevice(Device device)
   at TorchSharp.torch..cctor()
    ❌ Microsoft.ML.TorchSharp.Tests.TextClassificationTests.TestDoubleSentence2Classes [Console] [Details] [Artifacts] [3.32% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]

    Failing Configurations (8)
    Exception Message
    System.DllNotFoundException : Unable to load DLL 'LibTorchSharp': The specified procedure could not be found. (Exception from HRESULT: 0x8007007F)
    CallStack
       at TorchSharp.PInvoke.NativeMethods.THSNN_custom_module(String name, ForwardFunctionC forward, IntPtr& pBoxedModule)
   at TorchSharp.torch.nn.Module..ctor(String name)
   at Microsoft.ML.TorchSharp.NasBert.Models.BaseModel..ctor(NasBertOptions options) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\BaseModel.cs:line 25
   at Microsoft.ML.TorchSharp.NasBert.Models.NasBertModel..ctor(NasBertOptions options, Int32 padIndex, Int32 symbolsCount) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\Models\NasBertModel.cs:line 22
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase.CreateModule(IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 208
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.TrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 156
   at Microsoft.ML.TorchSharp.NasBert.NasBertTrainer`2.NasBertTrainerBase..ctor(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input, String modelUrl) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\NasBertTrainer.cs:line 188
   at Microsoft.ML.TorchSharp.NasBert.TextClassificationTrainer.CreateTrainer(TorchSharpBaseTrainer`2 parent, IChannel ch, IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\NasBert\TextClassificationTrainer.cs:line 105
   at Microsoft.ML.TorchSharp.TorchSharpBaseTrainer`2.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.TorchSharp\TorchSharpBaseTrainer.cs:line 106
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input) in D:\a\_work\1\s\src\Microsoft.ML.Data\DataLoadSave\EstimatorChain.cs:line 68
   at Microsoft.ML.RunTests.TestDataPipeBase.TestEstimatorCore(IEstimator`1 estimator, IDataView validFitInput, IDataView validTransformInput, IDataView invalidInput, IDataView validForFitNotValidForTransformInput, Boolean shouldDispose) in D:\a\_work\1\s\test\Microsoft.ML.TestFramework\DataPipe\TestDataPipeBase.cs:line 80
   at Microsoft.ML.TorchSharp.Tests.TextClassificationTests.TestDoubleSentence2Classes() in D:\a\_work\1\s\test\Microsoft.ML.TorchSharp.Tests\TextClassificationTests.cs:line 323

🔹 [All failing tests from MachineLearning-CodeCoverage]

    ⚠️ 8 failing tests not displayed for this pipeline due to space limitations

Was this helpful? Yes No

View more details on Build Analysis