Skip to content

Commit 7b2d017

Browse files
smk2007smk2007
authored
Merge pull request #434 from microsoft/user/sheilk/imagesharp-test
Add ImageSharp Interop Sample
2 parents f4e9586 + 8abbf66 commit 7b2d017

File tree

12 files changed

+727
-12
lines changed

12 files changed

+727
-12
lines changed

‎README.md‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -95,16 +95,28 @@ A computer vision technique that allows us to recompose the content of an image
9595

9696
These advanced samples show how to use various binding and evaluation features in Windows ML:
9797

98-
- **[Custom Tensorization](https://github.com/Microsoft/Windows-Machine-Learning/tree/master/Samples/CustomTensorization)**: a Windows Console Application (C++/WinRT) that shows how to do custom tensorization.
99-
- **[Custom Operator (CPU)](https://github.com/Microsoft/Windows-Machine-Learning/tree/master/Samples/CustomOperator)**: a desktop app that defines multiple custom cpu operators. One of these is a debug operator which we invite you to integrate into your own workflow.
100-
- **[Adapter Selection](https://github.com/Microsoft/Windows-Machine-Learning/tree/master/Samples/AdapterSelection)**: a desktop app that demonstrates how to choose a specific device adapter for running your model
98+
- **[Custom Tensorization](https://github.com/Microsoft/Windows-Machine-Learning/tree/master/Samples/CustomTensorization)**: A Windows Console Application (C++/WinRT) that shows how to do custom tensorization.
99+
100+
- **[Custom Operator (CPU)](https://github.com/Microsoft/Windows-Machine-Learning/tree/master/Samples/CustomOperator)**: A desktop app that defines multiple custom cpu operators. One of these is a debug operator which we invite you to integrate into your own workflow.
101+
102+
- **[Adapter Selection](https://github.com/Microsoft/Windows-Machine-Learning/tree/master/Samples/AdapterSelection)**: A desktop app that demonstrates how to choose a specific device adapter for running your model.
103+
101104
- **[Plane Identifier](https://github.com/Microsoft/Windows-AppConsult-Samples-UWP/tree/master/PlaneIdentifier)**: a UWP app and a WPF app packaged with the Desktop Bridge, sharing the same model trained using [Azure Custom Vision service](https://customvision.ai/). For step-by-step instructions for this sample, please see the blog post [Upgrade your WinML application to the latest bits](https://blogs.msdn.microsoft.com/appconsult/2018/11/06/upgrade-your-winml-application-to-the-latest-bits/).
105+
102106
- **[Custom Vision and Windows ML](https://github.com/Microsoft/Windows-AppConsult-Samples-UWP/tree/master/PlaneIdentifier)**: The tutorial shows how to train a neural network model to classify images of food using Azure Custom Vision service, export the model to ONNX format, and deploy the model in a Windows Machine Learning application running locally on Windows device.
107+
103108
- **[ML.NET and Windows ML](https://github.com/Microsoft/Windows-AppConsult-Samples-UWP/tree/master/PlaneIdentifier)**: This tutorial shows you how to train a neural network model to classify images of food using ML.NET Model Builder, export the model to ONNX format, and deploy the model in a Windows Machine Learning application running locally on a Windows device.
109+
104110
- **[PyTorch Data Analysis](https://github.com/Microsoft/Windows-AppConsult-Samples-UWP/tree/master/PlaneIdentifier)**: The tutorial shows how to solve a classification task with a neural network using the PyTorch library, export the model to ONNX format and deploy the model with the Windows Machine Learning application that can run on any Windows device.
111+
105112
- **[PyTorch Image Classification](https://github.com/Microsoft/Windows-AppConsult-Samples-UWP/tree/master/PlaneIdentifier)**: The tutorial shows how to train an image classification neural network model using PyTorch, export the model to the ONNX format, and deploy it in a Windows Machine Learning application running locally on your Windows device.
113+
106114
- **[YoloV4 Object Detection](https://github.com/Microsoft/Windows-AppConsult-Samples-UWP/tree/master/PlaneIdentifier)**: This tutorial shows how to build a UWP C# app that uses the YOLOv4 model to detect objects in video streams.
107-
- **[OpenCV Interop](Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/OpenCVInterop)**: This sample demonstrates how to interop between [Windows ML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/) and [OpenCV](https://github.com/opencv/opencv).
115+
116+
- **[OpenCV](Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/OpenCVInterop)**: See how to integrate [Windows ML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/) with [OpenCV](https://github.com/opencv/opencv).
117+
118+
- **[ImageSharp](Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/ImageSHarpInterop)**: See how to integrate [Windows ML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/) with [ImageSharp](https://github.com/SixLabors/ImageSharp).
119+
108120

109121
## Developer Tools
110122

‎Samples/WinMLSamplesGallery/README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,14 +30,15 @@ To learn how to implement these features in your application, or unlock addition
3030
- Build and deploy the **WinMLSamplesGallery (Package)** project.
3131

3232
## Samples
33-
- [Image Classifiation](./WinMLSamplesGallery/Samples/ImageClassifier): Pick an image and classify the scene into 1000 categories such as keyboard, mouse, pencil, and many animals. This sample demonstrates image classification using a large number of models taken from the [ONNX Model Zoo](https://github.com/onnx/models).
33+
- [Image Classifiation](./WinMLSamplesGallery/Samples/ImageClassifier): This sample demonstrates image classification using a large number of models taken from the [ONNX Model Zoo](https://github.com/onnx/models).
3434

35-
- [Image Effects](./WinMLSamplesGallery/Samples/ImageEffects): Pick an image and apply a variety of effects powered by Windows AI MachineLearning like Blur, Sharpen, Contrast, and many more. These effects can be used to perform model input image preprocessing, or postprocessing for visualizing outputs.
35+
- [Image Effects](./WinMLSamplesGallery/Samples/ImageEffects): See how to apply preprocessing and postprocessing effects using platform and hardware agnostic ONNX Models and [chaining in Windows ML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/chaining).
3636

37-
- [Batched Inputs](./WinMLSamplesGallery/Samples/Batching): WinML enables batched inputs that allow callers to perform inference over multiple inputs at once in order to increase performance. Use this sample to compare inference runtime performace with and without batching.
37+
- [Batched Inputs](./WinMLSamplesGallery/Samples/Batching): See how to speed up inference with batched inputs.
3838

39-
- [OpenCV Interop](./WinMLSamplesGallery/Samples/OpenCVInterop): This sample demonstrates how to interop between [Windows ML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/) and [OpenCV](https://github.com/opencv/opencv). The demo will run [SqueezeNet](https://github.com/onnx/models/tree/master/vision/classification/squeezenet) image classification in WindowsML and consume images loaded and preprocessed using OpenCV.
39+
- [OpenCV](./WinMLSamplesGallery/Samples/OpenCVInterop): See how to integrate [Windows ML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/) with [OpenCV](https://github.com/opencv/opencv).
4040

41+
- [ImageSharp](./WinMLSamplesGallery/Samples/ImageSharpInterop): See how to integrate [Windows ML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/) with [ImageSharp](https://docs.sixlabors.com/articles/imagesharp/index.html).
4142
## Feedback
4243
Please file an issue [here](https://github.com/microsoft/Windows-Machine-Learning/issues/new) if you encounter any issues with the WinML Samples Gallery or wish to request a new sample.
4344

‎Samples/WinMLSamplesGallery/WinMLSamplesGallery/Pages/SampleBasePage.xaml.cs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ protected override async void OnNavigatedTo(NavigationEventArgs e)
3131
case "OpenCVInterop":
3232
SampleFrame.Navigate(typeof(Samples.OpenCVInterop));
3333
break;
34+
case "ImageSharpInterop":
35+
SampleFrame.Navigate(typeof(Samples.ImageSharpInterop));
36+
break;
3437
}
3538
if (sampleMetadata.Docs.Count > 0)
3639
DocsHeader.Visibility = Visibility.Visible;

‎Samples/WinMLSamplesGallery/WinMLSamplesGallery/SampleMetadata/SampleMetadata.json‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,24 @@
3636
]
3737
},
3838
{
39-
"Title": "OpenCV Interop",
40-
"DescriptionShort": "The sample uses Windows ML to classify images that have been denoised using OpenCV.",
39+
"Title": "OpenCV",
40+
"DescriptionShort": "The sample uses Windows ML to classify images that have been denoised natively using OpenCV.",
4141
"Description": "This sample demonstrates interop between Windows ML and OpenCV. The sample classifes images that have been denoised using OpenCV's medianBlur using the SqueezeNet model on Windows ML. Choose an image to get started.",
4242
"Icon": "\uE155",
4343
"Tag": "OpenCVInterop",
4444
"XAMLGithubLink": "https://github.com/microsoft/Windows-Machine-Learning/blob/master/Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/OpenCVInterop/OpenCVInterop.xaml",
4545
"CSharpGithubLink": "https://github.com/microsoft/Windows-Machine-Learning/blob/master/Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/OpenCVInterop/OpenCVInterop.xaml.cs",
4646
"Docs": []
47+
},
48+
{
49+
"Title": "ImageSharp",
50+
"DescriptionShort": "The sample uses Windows ML to classify images that have been loaded by the managed ImageSharp library.",
51+
"Description": "The sample uses Windows ML to classify images that have been rotated by the managed ImageSharp library. Choose an image to get started.",
52+
"Icon": "\uE155",
53+
"Tag": "ImageSharpInterop",
54+
"XAMLGithubLink": "https://github.com/microsoft/Windows-Machine-Learning/blob/master/Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/ImageSharpInterop/ImageSharpInterop.xaml",
55+
"CSharpGithubLink": "https://github.com/microsoft/Windows-Machine-Learning/blob/master/Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/ImageSharpInterop/ImageSharpInterop.xaml.cs",
56+
"Docs": []
4757
}
4858
]
4959
}

‎Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/ImageSharpInterop/ImageSharpExtensions.cs‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
using System;
2+
using System.Linq;
3+
using System.Runtime.InteropServices.WindowsRuntime;
4+
5+
using SixLabors.ImageSharp;
6+
using SixLabors.ImageSharp.Processing;
7+
using SixLabors.ImageSharp.PixelFormats;
8+
using SixLabors.ImageSharp.Advanced;
9+
using SixLabors.ImageSharp.Memory;
10+
11+
using Microsoft.AI.MachineLearning;
12+
using Windows.Graphics.Imaging;
13+
14+
namespace ImageSharpExtensionMethods
15+
{
16+
public static class ImageExtensions
17+
{
18+
public static Windows.Storage.Streams.IBuffer AsBuffer<TPixel>(this Image<TPixel> img) where TPixel : unmanaged, IPixel<TPixel>
19+
{
20+
var memoryGroup = img.GetPixelMemoryGroup();
21+
var memory = memoryGroup.ToArray()[0];
22+
var pixelData = System.Runtime.InteropServices.MemoryMarshal.AsBytes(memory.Span).ToArray(); // Can we get rid of this?
23+
var buffer = pixelData.AsBuffer();
24+
return buffer;
25+
}
26+
27+
public static ITensor AsTensor<TPixel>(this Image<TPixel> img) where TPixel : unmanaged, IPixel<TPixel>
28+
{
29+
var buffer = img.AsBuffer();
30+
var shape = new long[] { 1, buffer.Length };
31+
var tensor = TensorUInt8Bit.CreateFromBuffer(shape, buffer);
32+
return tensor;
33+
}
34+
35+
36+
public static SoftwareBitmap AsSoftwareBitmap<TPixel>(this Image<TPixel> img) where TPixel : unmanaged, IPixel<TPixel>
37+
{
38+
var buffer = img.AsBuffer();
39+
var format = BitmapPixelFormat.Unknown;
40+
if (typeof(TPixel) == typeof(Bgra32))
41+
{
42+
format = BitmapPixelFormat.Bgra8;
43+
}
44+
45+
var softwareBitmap = SoftwareBitmap.CreateCopyFromBuffer(buffer, BitmapPixelFormat.Bgra8, img.Width, img.Height);
46+
return softwareBitmap;
47+
}
48+
}
49+
}

‎Samples/WinMLSamplesGallery/WinMLSamplesGallery/Samples/ImageSharpInterop/ImageSharpInterop.xaml‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
<Page
2+
x:Class="WinMLSamplesGallery.Samples.ImageSharpInterop"
3+
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
4+
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
5+
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
6+
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
7+
xmlns:local_controls="using:WinMLSamplesGallery.Controls"
8+
xmlns:local_samples="using:WinMLSamplesGallery.Samples"
9+
mc:Ignorable="d"
10+
11+
Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
12+
13+
<Page.Resources>
14+
<DataTemplate x:Key="ImageTemplate" x:DataType="local_controls:Thumbnail">
15+
<Grid Width="200" Height="204">
16+
<Image Stretch="UniformToFill" Source="{x:Bind ImageUri}" HorizontalAlignment="Center" VerticalAlignment="Center" Height="204"/>
17+
</Grid>
18+
</DataTemplate>
19+
20+
<DataTemplate x:Name="InferenceResultsTemplate" x:DataType="local_controls:Prediction">
21+
<StackPanel Orientation="Horizontal">
22+
<TextBlock Width="414"
23+
FontSize="14"
24+
Foreground="Black"
25+
Padding="0,1,1,1"
26+
Typography.Capitals="AllSmallCaps"
27+
Typography.StylisticSet4="True"
28+
TextTrimming="CharacterEllipsis">
29+
<Run Text="[" />
30+
<Run Text="{Binding Index}" />
31+
<Run Text="] " />
32+
<Run Text="{Binding Name}" />
33+
</TextBlock>
34+
<TextBlock Width="120"
35+
FontSize="14"
36+
Foreground="Black"
37+
Padding="0,1,1,1"
38+
Typography.Capitals="AllSmallCaps"
39+
Typography.StylisticSet4="True">
40+
<Run Text="p =" />
41+
<Run Text="{Binding Probability}" />
42+
</TextBlock>
43+
</StackPanel>
44+
</DataTemplate>
45+
46+
<DataTemplate x:Name="AllModelsTemplate" x:DataType="local_samples:ClassifierViewModel">
47+
<Grid Background="#e6e6e6" BorderBrush="#12bef6" BorderThickness="1">
48+
<TextBlock FontSize="14" Text="{x:Bind Title}"
49+
Typography.Capitals="AllSmallCaps"
50+
Typography.StylisticSet4="True"
51+
VerticalAlignment="Top"
52+
Padding="10,2,10,2"
53+
MinWidth="136"
54+
/>
55+
</Grid>
56+
</DataTemplate>
57+
</Page.Resources>
58+
59+
<Grid>
60+
<ScrollViewer
61+
ZoomMode="Disabled"
62+
IsVerticalScrollChainingEnabled="True"
63+
HorizontalScrollMode="Enabled" HorizontalScrollBarVisibility="Disabled"
64+
VerticalScrollMode="Enabled" VerticalScrollBarVisibility="Visible">
65+
<Grid>
66+
<Grid.RowDefinitions>
67+
<RowDefinition Height="*" />
68+
<RowDefinition Height="Auto" />
69+
<RowDefinition Height="*" />
70+
</Grid.RowDefinitions>
71+
72+
<StackPanel Grid.Row="0" Orientation="Horizontal" Padding="0,10,0,0">
73+
<StackPanel Orientation="Vertical">
74+
<StackPanel Orientation="Horizontal" HorizontalAlignment="Left">
75+
<Button FontFamily="Segoe MDL2 Assets" Content="&#xE1A5;" Width="97" Height="50" HorizontalAlignment="Left" Click="OpenButton_Clicked" />
76+
<Grid Padding="5,0,0,0">
77+
<ComboBox x:Name="DeviceComboBox" SelectedIndex="0" Background="LightGray" PlaceholderText="Device" Height="50" Width="97"
78+
SelectionChanged="DeviceComboBox_SelectionChanged">
79+
<TextBlock Text="CPU" FontSize="18" Typography.Capitals="AllSmallCaps" Typography.StylisticSet4="True"/>
80+
<TextBlock Text="DML" FontSize="18" Typography.Capitals="AllSmallCaps" Typography.StylisticSet4="True"/>
81+
</ComboBox>
82+
</Grid>
83+
</StackPanel>
84+
<GridView
85+
x:Name="BasicGridView"
86+
ItemTemplate="{StaticResource ImageTemplate}"
87+
IsItemClickEnabled="True"
88+
SelectionChanged="SampleInputsGridView_SelectionChanged"
89+
SelectionMode="Single"
90+
Padding="0,6,0,0"
91+
HorizontalAlignment="Center">
92+
<GridView.ItemsPanel>
93+
<ItemsPanelTemplate>
94+
<StackPanel Orientation="Vertical" />
95+
</ItemsPanelTemplate>
96+
</GridView.ItemsPanel>
97+
<GridView.Items>
98+
<local_controls:Thumbnail ImageUri="ms-appx:///InputData/kitten.png" />
99+
</GridView.Items>
100+
</GridView>
101+
</StackPanel>
102+
103+
<Slider x:Name="RotationSlider" IsEnabled="False" ValueChanged="RotationSlider_ValueChanged" Orientation="Vertical" TickFrequency="60" TickPlacement="Outside" Maximum="360" Minimum="0"/>
104+
<Border BorderBrush="LightGray" Padding="5,0,0,0">
105+
<Image x:Name="InputImage" Stretch="UniformToFill" Height="260" HorizontalAlignment="Center"/>
106+
</Border>
107+
</StackPanel>
108+
109+
<StackPanel Orientation="Horizontal" Grid.Row="2"
110+
Padding="0,7,0,0">
111+
<ListView
112+
x:Name="InferenceResults"
113+
HorizontalAlignment="Stretch"
114+
Padding="0,2,0,0"
115+
ItemTemplate="{StaticResource InferenceResultsTemplate}"
116+
IsItemClickEnabled="False"
117+
SingleSelectionFollowsFocus="False">
118+
<ListView.ItemContainerStyle>
119+
<Style TargetType="ListViewItem">
120+
<Setter Property="Margin" Value="1,1,1,1"/>
121+
<Setter Property="MinHeight" Value="0"/>
122+
</Style>
123+
</ListView.ItemContainerStyle>
124+
125+
<ListView.ItemsPanel>
126+
<ItemsPanelTemplate>
127+
<ItemsWrapGrid x:Name="MaxItemsWrapGrid" Orientation="Vertical" HorizontalAlignment="Stretch"/>
128+
</ItemsPanelTemplate>
129+
</ListView.ItemsPanel>
130+
</ListView>
131+
<local_controls:PerformanceMonitor x:Name="PerformanceMetricsMonitor"/>
132+
</StackPanel>
133+
</Grid>
134+
</ScrollViewer>
135+
</Grid>
136+
</Page>

0 commit comments

Comments
 (0)