Skip to content
This repository was archived by the owner on Mar 5, 2022. It is now read-only.

Commit bcd42e8

Browse files
author
Gabriel Weyer
committed
Merge branch 'features/support-blobs'
2 parents 80629da + b5bfcd2 commit bcd42e8

30 files changed

+822
-63
lines changed

‎README.md‎

Lines changed: 53 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,11 @@
1-
# Windows Azure Diagnostics to CSV
1+
# Azure Diagnostics to CSV
22

3-
`.NET Core` [global tool][dotnet-global-tools] to retrieve logs from a `WADLogsTable` and write them to a `CSV` file.
4-
5-
The tool will attempt to obfuscate settings that have been logged due to this [issue][github-issue] in the [Microsoft.WindowsAzure.ConfigurationManager][configuration-manager-nuget].
3+
`.NET Core` [global tools][dotnet-global-tools] to retrieve Azure Diagnostics logs from a `WADLogsTable` or blobs and write them to a `CSV` file.
64

75
| Package | Release | Pre-release |
86
| --- | --- | --- |
9-
| `dotnet-wad-to-csv` | [![NuGet][nuget-tool-badge]][nuget-tool-command] | [![MyGet][myget-tool-badge]][myget-tool-command] |
7+
| `dotnet-wad-to-csv` | [![NuGet][nuget-wad-tool-badge]][nuget-wad-tool-command] | [![MyGet][myget-wad-tool-badge]][myget-wad-tool-command] |
8+
| `dotnet-blob-to-csv` | [![NuGet][nuget-blob-tool-badge]][nuget-blob-tool-command] | [![MyGet][myget-blob-tool-badge]][myget-blob-tool-command] |
109

1110
| CI | Status | Platform(s) | Framework(s) |
1211
| --- | --- | --- | --- |
@@ -18,7 +17,11 @@ The tool will attempt to obfuscate settings that have been logged due to this [i
1817
> dotnet tool install -g dotnet-wad-to-csv
1918
```
2019

21-
## Usage
20+
```posh
21+
> dotnet tool install -g dotnet-blob-to-csv
22+
```
23+
24+
## WAD to CSV Usage
2225

2326
```posh
2427
> dotnet wad-to-csv -l <last> -o <output-file-path>
@@ -49,6 +52,41 @@ The tool will prompt you for a [Shared Access Signature][sas] so that it doesn't
4952
- `Container` and `Object` resource types
5053
- A short expiration time
5154

55+
`WAD to CSV` will attempt to obfuscate settings that have been logged due to this [issue][github-issue] in the [Microsoft.WindowsAzure.ConfigurationManager][configuration-manager-nuget].
56+
57+
## Blob to CSV Usage
58+
59+
```posh
60+
> dotnet blob-to-csv -l <last> -o <output-file-path> -c <container> -p <prefix>
61+
```
62+
63+
```posh
64+
> dotnet blob-to-csv -f <from> -o <output-file-path> -c <container> -p <prefix>
65+
```
66+
67+
```posh
68+
> dotnet blob-to-csv -f <from> -t <to> -o <output-file-path> -c <container> -p <prefix>
69+
```
70+
71+
- `<last>`: based on the `time designator` of the [ISO 8601 durations][iso-8601-duration]. This duration is then substracted from the current `UTC` time. For example:
72+
- `5M`: get all the logs for the last `5` **minutes**
73+
- `1H`: get all the logs for the last `1` **hour**
74+
- You can combine them too if you feel like it, i.e. `2H3M5S` would get all the logs for the last `7385` **seconds**
75+
- `<from>`: `ISO 8601 date time` expressed in `UTC`. Cannot be combined with `<last>`, can be combined with `<to>`.
76+
- Valid date time: `2018-06-24T23:12:15`
77+
- The time component can be omitted: `2018-06-24`
78+
- `<to>`: `ISO 8601 date time` expressed in `UTC`. Must be be combined with `<from>`.
79+
- `<output-file-path>`: where you wish to write the output file, does not need to exist but should be valid. If a file exists with the same name it will be replaced.
80+
- `<container>`: The name of the container. For `https://account.blob.core.windows.net/container-name/prefix/2018/06/22/00/e872fe-54660.applicationLog.csv`, the container name is `container-name`.
81+
- `<prefix>`: The prefix (if any). For `https://account.blob.core.windows.net/container-name/prefix/2018/06/22/00/e872fe-54660.applicationLog.csv`, the prefix is `prefix`.
82+
83+
The tool will prompt you for a [Shared Access Signature][sas] so that it doesn't get saved to your `CLI` history. I recommend you restrict the `SAS` to:
84+
85+
- `Read` and `List` permissions
86+
- `Blobs` service
87+
- `Container` and `Object` resource types
88+
- A short expiration time
89+
5290
## Output file format
5391

5492
```csv
@@ -57,7 +95,7 @@ Generated,Level,Message
5795
```
5896

5997
- `Generated` is expressed in `UTC`
60-
- The `Level` is converted from the `Level` column:
98+
- The `Level` is written as-is for blobs and converted from the `Level` column for the `WADLogsTable`:
6199
- `1`: `Fatal`
62100
- `2`: `Error`
63101
- `3`: `Warning`
@@ -73,8 +111,12 @@ Generated,Level,Message
73111
[github-issue]: https://github.com/Azure/azure-sdk-for-net/issues/653
74112
[app-veyor]: https://ci.appveyor.com/project/GabrielWeyer/dotnet-wad-to-csv
75113
[app-veyor-shield]: https://img.shields.io/appveyor/ci/gabrielweyer/dotnet-wad-to-csv/master.svg?label=AppVeyor&style=flat-square
76-
[nuget-tool-badge]: https://img.shields.io/nuget/v/dotnet-wad-to-csv.svg?label=NuGet&style=flat-square
77-
[nuget-tool-command]: https://www.nuget.org/packages/dotnet-wad-to-csv
78-
[myget-tool-badge]: https://img.shields.io/myget/gabrielweyer-pre-release/v/dotnet-wad-to-csv.svg?label=MyGet&style=flat-square
79-
[myget-tool-command]: https://www.myget.org/feed/gabrielweyer-pre-release/package/nuget/dotnet-wad-to-csv
114+
[nuget-wad-tool-badge]: https://img.shields.io/nuget/v/dotnet-wad-to-csv.svg?label=NuGet&style=flat-square
115+
[nuget-wad-tool-command]: https://www.nuget.org/packages/dotnet-wad-to-csv
116+
[myget-wad-tool-badge]: https://img.shields.io/myget/gabrielweyer-pre-release/v/dotnet-wad-to-csv.svg?label=MyGet&style=flat-square
117+
[myget-wad-tool-command]: https://www.myget.org/feed/gabrielweyer-pre-release/package/nuget/dotnet-wad-to-csv
80118
[configuration-manager-nuget]: https://www.nuget.org/packages/Microsoft.WindowsAzure.ConfigurationManager/
119+
[nuget-blob-tool-badge]: https://img.shields.io/nuget/v/dotnet-blob-to-csv.svg?label=NuGet&style=flat-square
120+
[nuget-blob-tool-command]: https://www.nuget.org/packages/dotnet-blob-to-csv
121+
[myget-blob-tool-badge]: https://img.shields.io/myget/gabrielweyer-pre-release/v/dotnet-blob-to-csv.svg?label=MyGet&style=flat-square
122+
[myget-blob-tool-command]: https://www.myget.org/feed/gabrielweyer-pre-release/package/nuget/dotnet-blob-to-csv

‎dotnet-wad-to-csv.sln‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{99CDCAF2
1111
EndProject
1212
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "dotnet-wad-to-csv-tests", "tests\dotnet-wad-to-csv-tests\dotnet-wad-to-csv-tests.csproj", "{3C3806FE-50DC-472E-931F-ABFED71B8DD9}"
1313
EndProject
14+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "dotnet-blob-to-csv", "src\dotnet-blob-to-csv\dotnet-blob-to-csv.csproj", "{B3B45D11-AEB7-4423-BA6A-35337B2C1065}"
15+
EndProject
16+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "dotnet-ad-core", "src\dotnet-ad-core\dotnet-ad-core.csproj", "{46FF6147-017C-4511-8FD4-4003C91905CA}"
17+
EndProject
1418
Global
1519
GlobalSection(SolutionConfigurationPlatforms) = preSolution
1620
Debug|Any CPU = Debug|Any CPU
@@ -48,9 +52,35 @@ Global
4852
{3C3806FE-50DC-472E-931F-ABFED71B8DD9}.Release|x64.Build.0 = Release|Any CPU
4953
{3C3806FE-50DC-472E-931F-ABFED71B8DD9}.Release|x86.ActiveCfg = Release|Any CPU
5054
{3C3806FE-50DC-472E-931F-ABFED71B8DD9}.Release|x86.Build.0 = Release|Any CPU
55+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
56+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Debug|Any CPU.Build.0 = Debug|Any CPU
57+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Debug|x64.ActiveCfg = Debug|Any CPU
58+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Debug|x64.Build.0 = Debug|Any CPU
59+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Debug|x86.ActiveCfg = Debug|Any CPU
60+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Debug|x86.Build.0 = Debug|Any CPU
61+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Release|Any CPU.ActiveCfg = Release|Any CPU
62+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Release|Any CPU.Build.0 = Release|Any CPU
63+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Release|x64.ActiveCfg = Release|Any CPU
64+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Release|x64.Build.0 = Release|Any CPU
65+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Release|x86.ActiveCfg = Release|Any CPU
66+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065}.Release|x86.Build.0 = Release|Any CPU
67+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
68+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Debug|Any CPU.Build.0 = Debug|Any CPU
69+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Debug|x64.ActiveCfg = Debug|Any CPU
70+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Debug|x64.Build.0 = Debug|Any CPU
71+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Debug|x86.ActiveCfg = Debug|Any CPU
72+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Debug|x86.Build.0 = Debug|Any CPU
73+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Release|Any CPU.ActiveCfg = Release|Any CPU
74+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Release|Any CPU.Build.0 = Release|Any CPU
75+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Release|x64.ActiveCfg = Release|Any CPU
76+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Release|x64.Build.0 = Release|Any CPU
77+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Release|x86.ActiveCfg = Release|Any CPU
78+
{46FF6147-017C-4511-8FD4-4003C91905CA}.Release|x86.Build.0 = Release|Any CPU
5179
EndGlobalSection
5280
GlobalSection(NestedProjects) = preSolution
5381
{5F2DBADB-1A7D-43A7-88F2-698D96748A79} = {FEED035D-890B-4A6C-8418-822766086DDF}
5482
{3C3806FE-50DC-472E-931F-ABFED71B8DD9} = {99CDCAF2-373F-4334-8CE6-05B1566BEF97}
83+
{B3B45D11-AEB7-4423-BA6A-35337B2C1065} = {FEED035D-890B-4A6C-8418-822766086DDF}
84+
{46FF6147-017C-4511-8FD4-4003C91905CA} = {FEED035D-890B-4A6C-8418-822766086DDF}
5585
EndGlobalSection
5686
EndGlobal

‎src/dotnet-ad-core/Helpers/ConsoleHelper.cs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
using System;
2+
3+
namespace DotNet.AzureDiagnostics.Core.Helpers
4+
{
5+
public class ConsoleHelper
6+
{
7+
public static void WriteDebug(string line)
8+
{
9+
Console.ForegroundColor = ConsoleColor.Cyan;
10+
Console.WriteLine(line);
11+
Console.ResetColor();
12+
}
13+
14+
public static void WriteError(string line)
15+
{
16+
Console.ForegroundColor = ConsoleColor.Red;
17+
Console.WriteLine(line);
18+
Console.ResetColor();
19+
}
20+
}
21+
}

‎src/dotnet-ad-core/Helpers/StorageAccountHelper.cs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
namespace DotNet.AzureDiagnostics.Core.Helpers
2+
{
3+
public static class StorageAccountHelper
4+
{
5+
public static string GetStorageAccountName(string connectionString)
6+
{
7+
var lastIndex = connectionString.LastIndexOf(':');
8+
9+
if (lastIndex == -1) return null;
10+
11+
var storageAccountUri = connectionString.Substring(lastIndex + 3);
12+
13+
var firstindex = storageAccountUri.IndexOf('.');
14+
15+
return firstindex == -1 ? null : storageAccountUri.Substring(0, firstindex);
16+
}
17+
}
18+
}

src/dotnet-wad-to-csv/Models/Range.cs renamed to src/dotnet-ad-core/Models/Range.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
using System;
22

3-
namespace DotNet.WadToCsv.Models
3+
namespace DotNet.AzureDiagnostics.Core.Models
44
{
55
public class Range
66
{

src/dotnet-wad-to-csv/DateTimeParser.cs renamed to src/dotnet-ad-core/Parsers/DateTimeParser.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
using System;
22
using System.Globalization;
33

4-
namespace DotNet.WadToCsv
4+
namespace DotNet.AzureDiagnostics.Core.Parsers
55
{
66
public static class DateTimeParser
77
{

src/dotnet-wad-to-csv/DurationParser.cs renamed to src/dotnet-ad-core/Parsers/DurationParser.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
using System;
22
using System.Text.RegularExpressions;
33

4-
namespace DotNet.WadToCsv
4+
namespace DotNet.AzureDiagnostics.Core.Parsers
55
{
66
public static class DurationParser
77
{

src/dotnet-wad-to-csv/RangeParser.cs renamed to src/dotnet-ad-core/Parsers/RangeParser.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
using System;
22
using System.ComponentModel.DataAnnotations;
3-
using DotNet.WadToCsv.Models;
3+
using DotNet.AzureDiagnostics.Core.Models;
44

5-
namespace DotNet.WadToCsv
5+
namespace DotNet.AzureDiagnostics.Core.Parsers
66
{
77
public static class RangeParser
88
{
@@ -21,7 +21,7 @@ public static ValidationResult TryGetRange(string last, string from, string to,
2121
return new ValidationResult("'--to' can only be used in conjonction with '--from'.");
2222
}
2323

24-
var rangeFrom = DateTime.UtcNow;
24+
DateTime rangeFrom;
2525

2626
if (string.IsNullOrEmpty(from))
2727
{

src/dotnet-wad-to-csv/Validation/Iso8601DateTimeAttribute.cs renamed to src/dotnet-ad-core/Validation/Iso8601DateTimeAttribute.cs

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
using System.ComponentModel.DataAnnotations;
2+
using DotNet.AzureDiagnostics.Core.Parsers;
23

3-
namespace DotNet.WadToCsv.Validation
4+
namespace DotNet.AzureDiagnostics.Core.Validation
45
{
56
public class Iso8601DateTimeAttribute : ValidationAttribute
67
{

src/dotnet-wad-to-csv/Validation/Iso8601TimeDurationAttribute.cs renamed to src/dotnet-ad-core/Validation/Iso8601TimeDurationAttribute.cs

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
using System;
22
using System.ComponentModel.DataAnnotations;
3+
using DotNet.AzureDiagnostics.Core.Parsers;
34

4-
namespace DotNet.WadToCsv.Validation
5+
namespace DotNet.AzureDiagnostics.Core.Validation
56
{
67
public class Iso8601TimeDurationAttribute : ValidationAttribute
78
{

0 commit comments

Comments
 (0)