Skip to content

Commit fd029a2

Browse files
committed
Merge branch 'broadcast-refactor' - NumPy 2.x aligned Shape architecture (#538)
2 parents 191c914 + a217477 commit fd029a2

File tree

367 files changed

+67264
-46186
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

367 files changed

+67264
-46186
lines changed

.claude/CLAUDE.md

Lines changed: 151 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -44,9 +44,44 @@ np Static API class (like `import numpy as np`)
4444
| Decision | Rationale |
4545
|----------|-----------|
4646
| Unmanaged memory | Benchmarked fastest ~5y ago; Span/Memory immature then |
47+
| C-order only | Only row-major (C-order) memory layout. Uses `ArrayFlags.C_CONTIGUOUS` flag. No F-order/column-major support. The `order` parameter on `ravel`, `flatten`, `copy`, `reshape` is accepted but ignored. |
4748
| Regen templating | ~200K lines generated for type-specific code |
4849
| TensorEngine abstract | Future GPU/SIMD backends possible |
4950
| View semantics | Slicing returns views (shared memory), not copies |
51+
| Shape readonly struct | Immutable after construction (NumPy-aligned). Contains `ArrayFlags` for cached O(1) property access |
52+
| Broadcast write protection | Broadcast views are read-only (`IsWriteable = false`), matching NumPy behavior |
53+
54+
## Shape Architecture (NumPy-Aligned)
55+
56+
Shape is a `readonly struct` with cached `ArrayFlags` computed at construction:
57+
58+
```csharp
59+
public readonly partial struct Shape
60+
{
61+
internal readonly int[] dimensions; // Dimension sizes
62+
internal readonly int[] strides; // Stride values (0 = broadcast dimension)
63+
internal readonly int offset; // Base offset into storage
64+
internal readonly int bufferSize; // Size of underlying buffer
65+
internal readonly int _flags; // Cached ArrayFlags bitmask
66+
}
67+
```
68+
69+
**ArrayFlags enum** (matches NumPy's `ndarraytypes.h`):
70+
| Flag | Value | Meaning |
71+
|------|-------|---------|
72+
| `C_CONTIGUOUS` | 0x0001 | Data is row-major contiguous |
73+
| `F_CONTIGUOUS` | 0x0002 | Reserved (always false for NumSharp) |
74+
| `OWNDATA` | 0x0004 | Array owns its data buffer |
75+
| `ALIGNED` | 0x0100 | Always true for managed allocations |
76+
| `WRITEABLE` | 0x0400 | False for broadcast views |
77+
| `BROADCASTED` | 0x1000 | Has stride=0 with dim > 1 |
78+
79+
**Key Shape properties:**
80+
- `IsContiguous` — O(1) check via `C_CONTIGUOUS` flag
81+
- `IsBroadcasted` — O(1) check via `BROADCASTED` flag
82+
- `IsWriteable` — False for broadcast views (prevents corruption)
83+
- `IsSliced` — True if offset != 0, different size, or non-contiguous
84+
- `IsSimpleSlice` — IsSliced && !IsBroadcasted (fast offset path)
5085

5186
## Critical: View Semantics
5287

@@ -109,9 +144,12 @@ nd["..., -1"] // Ellipsis fills dimensions
109144
### Math Functions (`Math/`)
110145
| Function | File |
111146
|----------|------|
147+
| `np.add`, `np.subtract`, `np.multiply`, `np.divide` | `np.math.cs` |
148+
| `np.mod`, `np.true_divide` | `np.math.cs` |
149+
| `np.positive`, `np.negative`, `np.convolve` | `np.math.cs` |
112150
| `np.sum` | `np.sum.cs` |
113-
| `np.prod` | `NDArray.prod.cs` |
114-
| `np.cumsum` | `NDArray.cumsum.cs` |
151+
| `np.prod`, `nd.prod()` | `np.math.cs`, `NDArray.prod.cs` |
152+
| `np.cumsum`, `nd.cumsum()` | `APIs/np.cumsum.cs`, `Math/NDArray.cumsum.cs` |
115153
| `np.power` | `np.power.cs` |
116154
| `np.sqrt` | `np.sqrt.cs` |
117155
| `np.abs`, `np.absolute` | `np.absolute.cs` |
@@ -131,9 +169,9 @@ nd["..., -1"] // Ellipsis fills dimensions
131169
| `np.mean`, `nd.mean()` | `np.mean.cs`, `NDArray.mean.cs` |
132170
| `np.std`, `nd.std()` | `np.std.cs`, `NDArray.std.cs` |
133171
| `np.var`, `nd.var()` | `np.var.cs`, `NDArray.var.cs` |
134-
| `np.amax`, `nd.amax()` | `Sorting/np.amax.cs`, `NDArray.amax.cs` |
135-
| `np.amin`, `nd.amin()` | `Sorting/np.min.cs`, `NDArray.amin.cs` |
136-
| `np.argmax`, `nd.argmax()` | `Sorting/np.argmax.cs`, `NDArray.argmax.cs` |
172+
| `np.amax`, `nd.amax()` | `Sorting_Searching_Counting/np.amax.cs`, `NDArray.amax.cs` |
173+
| `np.amin`, `nd.amin()` | `Sorting_Searching_Counting/np.min.cs`, `NDArray.amin.cs` |
174+
| `np.argmax`, `nd.argmax()` | `Sorting_Searching_Counting/np.argmax.cs`, `NDArray.argmax.cs` |
137175
| `np.argmin`, `nd.argmin()` | `Sorting_Searching_Counting/np.argmax.cs`, `NDArray.argmin.cs` |
138176

139177
### Sorting & Searching (`Sorting_Searching_Counting/`)
@@ -168,7 +206,7 @@ nd["..., -1"] // Ellipsis fills dimensions
168206
| `np.swapaxes` | `np.swapaxes.cs`, `NdArray.swapaxes.cs` |
169207
| `np.moveaxis` | `np.moveaxis.cs` |
170208
| `np.rollaxis` | `np.rollaxis.cs` |
171-
| `nd.roll()` | `NDArray.roll.cs` | Partial: only Int32/Single/Double with axis; no-axis returns null |
209+
| `np.roll`, `nd.roll()` | `np.roll.cs`, `NDArray.roll.cs` | Fully implemented (all dtypes, with/without axis) |
172210
| `np.atleast_1d/2d/3d` | `np.atleastd.cs` |
173211
| `np.unique`, `nd.unique()` | `np.unique.cs`, `NDArray.unique.cs` |
174212
| `np.repeat` | `np.repeat.cs` |
@@ -244,7 +282,6 @@ nd["..., -1"] // Ellipsis fills dimensions
244282
| Function | File |
245283
|----------|------|
246284
| `np.size` | `np.size.cs` |
247-
| `np.cumsum` | `np.cumsum.cs` |
248285

249286
---
250287

@@ -334,22 +371,97 @@ Create issues on `SciSharp/NumSharp` via `gh issue create`. `GH_TOKEN` is availa
334371
## Build & Test
335372

336373
```bash
374+
# Build (silent, errors only)
337375
dotnet build -v q --nologo "-clp:NoSummary;ErrorsOnly" -p:WarningLevel=0
338-
dotnet test -v q --nologo "-clp:ErrorsOnly" test/NumSharp.UnitTest/NumSharp.UnitTest.csproj
376+
```
377+
378+
### Running Tests
379+
380+
Tests use **TUnit** framework with source-generated test discovery.
381+
382+
```bash
383+
# Run from test directory
384+
cd test/NumSharp.UnitTest
385+
386+
# All tests (includes OpenBugs - expected failures)
387+
dotnet test --no-build
388+
389+
# Exclude OpenBugs (CI-style - only real failures)
390+
dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category!=OpenBugs]"
391+
392+
# Run ONLY OpenBugs tests
393+
dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
394+
```
395+
396+
### Output Formatting
397+
398+
```bash
399+
# Results only (no messages, no stack traces)
400+
dotnet test --no-build 2>&1 | grep -E "^(failed|skipped|Test run| total:| failed:| succeeded:| skipped:| duration:)"
401+
402+
# Results with messages (no stack traces)
403+
dotnet test --no-build 2>&1 | grep -v "^ at " | grep -v "^ at " | grep -v "^ ---" | grep -v "^ from K:" | sed 's/TUnit.Engine.Exceptions.TestFailedException: //' | sed 's/AssertFailedException: //'
404+
405+
# Detailed output (shows passed tests too)
406+
dotnet test --no-build -- --output Detailed
339407
```
340408

341409
## Test Categories
342410

343-
Tests are filtered by `[TestCategory]` attributes. Adding new bug reproductions or platform-specific tests only requires the right attribute — no CI workflow changes.
411+
Tests use typed category attributes defined in `TestCategory.cs`. Adding new bug reproductions or platform-specific tests only requires the right attribute — no CI workflow changes.
412+
413+
| Category | Attribute | Purpose | CI Behavior |
414+
|----------|-----------|---------|-------------|
415+
| `OpenBugs` | `[OpenBugs]` | Known-failing bug reproductions. Remove when fixed. | **EXCLUDED** via filter |
416+
| `Misaligned` | `[Misaligned]` | Documents NumSharp vs NumPy behavioral differences. | Runs (tests pass) |
417+
| `WindowsOnly` | `[WindowsOnly]` | Requires GDI+/System.Drawing.Common | Runtime platform check |
418+
419+
### How CI Excludes OpenBugs
420+
421+
The CI pipeline (`.github/workflows/build-and-release.yml`) uses TUnit's `--treenode-filter` to exclude `OpenBugs`:
344422

345-
| Category | Purpose | CI filter |
346-
|----------|---------|-----------|
347-
| `OpenBugs` | Known-failing bug reproductions. Remove category when fixed. | `TestCategory!=OpenBugs` (all platforms) |
348-
| `WindowsOnly` | Requires GDI+/System.Drawing.Common | `TestCategory!=WindowsOnly` (Linux/macOS) |
423+
```yaml
424+
env:
425+
TEST_FILTER: '/*/*/*/*[Category!=OpenBugs]'
349426

350-
Apply at class level (`[TestClass][TestCategory("OpenBugs")]`) or individual method level (`[TestMethod][TestCategory("OpenBugs")]`).
427+
- name: Test
428+
run: dotnet run ... -- --treenode-filter "${{ env.TEST_FILTER }}"
429+
```
430+
431+
This filter excludes all tests with `[OpenBugs]` or `[Category("OpenBugs")]` from CI runs. Tests pass locally when the bug is fixed — then remove the `[OpenBugs]` attribute.
432+
433+
### Usage
351434

352-
**OpenBugs files**: `OpenBugs.cs` (broadcast bugs), `OpenBugs.Bitmap.cs` (bitmap bugs). When a bug is fixed, the test starts passing — remove the `OpenBugs` category and move to a permanent test class.
435+
```csharp
436+
// Class-level (all tests in class)
437+
[OpenBugs]
438+
public class BroadcastBugTests { ... }
439+
440+
// Method-level
441+
[Test]
442+
[OpenBugs]
443+
public async Task BroadcastWriteCorruptsData() { ... }
444+
445+
// Documenting behavioral differences (NOT excluded from CI)
446+
[Test]
447+
[Misaligned]
448+
public void BroadcastSlice_MaterializesInNumSharp() { ... }
449+
```
450+
451+
### Local Filtering
452+
453+
```bash
454+
# Exclude OpenBugs (same as CI)
455+
dotnet test -- --treenode-filter "/*/*/*/*[Category!=OpenBugs]"
456+
457+
# Run ONLY OpenBugs tests (to verify fixes)
458+
dotnet test -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
459+
460+
# Run ONLY Misaligned tests
461+
dotnet test -- --treenode-filter "/*/*/*/*[Category=Misaligned]"
462+
```
463+
464+
**OpenBugs files**: `OpenBugs.cs` (general bugs), `OpenBugs.Bitmap.cs` (bitmap bugs), `OpenBugs.ApiAudit.cs` (API audit failures).
353465

354466
## CI Pipeline
355467

@@ -387,9 +499,14 @@ NumSharp uses unsafe in many places, hence include `#:property AllowUnsafeBlocks
387499
|--------|----------------|
388500
| `shape.dimensions` | Raw int[] of dimension sizes |
389501
| `shape.strides` | Raw int[] of stride values |
390-
| `shape.size` | Total element count |
391-
| `shape.ViewInfo` | Slice/view metadata (null if not a view) |
392-
| `shape.BroadcastInfo` | Broadcast metadata (null if not broadcast) |
502+
| `shape.size` | Internal field: total element count |
503+
| `shape.offset` | Base offset into storage (NumPy-aligned) |
504+
| `shape.bufferSize` | Size of underlying buffer |
505+
| `shape._flags` | Cached ArrayFlags bitmask |
506+
| `shape.IsWriteable` | False for broadcast views (NumPy behavior) |
507+
| `shape.IsBroadcasted` | Has any stride=0 with dimension > 1 |
508+
| `shape.IsSimpleSlice` | IsSliced && !IsBroadcasted |
509+
| `shape.OriginalSize` | Product of non-broadcast dimensions |
393510
| `arr.Storage` | Underlying `UnmanagedStorage` |
394511
| `arr.GetTypeCode` | `NPTypeCode` of the array |
395512
| `arr.Array` | `IArraySlice` — raw data access |
@@ -399,7 +516,18 @@ NumSharp uses unsafe in many places, hence include `#:property AllowUnsafeBlocks
399516
| `NPTypeCode.GetPriority()` | Type priority for promotion |
400517
| `NPTypeCode.AsNumpyDtypeName()` | NumPy dtype name (e.g. "int32") |
401518
| `Shape.NewScalar()` | Create scalar shapes |
402-
| `Shape.ComputeHashcode()` | Recalculate shape hash |
519+
520+
### Common Public NDArray Properties
521+
522+
| Property | Description |
523+
|----------|-------------|
524+
| `nd.shape` | Dimensions as `int[]` |
525+
| `nd.ndim` | Number of dimensions |
526+
| `nd.size` | Total element count |
527+
| `nd.dtype` | Element type as `Type` |
528+
| `nd.typecode` | Element type as `NPTypeCode` |
529+
| `nd.T` | Transpose (swaps axes) |
530+
| `nd.flat` | 1D iterator over elements |
403531

404532
## Adding New Features
405533

@@ -433,7 +561,7 @@ A: Yes, 1-to-1 matching.
433561
A: Anything that can use the capabilities - porting Python ML code, standalone .NET scientific computing, integration with TensorFlow.NET/ML.NET.
434562

435563
**Q: Are there areas of known fragility?**
436-
A: Slicing/broadcasting system is complex with ViewInfo and BroadcastInfo interactions - fragile but working.
564+
A: Slicing/broadcasting system is complex — offset/stride calculations with contiguity detection require careful handling. The `readonly struct Shape` with `ArrayFlags` simplifies this but edge cases remain.
437565

438566
**Q: How is NumPy compatibility validated?**
439567
A: Written by hand based on NumPy docs and original tests. Testing philosophy: run actual NumPy code, observe output, replicate 1-to-1 in C#.
@@ -455,7 +583,7 @@ A: Implementations that differ from original NumPy 2.x behavior. A comprehensive
455583
A: `NDArray` (user-facing API), `UnmanagedStorage` (raw memory management), and `Shape` (dimensions, strides, coordinate translation). They work together: NDArray wraps Storage which uses Shape for offset calculations.
456584

457585
**Q: What is Shape responsible for?**
458-
A: Dimensions, strides, coordinate-to-offset translation, contiguity tracking, and slice/broadcast info. Key properties: `IsScalar`, `IsContiguous`, `IsSliced`, `IsBroadcasted`. Methods: `GetOffset(coords)`, `GetCoordinates(offset)`.
586+
A: Shape is a `readonly struct` containing dimensions, strides, offset, bufferSize, and cached `ArrayFlags`. Key properties: `IsScalar`, `IsContiguous`, `IsSliced`, `IsBroadcasted`, `IsWriteable`, `IsSimpleSlice`. Methods: `GetOffset(coords)`, `GetCoordinates(offset)`. NumPy-aligned: broadcast views are read-only (`IsWriteable = false`).
459587

460588
**Q: How does slicing work internally?**
461589
A: The `Slice` class parses Python notation (e.g., "1:5:2") into `Start`, `Stop`, `Step`. It converts to `SliceDef` (absolute indices) for computation. `SliceDef.Merge()` handles recursive slicing (slice of a slice).
@@ -502,10 +630,10 @@ A: Core ops (`dot`, `matmul`) in `LinearAlgebra/`. Advanced decompositions (`inv
502630
## Q&A - Development
503631

504632
**Q: What's in the test suite?**
505-
A: MSTest framework in `test/NumSharp.UnitTest/`. Many tests adapted from NumPy's own test suite. Decent coverage but gaps in edge cases.
633+
A: TUnit framework in `test/NumSharp.UnitTest/`. Many tests adapted from NumPy's own test suite. Decent coverage but gaps in edge cases. Uses source-generated test discovery (no special flags needed).
506634

507635
**Q: What .NET version is targeted?**
508-
A: Library and tests multi-target `net8.0` and `net10.0`. Dropped `netstandard2.0` in the dotnet810 branch upgrade.
636+
A: Library multi-targets `net8.0` and `net10.0`. Tests currently target `net10.0` only (TUnit requires .NET 9+ runtime). Dropped `netstandard2.0` in the dotnet810 branch upgrade.
509637

510638
**Q: What are the main dependencies?**
511639
A: No external runtime dependencies. `System.Memory` and `System.Runtime.CompilerServices.Unsafe` (previously NuGet packages) are built into the .NET 8+ runtime.

0 commit comments

Comments
 (0)