Skip to content

Commit 0d5c6cf

Browse files
Jade Wangclaude
andcommitted
fix(csharp): implement RefreshUrlsAsync for REST API with 1-hour URL expiration
- Added RefreshUrlsAsync implementation in StatementExecutionResultFetcher - REST API URLs expire after ~1 hour and need refresh via GetResultChunkAsync - Fixed DownloadResult constructor calls to include chunkIndex parameter - Fixed CloudFetchDownloadManager test constructor usage 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 0ccf82d commit 0d5c6cf

File tree

12 files changed

+1058
-6
lines changed

12 files changed

+1058
-6
lines changed

.claude/commands/PECO.Design.md

Lines changed: 325 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,325 @@
1+
---
2+
description: You are a senior developer and in this session, you need create a detailed design for a feature.
3+
---
4+
5+
### User Input
6+
```text
7+
$ARGUMENTS
8+
```
9+
10+
You **MUST** consider the user input before proceeding. If empty, ask the user for what to work on.
11+
12+
# Collect requirement from user
13+
14+
always collect enough information from user, this might be one or more of the following
15+
16+
## an existing design doc
17+
in this case, we are working an existing doc either addressing review comments or change some of the designs.
18+
19+
## an PM requirement document
20+
you will need review the doc and reconcile with existing design doc if there is one to make sure everything is in sync, if not, working on the change of design
21+
22+
## conversation with user
23+
always good to ask a lot of clarification questions and probe users on overall requirement and corner cases
24+
25+
26+
27+
# Start the design by following below best practises
28+
29+
## Overview
30+
This guide outlines best practices for writing technical design documents, based on lessons learned from code reviews.
31+
32+
---
33+
## 1. Use Visual Diagrams Over Text
34+
35+
### ✅ DO:
36+
- Use **mermaid diagrams** for all architectural illustrations
37+
- Include **class diagrams** to show relationships between components
38+
- Use **sequence diagrams** to illustrate data flow and interactions
39+
- Render diagrams inline in markdown using mermaid code blocks
40+
41+
### ❌ DON'T:
42+
- Use ASCII art for diagrams
43+
- Describe flows in long text paragraphs
44+
- Include large blocks of code to explain architecture
45+
46+
### Example:
47+
```markdown
48+
## Architecture
49+
```mermaid
50+
classDiagram
51+
class TelemetryCollector {
52+
+Record(event: TelemetryEvent)
53+
+Flush()
54+
}
55+
class TelemetryExporter {
56+
+Export(events: List~Event~)
57+
}
58+
TelemetryCollector --> TelemetryExporter
59+
```
60+
```
61+
62+
---
63+
64+
## 2. Focus on Interfaces and Contracts
65+
66+
### ✅ DO:
67+
- Document **public APIs** and **interfaces**
68+
- Show **contracts between components**
69+
- Specify **input/output** parameters
70+
- Define **error handling contracts**
71+
- Document **async/await patterns** where applicable
72+
73+
### ❌ DON'T:
74+
- Include detailed implementation code
75+
- Show private method implementations
76+
- Include complete class implementations
77+
78+
### Example:
79+
```markdown
80+
## ITelemetryCollector Interface
81+
82+
```csharp
83+
public interface ITelemetryCollector
84+
{
85+
// Records a telemetry event asynchronously
86+
Task RecordAsync(TelemetryEvent event, CancellationToken ct);
87+
88+
// Flushes pending events
89+
Task FlushAsync(CancellationToken ct);
90+
}
91+
```
92+
93+
**Contract:**
94+
- RecordAsync: Must be non-blocking, returns immediately
95+
- FlushAsync: Waits for all pending events to export
96+
- Both methods must never throw exceptions to caller
97+
```
98+
99+
---
100+
101+
## 3. Remove Implementation Details
102+
103+
### ✅ DO:
104+
- Focus on **what** the system does
105+
- Explain **why** design decisions were made
106+
- Document **integration points**
107+
- Describe **configuration options**
108+
109+
### ❌ DON'T:
110+
- Include internal implementation details
111+
- Show vendor-specific backend implementations
112+
- Document internal database schemas (unless part of public contract)
113+
- Include proprietary or confidential information
114+
115+
---
116+
117+
## 4. Simplify Code Examples
118+
119+
### ✅ DO:
120+
- Use **minimal code snippets** to illustrate concepts
121+
- Show only **signature changes** to existing APIs
122+
- Replace code with **diagrams** where possible
123+
- Use **pseudocode** for complex flows
124+
125+
### ❌ DON'T:
126+
- Include complete class implementations
127+
- Show detailed algorithm implementations
128+
- Copy-paste large code blocks
129+
130+
### Example:
131+
```markdown
132+
## DatabricksConnection Changes
133+
134+
**Modified Methods:**
135+
```csharp
136+
// Add telemetry initialization
137+
public override async Task OpenAsync(CancellationToken ct)
138+
{
139+
// ... existing code ...
140+
await InitializeTelemetryAsync(ct); // NEW
141+
}
142+
```
143+
144+
**New Fields:**
145+
- `_telemetryCollector`: Optional collector instance
146+
- `_telemetryConfig`: Configuration from connection string
147+
```
148+
149+
---
150+
151+
## 5. Simplify Test Sections
152+
153+
### ✅ DO:
154+
- List **test case names** with brief descriptions
155+
- Group tests by **category** (unit, integration, performance)
156+
- Document **test strategy** and coverage goals
157+
- Include **edge cases** to be tested
158+
159+
### ❌ DON'T:
160+
- Include complete test code implementations
161+
- Show detailed assertion logic
162+
- Copy test method bodies
163+
164+
### Example:
165+
```markdown
166+
## Test Strategy
167+
168+
### Unit Tests
169+
- `TelemetryCollector_RecordEvent_AddsToQueue`
170+
- `TelemetryCollector_Flush_ExportsAllEvents`
171+
- `CircuitBreaker_OpensAfter_ConsecutiveFailures`
172+
173+
### Integration Tests
174+
- `Telemetry_EndToEnd_ConnectionToExport`
175+
- `Telemetry_WithFeatureFlag_RespectsServerSide`
176+
```
177+
178+
---
179+
180+
## 6. Consider Existing Infrastructure
181+
182+
### ✅ DO:
183+
- **Research existing solutions** before designing new ones
184+
- Document how your design **integrates with existing systems**
185+
- Explain why existing solutions are **insufficient** (if creating new)
186+
- **Reuse components** where possible
187+
188+
### ❌ DON'T:
189+
- Reinvent the wheel without justification
190+
- Ignore existing patterns in the codebase
191+
- Create parallel systems without explaining why
192+
193+
### Example:
194+
```markdown
195+
## Alternatives Considered
196+
197+
### Option 1: Extend Existing ActivityTrace Framework (PR #3315)
198+
**Pros:** Reuses existing infrastructure, familiar patterns
199+
**Cons:** ActivityTrace is designed for tracing, not metrics aggregation
200+
201+
### Option 2: New Telemetry System (Chosen)
202+
**Rationale:** Requires aggregation across statements, batching, and different export format than traces
203+
```
204+
205+
---
206+
207+
## 7. Address Concurrency and Async Patterns
208+
209+
### ✅ DO:
210+
- Clearly mark **async operations** in interfaces
211+
- Document **thread-safety** guarantees
212+
- Explain **blocking vs non-blocking** operations
213+
- Show **cancellation token** usage
214+
215+
### ❌ DON'T:
216+
- Mix sync and async without explanation
217+
- Leave thread-safety unspecified
218+
- Ignore backpressure and resource exhaustion scenarios
219+
220+
### Example:
221+
```markdown
222+
## Concurrency Model
223+
224+
### Thread Safety
225+
- `TelemetryCollector.RecordAsync()`: Thread-safe, non-blocking
226+
- `TelemetryExporter.ExportAsync()`: Called from background thread only
227+
228+
### Async Operations
229+
All telemetry operations are async to avoid blocking driver operations:
230+
```mermaid
231+
sequenceDiagram
232+
Driver->>+Collector: RecordAsync(event)
233+
Collector->>Queue: Enqueue(event)
234+
Collector-->>-Driver: Task completed (non-blocking)
235+
Collector->>+Exporter: ExportAsync(batch)
236+
Exporter-->>-Collector: Task completed
237+
```
238+
```
239+
240+
---
241+
242+
## 8. Document Edge Cases and Failure Modes
243+
244+
### ✅ DO:
245+
- Explain what happens during **failures**
246+
- Document **circuit breaker** or retry logic
247+
- Address **data loss** scenarios
248+
- Show how **duplicate events** are handled
249+
250+
### ❌ DON'T:
251+
- Only show happy path
252+
- Ignore error scenarios
253+
- Leave failure behavior undefined
254+
255+
### Example:
256+
```markdown
257+
## Error Handling
258+
259+
### Circuit Breaker Behavior
260+
When export fails 5 consecutive times:
261+
1. Circuit opens, drops new events (avoids memory exhaustion)
262+
2. Sends circuit breaker event to server
263+
3. Attempts recovery after 60s
264+
265+
### Duplicate Handling
266+
If same statement reported multiple times:
267+
- Backend merges by `statement_id`
268+
- Uses latest timestamp for each metric type
269+
```
270+
271+
---
272+
273+
## 9. Include Configuration Options
274+
275+
### ✅ DO:
276+
- Document **all configuration parameters**
277+
- Show **default values** and acceptable ranges
278+
- Explain **opt-out mechanisms**
279+
- Document **feature flags** and server-side controls
280+
281+
### Example:
282+
```markdown
283+
## Configuration
284+
285+
| Parameter | Type | Default | Description |
286+
|-----------|------|---------|-------------|
287+
| `telemetry.enabled` | bool | `true` | Enable/disable telemetry |
288+
| `telemetry.batch_size` | int | `100` | Events per batch (1-1000) |
289+
| `telemetry.flush_interval_ms` | int | `5000` | Flush interval (1000-30000) |
290+
291+
**Feature Flag:** `spark.databricks.adbc.telemetry.enabled` (server-side)
292+
```
293+
294+
---
295+
296+
## 10. Keep Sections Focused
297+
298+
### ✅ DO:
299+
- Include only **necessary sections**
300+
- Each section should answer **specific questions**
301+
- Remove sections that don't add value
302+
303+
### ❌ DON'T:
304+
- Include boilerplate sections "just because"
305+
- Add sections that duplicate information
306+
- Keep sections that reviewers flag as unnecessary
307+
308+
---
309+
310+
## Summary Checklist
311+
312+
Before submitting your design doc:
313+
314+
- [ ] All diagrams are in **mermaid format**
315+
- [ ] Focus is on **interfaces, not implementations**
316+
- [ ] **Internal details** removed
317+
- [ ] Code examples are **minimal and relevant**
318+
- [ ] Test sections show **case names, not code**
319+
- [ ] **Existing infrastructure** considered and discussed
320+
- [ ] **Async/thread-safety** clearly documented
321+
- [ ] **Edge cases and failures** addressed
322+
- [ ] **Configuration options** fully documented
323+
- [ ] All sections are **necessary and focused**
324+
- [ ] Design explains **why**, not just **what**
325+
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
description: Sprint planning assistant that creates a story and sub-tasks for a 2-week sprint based on high-level and detailed design documents.
3+
---
4+
5+
### User Input
6+
```text
7+
$ARGUMENTS
8+
```
9+
10+
You **MUST** consider the user input before proceeding. If empty, ask the user for a ticket number or task description to work on.
11+
12+
## Goal
13+
Create a comprehensive sprint plan including a JIRA story and sub-tasks for a 2-week sprint cycle.
14+
15+
## Required Information
16+
- High-level design document
17+
- Detailed design document(s)
18+
- Current project status and past tickets
19+
20+
You can ask for the exact path to design documents, or search the current folder based on a task description provided by the user.
21+
22+
## Steps
23+
24+
### Step 1: Gather Required Information
25+
Ensure you have all necessary documents and context. Ask for additional details if needed:
26+
- Design document paths or descriptions
27+
- Related EPIC or parent ticket information
28+
- Any specific constraints or requirements
29+
30+
### Step 2: Understand the Problem
31+
Analyze the current state of the project:
32+
- Read through the design documents thoroughly
33+
- Review past tickets and their status
34+
- Examine the current codebase to understand implementation status
35+
- Identify what has been completed and what remains to be done
36+
37+
### Step 3: Define the Sprint Goal
38+
Based on your analysis, propose a realistic goal for the 2-week sprint. Discuss the proposed goal with the user to ensure alignment and feasibility before proceeding.
39+
40+
### Step 4: Break Down Work into Sub-Tasks
41+
After goal confirmation, create a detailed breakdown of work items:
42+
- Each task should ideally be scoped to ~2 days of work
43+
- Focus strictly on items within the sprint goal scope
44+
- Ensure tasks are concrete and actionable
45+
46+
### Step 5: Create JIRA Tickets
47+
After user confirmation of the task breakdown, create:
48+
- One parent story for the sprint goal
49+
- Individual sub-tasks for each work item identified in Step 4
50+
51+

0 commit comments

Comments
 (0)