[CHORE]: Disable S3heap service and remove nonce-related logic #5866
Conversation
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
|
|
Retire S3heap scheduler and purge nonce sequencing from task pipeline This PR fully decommissions the experimental S3-backed task-ordering service ("S3heap") and removes the nonce-based sequencing mechanism that once coordinated attached-function execution. The storage schema, coordinator logic, protobuf contracts, build assets, and deployment manifests are all updated to operate without S3heap and nonce fields. This significantly simplifies the task pipeline, trims an entire micro-service from the fleet, and lowers operational complexity and cost, but introduces a required DB migration and minor breaking API changes. Key Changes• Disabled rust/s3heap-service binary and stripped its runtime entry-point Affected Areas• Coordinator (Go) This summary was automatically generated by @propel-code-bot |
tanujnay112
force-pushed
the
remove_nonce
branch
from
c8e5cb7 to
3417b01
Compare
![propel-code-bot[bot]](https://avatars.githubusercontent.com/in/1179004?s=60&v=4){kind=small}
rust/s3heap-service/src/lib.rs
Outdated
| /// Note: This service is currently not fully functional due to nonce removal | ||
| pub async fn entrypoint() { | ||
| eprintln!("Heap tender service is not currently implemented"); | ||
| eprintln!("The heap scheduling functionality was removed"); | ||
| std::process::exit(1); |
There was a problem hiding this comment.
[BestPractice]
Dead code after unconditional process exit: The entrypoint() function always exits with std::process::exit(1), making it impossible to return normally. This prevents proper cleanup and makes the function signature (async fn) misleading.
// Current - unreachable return
pub async fn entrypoint() {
eprintln!("Heap tender service is not currently implemented");
eprintln!("The heap scheduling functionality was removed");
std::process::exit(1);
} // implicit return is unreachable
// Better - explicit never return or remove async
pub fn entrypoint() -> ! {
eprintln!("Heap tender service is not currently implemented");
eprintln!("The heap scheduling functionality was removed");
std::process::exit(1)
}Context for Agents
[**BestPractice**]
**Dead code after unconditional process exit**: The `entrypoint()` function always exits with `std::process::exit(1)`, making it impossible to return normally. This prevents proper cleanup and makes the function signature (`async fn`) misleading.
```rust
// Current - unreachable return
pub async fn entrypoint() {
eprintln!("Heap tender service is not currently implemented");
eprintln!("The heap scheduling functionality was removed");
std::process::exit(1);
} // implicit return is unreachable
// Better - explicit never return or remove async
pub fn entrypoint() -> ! {
eprintln!("Heap tender service is not currently implemented");
eprintln!("The heap scheduling functionality was removed");
std::process::exit(1)
}
```
File: rust/s3heap-service/src/lib.rs
Line: 516| pub async fn tend_to_heap(&self) -> Result<(), Error> { | ||
| let (witness, cursor, tended) = self.read_and_coalesce_dirty_log().await?; | ||
| if !tended.is_empty() { | ||
| let collection_ids = tended.iter().map(|t| t.0).collect::<Vec<_>>(); | ||
| let scheduled = self | ||
| .sysdb | ||
| .clone() | ||
| .peek_schedule_by_collection_id(&collection_ids) | ||
| .await?; | ||
| let triggerables: Vec<Option<Schedule>> = scheduled | ||
| .into_iter() | ||
| .map(|s: ScheduleEntry| -> Result<_, Error> { | ||
| let triggerable = Triggerable { | ||
| partitioning: s3heap::UnitOfPartitioningUuid::new(s.collection_id.0), | ||
| scheduling: s3heap::UnitOfSchedulingUuid::new(s.attached_function_id), | ||
| }; | ||
| if let Some(next_scheduled) = s.when_to_run { | ||
| let schedule = Schedule { | ||
| triggerable, | ||
| next_scheduled, | ||
| nonce: s.attached_function_run_nonce.0, | ||
| }; | ||
| Ok(Some(schedule)) | ||
| } else { | ||
| Ok(None) | ||
| } | ||
| }) | ||
| .collect::<Result<Vec<_>, _>>()?; | ||
| let triggerables: Vec<Schedule> = triggerables.into_iter().flatten().collect(); | ||
| if !triggerables.is_empty() { | ||
| self.writer.push(&triggerables).await?; | ||
| } | ||
| } | ||
| if let Some(witness) = witness.as_ref() { | ||
| self.cursor | ||
| .save(&HEAP_TENDER_CURSOR_NAME, &cursor, witness) | ||
| .await?; | ||
| } else { | ||
| self.cursor | ||
| .init(&HEAP_TENDER_CURSOR_NAME, cursor.clone()) | ||
| .await?; | ||
| } | ||
| Ok(()) | ||
| Err(Error::Internal("Not implemented".to_string())) |
There was a problem hiding this comment.
[BestPractice]
Unimplemented function returns error instead of using proper Rust patterns: The tend_to_heap() function immediately returns an error string instead of using unimplemented!() or todo!() macros, which provide better stack traces and clearer intent.
// Current - generic error message
pub async fn tend_to_heap(&self) -> Result<(), Error> {
Err(Error::Internal("Not implemented".to_string()))
}
// Better - explicit unimplemented with context
pub async fn tend_to_heap(&self) -> Result<(), Error> {
unimplemented!("tend_to_heap removed after nonce removal - see TODO(tanujnay112)")
}Context for Agents
[**BestPractice**]
**Unimplemented function returns error instead of using proper Rust patterns**: The `tend_to_heap()` function immediately returns an error string instead of using `unimplemented!()` or `todo!()` macros, which provide better stack traces and clearer intent.
```rust
// Current - generic error message
pub async fn tend_to_heap(&self) -> Result<(), Error> {
Err(Error::Internal("Not implemented".to_string()))
}
// Better - explicit unimplemented with context
pub async fn tend_to_heap(&self) -> Result<(), Error> {
unimplemented!("tend_to_heap removed after nonce removal - see TODO(tanujnay112)")
}
```
File: rust/s3heap-service/src/lib.rs
Line: 2493417b01 to
74a9e62
Compare
This comment has been minimized.
This comment has been minimized.
tanujnay112
force-pushed
the
remove_nonce
branch
from
74a9e62 to
d730c23
Compare
tanujnay112
force-pushed
the
remove_nonce
branch
2 times, most recently
from
e7dde61 to
72e8a81
Compare
| assert!(found_ids.contains(&id)); | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
[BestPractice]
These tests are being ignored. Since the s3heap-service is being gutted, this is understandable. However, it would be good practice to add a comment explaining why they are ignored (e.g., # [ignore = "Heap tender service is deprecated and will be removed"]) to provide context for future developers.
Context for Agents
[**BestPractice**]
These tests are being ignored. Since the `s3heap-service` is being gutted, this is understandable. However, it would be good practice to add a comment explaining why they are ignored (e.g., `#
[ignore = "Heap tender service is deprecated and will be removed"]`) to provide context for future developers.
File: rust/s3heap-service/tests/test_k8s_integration_00_heap_tender.rs
Line: 3334104603 to
2e336c3
Compare
tanujnay112
force-pushed
the
remove_nonce
branch
from
72e8a81 to
efc6ae0
Compare
tanujnay112
changed the base branch from
graphite-base/5866
to
refactor_compactor
rescrv
left a comment
rescrv
left a comment
There was a problem hiding this comment.
I think you may have left stuff.
| _, err = db.Exec(`UPDATE public.tasks SET lowest_live_nonce = NULL WHERE task_id = $1`, originalTaskID) | ||
| suite.NoError(err, "Should be able to corrupt task in database") | ||
| suite.T().Logf("Made task partial by setting lowest_live_nonce = NULL") | ||
| // TODO: Uncomment after proto regeneration |
There was a problem hiding this comment.
TODO(who); also, maybe you meant to do before review.
| attached_function_soft_delete_absolute_cutoff_time, | ||
| ); | ||
| self.prune_heap_across_shards(cutoff_time).await; | ||
| // let cutoff_time = chrono::DateTime::<chrono::Utc>::from( |
tanujnay112
force-pushed
the
refactor_compactor
branch
from
2e336c3 to
85e37d5
Compare
tanujnay112
force-pushed
the
remove_nonce
branch
from
efc6ae0 to
5b79840
Compare
tanujnay112
changed the base branch from
refactor_compactor
to
graphite-base/5866
tanujnay112
force-pushed
the
graphite-base/5866
branch
from
85e37d5 to
91beb84
Compare
tanujnay112
force-pushed
the
remove_nonce
branch
from
5b79840 to
f655c4c
Compare
Description of changes
Summarize the changes made by this PR.
next_nonce orlowest_live_nonce columns in theattached_functions table.Test plan
How are these changes tested?
pytestfor python,yarn testfor js,cargo testfor rustMigration plan
Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?
Observability plan
What is the plan to instrument and monitor this change?
Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the _docs section?_