atelet: prune local pause snapshots - #705
Conversation
Pause snapshots accumulated until the node disk filled; the control plane only ever references the latest one. Pause now keeps just the snapshot it wrote, suspend removes them all. Snapshot prefixes are validated as single path segments at the RPC boundary: a nested prefix would nest on write but be pruned as its top-level directory, deleting the fresh snapshot.
32ebc1f to
c165fbf
Compare
| if err := s.uploadExternalCheckpoint(ctx, req, checkpointDir, sandboxRec); err != nil { | ||
| return nil, ateerrors.NewGRPCError(ctx, codes.DataLoss, ateerrors.ReasonFaileSaveSnapshot, ateerrors.ActorCrashedMetadata(), fmt.Errorf("%w: while uploading external snapshot: %w", ateerrors.ReasonFaileSaveSnapshot, err)) | ||
| } | ||
| // The durable snapshot supersedes local pause snapshots. |
There was a problem hiding this comment.
There are 2 possible flows:
- run->pause [1]->resume->pause [2]
- run->pause[1] -> resume -> suspend
In both cases, all old snapshots can be deleted, regardless if it is just pause[1] or any other old snapshot.
Current implementation of the "func (s *AteomHerder) copyLocalCheckpoint(ctx context.Context, snapshotPrefix string, srcDir, dstDir string, files []string) error {
" method, copied previous snapshot to new location that takes time. I did in one of my PoC following code:
func (s *AteomHerder) copyLocalCheckpoint(ctx context.Context, snapshotPrefix string, srcDir, dstDir string, files []string) error {
for _, fileName := range files {
if ctx.Err() != nil {
return fmt.Errorf("context cancelled: %w", ctx.Err())
}
src := filepath.Join(srcDir, snapshotPrefix, fileName)
dst := filepath.Join(dstDir, fileName)
// Hardlink first — constant-time regardless of file size, safe here
// because runsc restore only reads these files. Fall back to a byte
// copy on any error (cross-device, permissions, etc.).
if err := os.Link(src, dst); err == nil {
span.SetAttributes(attribute.String("method", "link"))
span.End()
continue
}
n, err := copyFile(src, dst)
if err != nil {
return fmt.Errorf("failed to copy %s to %s: %w", src, dst, err)
}
}
return nil
}
it significantly improved the performance. The point if, you will do the "os.link" optimization, the pruneLocalCheckpoints(ctx, actorUID, "") logic need to be executed before new snapshot been copied either to external storage or moved to local storage location.
It means the logic can be moved above the "switch req.GetType() {" (line 396)
2 participants
Summary
atelet never deletes local pause snapshots, so they accumulate until the node disk fills (#668). The control plane only ever references the latest one (
LocalSnapshotInfois single-valued), so older directories are dead weight by design. Pause now prunes everything except the snapshot it just wrote; suspend removes them all, since the durable snapshot supersedes local state. Pruning is best-effort: a failed delete is logged and retried by the next prune, never failing the checkpoint.Snapshot prefixes are now validated as single path segments at the Checkpoint/Restore RPC boundary (shared
ValidateLocalSnapshotPrefix). Previously only checked non-empty: a nested prefix likepause/2would write nested but be pruned as its top-level directory, deleting the fresh snapshot;..could escape the actor's directory entirely.Fixes #668
Test plan