2026 Remote Mac disk full causes SFTP and rsync upload failure: APFS watermarks, artifact LRU, CI job gate decision matrix

When a remote Mac returns No space left on device, SFTP and rsync often fail mid transfer while ssh login still works. This guide separates APFS container pressure, inode exhaustion, and quotas, then applies twenty, twelve, and ten percent watermarks, artifact LRU, and pre upload probes.

Build hosts that compile Xcode projects, upload artifacts, and retain link dest snapshots rarely fail in a single folder. APFS purgeable bytes, local snapshots, and parallel SFTP compound the problem. Align cleanup with atomic release, link dest, SHA256 gates, concurrent SFTP, and DerivedData locks.

Six pain points when a full disk masquerades as network failure

Pain one: treating df Available as permission to upload. APFS purgeable space and local snapshots can collapse under pressure. Probe container free percent and tmutil listlocalsnapshots before bulk SFTP, not only the Finder chart.

Pain two: in place rsync over live release while full. Half written bundles break atomic publish assumptions. Stop new jobs, reclaim reproducible trees, dry run staging first.

Pain three: link dest retention N too large. Hardlink snapshots save space yet generations still fill NVMe. Lower N before DerivedData and never delete the generation your active staging pointer needs.

Pain four: parallel SFTP racing compiles on one volume. Multiple large uploads amplify metadata IO. Read concurrent SFTP and DerivedData throttling together.

Pain five: mixing ENOSPC with Operation not permitted. The latter may be TCC, SIP, or read only mounts. Correlate errno with unified logs before destructive cleanup.

Pain six: retrying uploads without SHA256 gates. Recover with free space, dry run, integrity verification, then visibility swap.

Disk full governance matrix: watermarks, LRU, and CI gates

Classify true full disk versus inode versus quota first, then pick actions by free percentage. Never start a full mirror or four parallel large uploads below twelve percent free.

How-to: eight steps for APFS probes, artifact LRU, and upload recovery

Before expansion, snapshot manifests per integrity guide and ticket fields from atomic publish so LRU scripts do not delete rollback generations.

# APFS disk watermark — pre-upload probe (remote Mac)
df -H / /System/Volumes/Data 2>/dev/null
diskutil info / | rg -i 'Volume Free Space|Container Free Space|APFS'
tmutil listlocalsnapshots / 2>/dev/null | head
du -sh ~/Library/Developer/Xcode/DerivedData/* 2>/dev/null | sort -hr | head -5
FREE_PCT=$(df -g / | awk 'NR==2{print int($4*100/($3+$4))}')
[ "$FREE_PCT" -lt 12 ] && echo "DEFER_CI_UPLOAD" && exit 75
rsync -a --info=stats2 --dry-run ./artifact/ user@remote:/var/staging/incoming/ || true

1. Freeze change: record df, container UUID, link dest generation, running CI jobs.
2. Classify layer: true full disk, inode exhaustion, quota, middlebox reset.
3. Apply watermarks: at least twenty percent normal; twelve to twenty incremental only; below twelve defer uploads with exit seventy five.
4. LRU cleanup: DerivedData, old releases, lower link dest N, local snapshots last if reproducible.
5. Pre upload probes: df plus rsync dry run; lower parallelism with watermarks.
6. Resume transfers: staging first, SHA256, atomic swap last.
7. Close ticket: gigabytes freed, categories deleted, rollback impact, weekly top ten report.

Reference numbers: APFS free ratios and cleanup boundaries

Medians for planning spreadsheets; measure your fleet before encoding SLOs.

Dashboard both purgeable and available columns because macOS can reclaim purgeable space under pressure and delay alerts by one release window.

Mount link dest staging on its own volume or quota; when full, lower retention generations before deleting the generation your active staging pointer still needs.

Throttle parallel SFTP uploads below twelve percent free space or multiple large files will trigger APFS metadata storms on shared hosts.

Treat CI gate exit seventy five as deferrable, not failed, and document retry backoff so orchestrators do not confuse disk pressure with compile errors.

Time Machine local snapshots on Sequoia can still consume container space; list with tmutil before cleanup and never confuse them with release trees.

Pods and SPM caches are reproducible yet deleting them during active xcodebuild runs causes torn reads; align with the DerivedData concurrency article.

Pre upload rsync dry runs surface target full conditions without writing half files; chain df, dry run, transfer, then SHA256 verification.

Object storage suits cold release archives, not hot compile paths that still need POSIX trees and atomic swaps documented elsewhere.

Operation not permitted during uploads may be TCC or SIP paths rather than ENOSPC; correlate errno with unified log signatures before wiping caches.

Publish a weekly top ten directory report per remote Mac covering DerivedData, link dest generations, releases, and VM images.

When inode exhaustion appears, df may still show free gigabytes; run df -i and separate inode incidents from byte incidents in ticketing.

Bandwidth limits do not fix disk full; pairing ionice with watermark gates prevents compile IO from racing artifact uploads on one NVMe.

Document which artifacts are reproducible versus audit mandated so LRU scripts never delete signed bundles required for compliance replay.

Game day quarterly by filling a staging volume to eleven percent free and verifying defer gates, LRU scripts, and operator runbooks still agree.

Step one evidence matters: paste df, diskutil, tmutil, and job lists into the ticket so LRU has an audit chain.

Step three watermarks belong in the orchestrator: below twelve percent exit seventy five defers scheduling instead of retrying half files.

Step four LRU order is fixed: reproducible caches, old releases, lower link dest N, snapshots last.

Step five couples parallelism to watermarks per the concurrent SFTP article MaxStartups guidance.

Step six lands on staging, verifies SHA256, then swaps visibility per atomic publish.

Hardlink incremental trees remain valuable when POSIX rollback matters; link dest article covers snapshot retention separate from emergency LRU.

Two stage object pipelines help cold archives but hot compile paths still need local headroom before SFTP handoff.

Concurrency locks on shared caches interact with upload jobs; mirror traffic is another writer class that needs the same respect as rsync.

The planning table medians assume a conservative shared remote Mac; adjust MaxParallel when compile jobs spike.

Related runbooks and execution order

Read link dest narrowing, this watermark page, SHA256, atomic swap, then concurrency throttling.

Operators who only watch absolute free gigabytes miss purgeable reclamation timing: a host can accept writes at nine percent displayed available, then reject a multi gigabyte rsync ten minutes later when snapshots expand.

Separate build roots from artifact roots at provisioning time so emergency LRU does not delete the same directory tree your rollback symlink still references.

Teach CI to fail closed on pre upload df checks rather than retrying rsync with identical flags, because each retry can leave another partial object under staging.

When multiple teams share one remote Mac, publish a written priority order: interactive developers beat nightly mirrors, and mirrors beat optional cache warming.

Legal hold and compliance bundles must be tagged non reproducible in your artifact catalog so automated LRU never targets audit mandated paths.

If your orchestrator supports exit seventy five as deferred, document mean time to recovery expectations so product owners do not treat deferral as outage.

Correlate disk pressure metrics with SFTP authentication success: a full disk can surface as generic channel closed errors in client logs.

After emergency cleanup, rerun SHA256 sampling on the last successful staging generation before swapping visibility, even when rsync reported success.

Capacity planning should include link dest retention math: five generations of a two hundred gigabyte tree may need a terabyte of effective space even with hardlinks.

NVMe metadata storms from four parallel uploads feel like CPU saturation; lowering parallelism often restores interactive responsiveness faster than buying another core.

Document whether your release process allows object storage offload for cold generations while keeping hot POSIX trees local for compile linked tools.

Run df -i during every incident postmortem where df -h looked healthy; inode regressions are common after millions of small resource fork files land on macOS.

Avoid deleting Time Machine local snapshots without listing them first; conflating snapshot purge with release tree deletion has caused multi hour rollback gaps.

Instrument staging volume free percent in the same dashboard as gateway CPU so on call engineers see correlated spikes during release windows.

When expanding APFS containers, expand before deleting irreplaceable signed bundles, not after, so verification tools still have headroom to rewrite manifests.

Change tickets for disk incidents should list which watermark tier triggered deferral, which LRU category freed space, and whether link dest generation N was reduced, so the next shift does not repeat incompatible cleanup.

If you mirror from laptops over WiFi, remember that local sleep does not pause server side snapshot growth; remote Mac disks can fill while operators are offline.

Pair watermark gates with MaxStartups tuning on sshd so deferred uploads do not collide with authentication storms when many jobs retry simultaneously.

Signed bundle directories should never be the first LRU target; start with DerivedData and reproducible package caches unless legal approves broader deletion.

After restoring headroom, schedule a follow up scan comparing inode usage week over week, because some CI scripts create millions of tiny files that byte based alerts miss entirely.

Treat APFS container expansion as a planned change with rollback notes, not an emergency click during an active upload, because growing containers while writers are active can still fail if metadata journals are already saturated.

Finally, rehearse communication: when exit seventy five defers a release, product and SRE should see the same dashboard tile explaining deferred versus failed so disk pressure does not become a silent queue of angry stakeholders.

Publish the watermark thresholds in your internal wiki alongside on call runbooks so contractors and full time engineers defer uploads with the same exit codes during holidays.

FAQ

Q Is every mid transfer drop disk full? A No. Separate ENOSPC, quotas, inodes, and middlebox resets. Full disks often still allow ssh while writes fail.

Q Can we rsync in place while full? A Avoid it. Stop jobs, reclaim reproducible artifacts or expand, then follow atomic publish and SHA256 gate.

Q How large are link dest snapshots? A Depends on retention and deltas. When full, lower N before DerivedData without deleting active staging.

Conclusion and SFTPMAC hosted remote Mac trade space

This article turns vague rsync failures into APFS watermarks, artifact LRU, and CI gates you can ticket, aligned with atomic publish, integrity, and concurrency runbooks on SFTPMAC.

Self hosted remote Mac limits remain: retention, cleanup windows, and monitoring stay on your team. One full disk event can stall compile, upload, and rollback together, which costs more SLA than a one off rm.

For separated build and artifact volumes with watermark alerts and upload gates, rent SFTPMAC remote Mac capacity. See plans and home.