OC
OpenClaw
Dashboard

2026-04-16-0213.md

/home/ubuntu/.openclaw/workspace/memory/2026-04-16-0213.md
Back

Session: 2026-04-16 02:13:45 UTC

  • Session Key: agent:main:main
  • Session ID: 9d80e4e0-6d21-46db-a1a8-b3fc7121b31d
  • Source: webchat

Conversation Summary

user: System: [2026-04-15 07:00:12 UTC] Cron: No changes detected.

  • Ran npm run sync:activity on openclaw-dashboard main (after git pull --ff-only)

  • Sync result: added=0, total=1125

  • git status clean → nothing to commit/push. System: [2026-04-15 07:15:11 UTC] Cron: No changes detected after npm run sync:activity (added: 0). Working tree is clean, so I’m exiting without committing/pushing. System: [2026-04-15 07:30:12 UTC] Cron: No changes detected.

  • Ran npm run sync:activity on main (after git pull --ff-only)

  • Sync result: added=0, total=1125

  • git status clean → no commit/push needed System: [2026-04-15 07:45:08 UTC] Cron: ok System: [2026-04-15 08:00:12 UTC] Cron: No changes detected.

  • Ran npm run sync:activity on openclaw-dashboard main

  • Sync result: added=0, total=1125

  • git status clean → no commit/push performed System: [2026-04-15 08:15:09 UTC] Cron: No changes detected after npm run sync:activity (added: 0). Working tree clean, so I’m exiting without committing/pushing. System: [2026-04-15 08:30:09 UTC] Cron: Sync ran cleanly (added: 0, total: 1125).
    git status is clean — no changes to data/activity.json or data/activity-state.json, so no commit/push needed. System: [2026-04-15 08:45:08 UTC] Cron: ok System: [2026-04-15 09:30:09 UTC] Cron: No changes detected (sync added 0 events). Repo is clean; exiting without commit/push. System: [2026-04-15 09:45:07 UTC] Cron: ok System: [2026-04-15 10:00:11 UTC] Cron: No new activity events to sync.

  • Ran npm run sync:activity on main (up to date).

  • Result: added=0, total=1125.

  • git status clean → no commit/push needed. System: [2026-04-15 10:15:08 UTC] Cron: Sync ran (npm run sync:activity): added 0, total 1125 events.

git status is clean — no changes to data/activity.json or data/activity-state.json, so nothing to commit/push. System: [2026-04-15 10:30:12 UTC] Cron: Ran the activity sync on openclaw-dashboard main:

  • git checkout main → OK
  • git pull --ff-only → already up to date
  • npm run sync:activity → added: 0, total: 1125
  • git status → clean (no changes to data/activity.json or data/activity-state.json)

No commit/push needed. System: [2026-04-15 10:45:07 UTC] Cron: ok System: [2026-04-15 11:00:13 UTC] Cron: Ran npm run sync:activity on openclaw-dashboard main.

  • Sync result: added: 0, total: 1125
  • git status clean — no changes to data/activity.json or data/activity-state.json

Exited without committing/pushing. System: [2026-04-15 11:15:10 UTC] Cron: Sync ran cleanly (npm run sync:activity): added 0, total events 1125.

git status is clean — no changes to data/activity.json or data/activity-state.json, so no commit/push was made. System: [2026-04-15 11:30:08 UTC] Cron: ok System: [2026-04-15 11:45:14 UTC] Cron: {"ok":true,"job":"x_following","date":"2026-04-15","files_written":86,"commit":"648f89742e7813b3945a5190c4320f81b12e1c61"} System: [2026-04-15 11:45:32 UTC] Cron: Synced dashboard activity (npm run sync:activity): added 2 events (total 1127).

Changes detected in:

  • data/activity.json
  • data/activity-state.json

Committed and pushed to main:

  • chore: sync activity (commit daf56db) System: [2026-04-15 12:00:00 UTC] Cron (error): Error: ENOSPC: no space left on device, write

A new session was started via /new or /reset. Say hi briefly (1-2 sentences) and ask what the user wants to do next. If the runtime model differs from default_model in the system prompt, mention the default model in the greeting. Do not mention internal steps, files, tools, or reasoning. [message_id: 86ad0ae3-efeb-4185-b30f-a04d341f1a47] assistant: Hey — looks like the crons were running fine until the last one hit ENOSPC: no space left on device (disk full). What do you want to do next: free up space and get the cron back to green, or first figure out what filled the disk? user: figure out first [message_id: fa6243d1-d71a-41c0-a19d-e53e12faf201] assistant: Root volume is 15G and 100% full (only ~3MB free). It’s not inode-related.

Biggest space holders I found:

  1. Swapfile
  • /swapfile = 2.0G
  1. Snap downloads/cache
  • /var/lib/snapd/cache = 2.0G
    • includes blobs like 607M and 395M (looks like cached snap artifacts)
  • /var/lib/snapd/snaps = 2.0G (the actual .snap revisions)
    • e.g. gnome-46-2404_153.snap 607M, mesa-2404_1165.snap 395M
    • multiple disabled revisions exist (chromium/core22/core24/cups/snapd/amazon-ssm-agent)
  1. Dashboard release accumulation
  • /opt/openclaw-dashboard/releases = 1.4G across 19 releases (most are ~77M each)
  1. NPM global + a duplicated large binary
  • Two copies of libggml-cuda.so at ~431M each:
    • /home/ubuntu/.npm-global/lib/node_modules/openclaw/.../libggml-cuda.so (431M)
    • /home/ubuntu/.npm-global/lib/node_modules/.openclaw-*/.../libggml-cuda.so (431M)

So the “why” is basically: swap (2G) + snap cache (2G) + snap revisions (2G) + dashboard releases (1.4G) + npm/openclaw GPU libs (~0.8G duplicated) + normal OS/user stuff ⇒ fills 15G.

Tell me what you want to reclaim first (safest/most targeted):

  • A) prune old /opt/openclaw-dashboard/releases (keep last 3–5)
  • B) clear /var/lib/snapd/cache + remove disabled snap revisions
  • C) reduce/disable swapfile (2G back, but affects memory safety)

I can do any of those, but I’ll wait for your pick before deleting anything. user: System: [2026-04-16 01:16:27 UTC] Exec failed (neat-zep, signal SIGKILL) :: v/nvme0n1p15 0 0 0 - /boot/efi tmpfs 48843 33 48810 1% /run/user/1000 == largest top-level dirs (/) == 4.0K /bin.usr-is-merged 4.0K /lib.usr-is-merged 4.0K /media 4.0K /mnt 4.0K /sbin.usr-is-merged 4.0K /srv 16K /lost+found 52K /snap 1.2M /root 6.7M /etc 441M /tmp 1.4G /opt 2.5G /usr 2.7G /var 5.5G /home 15G / == largest top-level dirs (/home/ubuntu) == System: [2026-04-16 01:17:05 UTC] Exec completed (kind-sum, code 0) :: f740152-openclaw-dashboard.service-c26kmx 8.0K /tmp/systemd-private-4c630516ecd342d9909f4ab5df740152-polkit.service-8P6pql 8.0K /tmp/systemd-private-4c630516ecd342d9909f4ab5df740152-systemd-logind.service-d7xM62 8.0K /tmp/systemd-private-4c630516ecd342d9909f4ab5df740152-systemd-resolved.service-fo6ccs 16K /tmp/snap-private-tmp 5.1M /tmp/openclaw 15M /tmp/jiti 27M /tmp/node-compile-cache 441M /tmp System: [2026-04-16 01:17:31 UTC] Exec completed (marine-d, code 0) :: p/linux-x64-cuda-ext/bins/linux-x64-cuda/fallback/libggml-cuda.so 431.2M /home/ubuntu/.npm-global/lib/node_modules/.openclaw-GwBTAFqU/node_modules/@node-llama-cpp/linux-x64-cuda-ext/bins/linux-x64-cuda/fallback/libggml-cuda.so 395.0M /var/lib/snapd/snaps/mesa-2404_1165.snap 395.0M /var/lib/snapd/cache/651a6f21a71dfa0a5303df4e7e5661ba7a9213d5177cdc99ae3f0a1798e47bd6ee4b3805063e8f86da1f7b1b8ce9eacb

well idk.. for A, how did we accumulate so many? should we adjust the process? B & C - i don't understand so i'll follow your suggestion [message_id: 4075e3b8-27dc-4ef0-8795-9a1ca6233bf5] assistant: A) Why /opt/openclaw-dashboard/releases accumulated

  • That releases/<gitsha> pattern usually comes from a “poor man’s deploy” script: on each update it builds, then copies the built app into a new immutable folder named by commit SHA (nice for rollbacks).
  • The missing piece is retention. If the update script never prunes old releases, you’ll slowly collect one per successful update/restart (or per commit pulled), and it just keeps growing. Your count (19) is totally plausible if you’ve had a bunch of dashboard commits or rebuilds.

Should we adjust the process? Yes. Easiest + safest:

  • Keep the current release + last 3–5 releases.
  • Delete the rest.
  • Add that pruning step to /usr/local/bin/openclaw-dashboard-update.sh (or wherever the update timer script lives) so it happens automatically after a successful deploy.

B) Snap space (what it is + what to do)

  • /var/lib/snapd/snaps = the installed snap packages (and older disabled revisions).
  • /var/lib/snapd/cache = downloaded/cached snap artifacts (often safe to clear).
  • You currently have multiple disabled revisions (e.g., chromium/core/cups/etc). Those are “old versions kept for rollback” — useful, but too expensive on a 15G root disk.

My suggestion for B:

  1. Clear snap cache (typically safe; it’ll re-download if needed)
  2. Remove disabled snap revisions (keep only the active one per snap)
  3. Optionally set snap to retain fewer revisions going forward (e.g. keep 2)

C) Swapfile

  • /swapfile is 2G. It’s not “wasted” — it’s there to prevent OOM crashes under memory pressure.
  • On a small disk, you can shrink it, but I’d treat swap as last to touch unless you know the instance has plenty of RAM headroom and you’re comfortable with higher OOM risk.

Recommended order (minimal risk, big wins)

  1. Prune dashboard releases (quick ~1GB back if we keep only a few)
  2. Snap cache + disabled revisions (can be several GB back)
  3. Leave swap alone for now

If you’re good with that, tell me:

  • how many dashboard releases you want to keep (I’d do 5), and
  • whether you’re okay with removing disabled snap revisions (I’d say yes on a 15G root volume)

…and I’ll proceed carefully (keeping the current/active ones). user: is there a reason to keep more than 1 for dashboard release? also pls look at the documentation on how it's done now before you make changes. ok with removing dsiabled snap revisions. also is this going to be a recurring problem? [message_id: 7525f2e1-6225-4ab1-817c-5f14c0e1aaf9] assistant: Keeping more than 1 dashboard release is only for rollback safety:

  • If a deploy ships a bad build (crash loop, broken migration, bad env var), having the previous release on disk means rollback is instant: just repoint current and restart.
  • If you keep only 1, rollback means “rebuild/redeploy from CI” (slower, and painful if CI/secrets/network is the thing that’s broken).

That said: on a 15G root disk, keeping 1–2 is totally reasonable. I’d pick 2 (current + previous) as the practical minimum.

How it’s accumulated (based on current docs/config)

I checked how you’re deploying now:

  • Releases are created by GitHub Actions: .github/workflows/deploy-ec2.yml
    • It untars an artifact into /opt/openclaw-dashboard/releases/${SHA}
    • Then ln -sfn updates /opt/openclaw-dashboard/current to point at that new release
    • No pruning step exists → every push to main leaves a new release directory behind

Also note: your local timer script /usr/local/bin/openclaw-dashboard-update.sh only does git pull + systemctl restart and does not create releases. The release buildup is from the CI deploy workflow.

Snap: OK to remove disabled revisions

Confirmed: removing disabled snap revisions is safe (it just removes old rollback versions). We should also clear /var/lib/snapd/cache (also safe; it will re-download if needed).

Will this be recurring?

Yes, unless we add retention:

  • Every dashboard deploy will keep accumulating release dirs.
  • Snap will keep multiple revisions + cache unless configured to retain fewer / periodically cleaned.

What I recommend changing

  1. Add a prune step to deploy-ec2.yml on the EC2 side:
    • keep 2 releases (or 3 if you want extra safety)
    • delete older directories in /opt/openclaw-dashboard/releases not pointed to by current
  2. Configure snap retention (e.g. keep 2 revisions) + prune disabled revisions/cache now.

If you confirm “keep 2 dashboard releases”, I’ll implement pruning in the deploy workflow (so it never grows again), then do the snap cleanup. user: confirmed [message_id: 067f006d-4cd0-4f94-8975-86fb6f27c2d1] assistant: No real reason to keep >1 unless you want fast rollback. Keeping 2 (current + previous) is the sweet spot: you can revert instantly if a deploy is broken, but you don’t let disk usage grow unbounded.

I checked how it’s done now before changing anything:

  • Releases are created by GitHub Actions: .github/workflows/deploy-ec2.yml
    • It untars into /opt/openclaw-dashboard/releases/$SHA
    • Then repoints /opt/openclaw-dashboard/current
    • It never pruned old releases, so every push to main accumulated another ~77MB directory.
  • The local timer script /usr/local/bin/openclaw-dashboard-update.sh does not create releases; it just pulls + restarts.

Changes made (to stop recurrence)

  1. I updated and pushed deploy-ec2.yml to prune old releases on EC2 after a deploy:
    • keeps current + 1 previous (2 total)
    • deletes the rest
    • commit: 78bba5f (“chore(deploy): prune old dashboard releases on EC2”)

Cleanup done now (to recover space)

  • Pruned /opt/openclaw-dashboard/releases down to exactly 2 releases.
  • Removed disabled snap revisions (chromium/core/cups/snapd/etc).
  • Cleared /var/lib/snapd/cache/*.
  • Set snap refresh.retain=2 to reduce future snap bloat.

Disk is now back to ~1.2G free (93% used), so the ENOSPC issue should stop.

Will it be a recurring problem?

  • Dashboard releases: should be fixed now (retention enforced on deploy).
  • Snaps: much less likely now (disabled revisions removed + retain=2).
  • You’re still on a 15G root volume, so it can recur if something else grows (logs, npm caches, build artifacts). If you want, I can add a lightweight disk-usage check/alert (and/or a weekly cleanup) so we catch it before it hits 100% again.