Hermes chat -Q -q returncode -6 diagnosis

Summary

P0 Agent profile boot/sequential-loop 중 관측된 hermes --profile ... chat -Q -q 종료 오류는 profile identity/SOUL 문제가 아니라, memory.provider: mem0가 활성화된 짧은 CLI subprocess 종료 경로의 Mem0 provider thread teardown 문제로 판정한다.

• 직접 실행에서는 SIGABRT로 exit=134가 기록된다.
• 상위 Python wrapper가 subprocess signal 값을 그대로 해석하면 returncode=-6로 보일 수 있다.
• stdout에는 정상 응답이 출력된 뒤, stderr에 FATAL: exception not rethrown가 남고 core dump가 발생한다.

Environment

• Hermes Agent: v0.16.0 (2026.6.5) · upstream aca11c22
• Project: /home/administrator/.hermes/hermes-agent
• Python: 3.11.15
• Host: WSL2 Linux 6.6.114.1-microsoft-standard-WSL2
• Evidence dir: wiki/work/logs/evidence/hermes-returncode-6-diagnosis-2026-06-15/

Reproduction matrix

Case	Config	Result
default_simple	default profile, no external memory.provider	0
sophie_simple	Sophie profile, memory.provider: mem0	134
scout_simple	Scout profile, memory.provider: mem0	134
diag_scout_nomem	Scout clone, memory.provider: '', memory disabled	0
diag_default_mem0	Default clone + memory.provider: mem0	134
diag_scout_noext	Scout clone, skills external dirs removed	134
diag-mem-disabled-provider-mem0	provider: mem0, memory flags disabled	134

Conclusion: memory.provider: mem0 activation alone is sufficient to reproduce. P0 profile prompts, SOUL files, external skill dirs, or profile identity are not required.

Crash evidence

PYTHONFAULTHANDLER=1 run showed:

session_id: 20260615_215750_a8cb45
FATAL: exception not rethrown
Fatal Python error: Aborted
 
Thread ...:
  <no Python frame>
 
Extension modules: ..., google._upb._message, grpc._cython.cygrpc

This is a native abort during interpreter/process teardown, not a normal Python exception from the model call.

Isolation tests

Temporary source patches were applied and restored to isolate Mem0 paths:

Patch variant	Result	Interpretation
no-op Mem0MemoryProvider.sync_turn() only	134	queue_prefetch() alone can trigger abort
no-op Mem0MemoryProvider.queue_prefetch() only	134	sync_turn() alone can trigger abort
no-op both sync_turn() and queue_prefetch()	0	abort requires Mem0 post-turn work
close underlying httpx.Client in shutdown	134	client close alone is insufficient
make both methods synchronous instead of spawning inner threads	0	root cause is Mem0 provider’s inner background-thread lifecycle interacting with short-lived CLI teardown

Root cause

Hermes MemoryManager.sync_all() and queue_prefetch_all() already dispatch provider work on a single background executor. However, plugins/memory/mem0/__init__.py also creates additional daemon threads inside both methods:

• queue_prefetch() creates threading.Thread(..., daemon=True, name="mem0-prefetch")
• sync_turn() creates threading.Thread(..., daemon=True, name="mem0-sync")

In a one-shot hermes chat -Q -q run, the model response is printed successfully, then the CLI enters session cleanup. The manager drains its own executor, but the provider-level nested daemon threads and their native/runtime side effects can still interact badly with interpreter shutdown. The observed outcome is a native abort: FATAL: exception not rethrown.

Not root causes

• P0 runtime-SOUL.md deployment
• Scout/Nora/Mason/Grace/Archie prompt contents
• Profile creation status
• skills.external_dirs
• Tirith fallback warning
• MCP server startup warnings
• OpenAI Codex model response generation itself

Workaround

For short-lived profile boot tests and automation loops:

1. Use a clone profile with memory.provider: '' for hermes --profile ... chat -Q -q subprocesses; or
2. Temporarily disable Mem0 for P0 worker profiles used only as subprocess workers; or
3. Treat stdout as valid if response is present, but classify 134/-6 as known teardown fault until fixed.

Operationally safest option: create dedicated *-worker profiles with external memory provider disabled, while Sophie/main gateway keeps Mem0 enabled.

Fix candidate

Change Mem0 provider so sync_turn() and queue_prefetch() execute synchronously within MemoryManager’s existing background executor. Do not create provider-internal daemon threads.

Minimal direction:

# plugins/memory/mem0/__init__.py
# Replace per-method threading.Thread(...).start() with direct API work.
# MemoryManager already calls provider.sync_turn / queue_prefetch from its executor.

Optional but not sufficient by itself:

# in shutdown(), close the underlying httpx client if present
http_client = getattr(client, "client", None)
if hasattr(http_client, "close"):
    http_client.close()

Evidence Package

• exit-code-matrix.txt
• env.txt
• faulthandler-mem0.err
• default_simple.*, sophie_simple.*, scout_simple.*
• diag_scout_nomem.*, diag_default_mem0.*
• patch_mem0_no_sync.*, patch_mem0_no_prefetch.*, patch_mem0_no_both.*
• patch_mem0_closeclient.*, patch_mem0_syncinline.*
• root-cause-summary.json