Choosing a Mac Mini model for Hermes and a 119K-token chat

I wanted to test a shared household model without turning my wife’s workday into a maintenance window. She does not enjoy live surgery with herself as the patient. Fair objection.

After researching Apple Minions, I thought my own MacBook Pro would make a convenient starting point. Find a model that fits comfortably there, copy it to the other willing machines, and explore PAIR once I had replicas.

Then I measured the laptop’s memory. Later, I asked how much context my wife’s Telegram conversations consume. I had picked a convenient test machine before understanding the service I wanted to build.

My daily laptop set the wrong ceiling

My M3 Pro MacBook Pro has 18GB of unified memory. With my normal apps open, one Activity Monitor snapshot showed this:

Chrome and its helpers accounted for roughly 3.8GB in summed process figures. Codex and matching helpers accounted for roughly 3GB. Those process figures do not map cleanly to memory I could reclaim by quitting. Shared memory and compression complicate the arithmetic.

Chrome had brought its own appetite to the model-sizing exercise. I don’t even think I run that many tabs. Chromium’s site isolation design explains why tab count is a poor process counter: cross-site frames can need separate renderers too. I appreciate the security boundary. I would also appreciate leaving a little RAM for the computer’s other hobbies.

The readings fluctuated during inspection. Nothing established a leak. The laptop was handling my work successfully. It just did not have a spare 16GB waiting for a model.

That pushed the initial search toward 2B and 4B models. It also exposed a bad constraint. I was about to size a household service around the apps open on my personal laptop.

The M4 Mac mini has 24GB and runs as a headless Hermes server. It does not carry my Chrome and Codex workload. It is not my wife’s daily work machine either. I still need to arrange access, but the Mini is the right target for the evaluation.

This revises my earlier redesign, which made the Mini an agent client borrowing inference elsewhere. I now want to evaluate keeping the harness and a useful smaller model together. I have not measured the Mini’s baseline memory or installed either candidate.

Her compressed chat still carries 61K tokens

My wife uses Hermes through Telegram and keeps conversations running. I asked for numbers rather than guessing from the number of messages.

The excerpts her agent supplied reported:

That compression removed 57,791 tokens, or 48.5%. A freshly compressed chat still started above 61K. My proposed 8K or 16K test would have answered a question nobody in this house was asking.

The agent also reported cache hits near 99% before compression, then 27% afterward, with a 59.1-second post-compression turn. I have the excerpts, not an independent inspection of the logs. That single duration does not isolate cache rebuild time or establish how fast the next turn became.

A prompt cache can save recomputation. It does not make the retained context disappear from the memory budget. Changing models also changes tokenization, so I need to recount a representative transcript with each candidate.

I already learned a version of this in the 72.7M-token context experiment. A configured ceiling tells me little until I fill it. The hidden tool-schema problem supplied another reminder. Instructions and tool definitions count too.

Context memory narrowed the field to two models

The dedicated Mini initially opened up the 8B to 14B range. Her conversation depth made architecture more important than parameter count.

I considered more than Qwen. The final screen looked like this:

The context limits come from the respective Phi-4, Phi reasoning, Phi-mini, Nemotron 9B, and Ministral documentation. Three billion active parameters in the larger Nemotron do not mean only three billion parameters need to reside in memory.

I nearly stopped at Qwen before properly considering Nemotron. That was a narrow shortlist, not evidence that Qwen had won anything. I do not need another sprawling bakeoff to correct it. Two candidates are enough to start.

Both candidates have the same growing FP16 cache component

Qwen3.5-9B combines 24 linear-attention layers with eight full-attention layers. The Q5_K_M file is 6.58GB. That is a download size, not a runtime measurement.

Nemotron 3 Nano 4B has an official Q4_K_M GGUF and a hybrid Mamba/attention design. Its smaller parameter count makes it the efficiency challenger. Whether it handles her conversations as well is still an open question.

The published Qwen configuration and Nemotron configuration produce an interesting match. Their full-attention FP16 KV components each grow by 32KiB per token:

Qwen:     8 layers x 4 KV heads x 256 dimensions x K/V x 2 bytes
Nemotron: 4 layers x 8 KV heads x 128 dimensions x K/V x 2 bytes
 
Both = 32,768 bytes per token

These are component estimates. They exclude weights, recurrent state, runtime buffers, padding, additional cache checkpoints, and macOS. An engine may allocate for the configured capacity before the conversation fills it. File sizes use decimal GB; this cache table uses binary GiB.

The arithmetic makes both worth testing. It does not prove peak memory, prompt-processing speed, or reliable recall. A model can accept a long conversation and still forget the preference that matters.

The Mini gets one service before the house gets a pool

The proposed shape is simple:

Telegram
   |
Hermes on the always-on Mini
   |
   +--> chosen local model on the Mini
   |
   +--> policy router --> specialist model when the job needs it
 
Later, if useful:
   shared-model replicas --> optional PAIR placement
   independent batch jobs --> optional durable worker queue

The Mini will need its own serving qualification, using native llama.cpp with Metal as a candidate path. GGUF availability does not prove that a particular runtime handles long-context caching and Hermes tool parsing correctly.

The trial stays small:

• One model loaded at a time and one active sequence.
• Text-only requests with the same bounded toolset.
• A representative synthetic or redacted conversation at roughly 64K and 128K, plus a 240K stress case.
• Cold, warm, and post-compression timings alongside peak memory and swap.
• Checks for earlier preferences, later corrections, and correct tool names and arguments.

I will control thinking mode where the serving stack supports it. I will not move her live Telegram workflow until a candidate passes. A fast answer to arithmetic is not that qualification.

The Spark cluster keeps its tuned DeepSeek service. The 5090 keeps Muse when it is on. Strix stays available for specific work and testing. The incoming Studio, expected November 9, gets the large-model evaluation. I have not decided which of Spark and Studio ultimately owns coding versus everyday chat.

If the Mini’s model earns the job, I can qualify replicas on the 5080 or the larger laptop later. My M3 Pro no longer sets the quality ceiling. For her long chat, I would favor keeping the session on a warm worker rather than chasing whichever machine looks idle. Independent household requests can use the extra capacity.

Apple Minions remains research to learn from. A future queue might process documents in the background. PAIR might place requests among replicas. Neither needs to sit between my wife and a working assistant on day one.

My 5090 gets the staging session before her Mini

My wife has already used Nemotron and Qwen models through Hermes. She found that the assistant could not handle some of her more complex tasks. That experience does not isolate a model failure from a harness or tool limitation, but it gives this experiment a concrete goal. I want fast local tool calls without asking the resident model to carry every workflow.

Two requests illustrate the split:

The digest needs scheduling and search tools as well as deeper reasoning. Routing it to a larger model will not create those capabilities by itself. The test is whether Hermes, the router, and the selected model can complete the whole workflow together.

Before touching her setup, I will stage the candidate model with my own Hermes instance on the RTX 5090 workstation. Muse can give up its seat for those sessions. I want to tune the policy router against real requests before copying the arrangement to the Mini.

• Simple tool calls should stay with the machine-resident model.
• Workflows needing deeper reasoning should reach Spark, or the Studio once it arrives.
• Route logs should show which model handled each request and why.

I will test the handoff with conversation history and tool definitions included. A router that sends every long chat to Spark would defeat the point of qualifying a local model for her sessions.

Once that behavior works, I will replicate the model, Hermes setup, and routing rules as closely as I can on the Mini. CUDA results will not establish Metal performance or the Mini’s memory fit. Those checks still happen on the target machine.

That is the next homelab post: stage the local-versus-specialist decision on my hardware, then find out how well it survives the move to hers.

I started by trying to use spare Macs. I ended by choosing the machine that can stay useful without anyone noticing it is doing work. I suspect that is the more durable household metric.

By the Numbers

• 18GB in the laptop that looked convenient until I counted its existing commitments.
• 24GB in the dedicated Mini, whose browser tabs do not get a vote.
• 119,238 input tokens before compression. This household has follow-up questions.
• 61,447 tokens afterward, still larger than my original test plan.
• 48.5% less context after compression, with no independently established latency win yet.
• 2 candidates, Qwen3.5-9B and Nemotron 3 Nano 4B. I am resisting the tournament bracket.
• 1 RTX 5090 staging host volunteering for surgery before anyone else’s machine.
• 0 candidate runs on the Mini so far, and 0 reasons to interrupt my wife’s working chat yet.