neuralOS Usage & Operational Guide¶

Step-by-step instructions for installing, configuring, building, deploying, and operating neuralOS instances. Every command is copy-paste ready.

Table of Contents¶

Install
Verify the Model
Build an Instance
Verify the Instance
Deploy into a Sandbox
Ask Questions
Run as a Service
Monitor and Audit
Manage Lifecycle
Multi-Instance Operations
Troubleshooting

1. Install¶

Option A — single-file binary (no Python needed)¶

Every tagged release ships standalone Nuitka binaries in three variants. The Python runtime, the neuralosd framework, the four documentation volumes and the bundled skills are all embedded in one file.

Variant	Asset	Adds
base	`neuralosd-<platform>`	framework + docs + skills + model
boxlite	`neuralosd-boxlite-<platform>`	+ BoxLite microVM engine
msb	`neuralosd-msb-<platform>`	+ Microsandbox `msb` runtime

Platforms: macos-arm64, macos-x64, linux-x64, linux-arm64, windows-x64 (.exe). Verify what a binary carries with neuralosd backends.

Availability: base everywhere; boxlite on macOS-arm64 + Linux; msb on macOS-arm64 + Linux + Windows. (No upstream wheel for the rest — boxlite has no Windows/Intel-macOS build, microsandbox no Intel-macOS build.)

# macOS Apple Silicon
curl -L -o neuralosd \
  https://github.com/DrOlu/neuralosd/releases/latest/download/neuralosd-macos-arm64
chmod +x neuralosd && sudo mv neuralosd /usr/local/bin/

neuralosd --help

Startup: the first run extracts the payload to a cache dir (~10 s on a cold machine); every later run reuses the cache and starts in ~0.25 s. Set a custom cache location with NEURALOSD_CACHE_DIR if needed.

The binary is self-contained for the framework — init, ask, serve, lint, openapi, golden, truth, invariants, calibrate, gaps, diff, docs, backends all work with no Python installed. Sandbox deployment (neuralosd deploy) still needs BoxLite or the msb CLI on the host, exactly as the pip install does.

Beyond what's baked in: the sidecar¶

A frozen binary cannot import the host's packages — installing a library does not make it visible to the binary. When a probe needs something the binary does not carry, it runs that probe in the host Python instead:

# on the host:
pip install neuralosd        # provides the neuralosd-sidecar command
pip install pypdf pywinrm    # whatever your probes need

# use the binary normally — delegation is automatic
./neuralosd-msb ask --instance-dir ./mydata "pdf pages"

Situation	Behaviour
all of a probe's imports resolve in the binary	runs in-process (fast)
an import is missing, even lazily inside the function	retried in the sidecar
probe declared `tier="sidecar"`	always runs in the sidecar
menu entry declared `"tier": "sidecar"`	always runs in the sidecar (classic layouts)
classic menu-only instance (no probes.py)	fully supported: the sidecar lifts `needle_menu.json` onto bridge.py
missing import and no sidecar	exits with `pip install <lib>`

NEURALOSD_SIDECAR=off disables delegation; NEURALOSD_SIDECAR=/path/to/helper points at a specific helper. The binary stays one stable file — you never rebuild it to gain a library.

Provisioning the sidecar (uv)¶

If the machine has no usable Python, or you would rather not touch its environment, let uv build a dedicated one:

neuralosd sidecar --status                       # uv found? provisioned?
neuralosd sidecar --setup --with pypdf,pywinrm   # create it
neuralosd sidecar --setup --python 3.11          # a specific interpreter
neuralosd sidecar --setup --force                # rebuild from scratch

Under the hood that is exactly:

uv python install 3.12                            # non-fatal if it fails
uv venv --python 3.12 ~/.neuralosd/sidecar
uv pip install --python ~/.neuralosd/sidecar/bin/python \
    "neuralosd==<this build>" pypdf pywinrm

The environment lives at ~/.neuralosd/sidecar (override with $NEURALOSD_HOME) and is found automatically — no environment variables required. It is pinned to the same version as the binary, and provisioning is always explicit because it touches the network.

Requirements: uv on PATH (or $NEURALOSD_UV).

What `init` generates¶

For every data source it produces four kinds of probe:

Probe	Answers
`row_count`, `list_rows`	how many rows; a sample
`distinct_<col>`	which values a low-cardinality column has
`total_<col>`, `average_<col>`	flat aggregates over numeric columns
`breakdown`	group-by: a measure per dimension

breakdown takes caged arguments, so plain English fills them:

neuralosd ask --instance-dir ./superstore "revenue breakdown by year"
neuralosd ask --instance-dir ./superstore "sales by market"
neuralosd ask --instance-dir ./superstore "profit by category"

2012 = 2,259,451   2013 = 2,677,439
2014 = 3,405,746   2015 = 4,299,866      total = 12,642,501

Its dimensions are the low-cardinality categorical columns plus, when a date column is detected, the derived parts year / quarter / month. Its measures are the numeric columns that are not identifiers — and the measure is optional, so "revenue breakdown by year" works without naming a column (it uses the first measure, which is why "revenue" resolves to the Sales column).

A note on selection: the router prefers the probe that accounts for the most of the question through its extracted arguments. Without that, "sales by quarter" matched a flat total_sales and returned a single grand total — a confident answer to a question nobody asked.

The model fallback (`--model`)¶

Deterministic routing runs first, always. The model is consulted only when lexical retrieval and argument extraction cannot produce an answer:

neuralosd ask --instance-dir ./inst "total revenue"          # deterministic
neuralosd ask --instance-dir ./inst --model "what did we earn"   # -> model

Retrieval	With `--model`
a probe matches and its args extract	runs deterministically — the model is never loaded
a probe matches but args are missing	model chooses among the retrieved candidates
nothing matches at all	model is shown the whole menu
nothing matches, no `--model`	honest refusal: `no probe matched this question`

The model routes; it does not answer. It may only call the instance's real probes, and those still execute in the data layer, so results stay verified.

Thread safety. The on-device engine is not thread-safe — building two engines on two threads segfaults the process (reproduced with six parallel /ask calls against serve --model). All model work therefore runs behind a single lock, so model questions are serialised. The deterministic path is unaffected and stays fully concurrent.

Keeping up with the components¶

neuralosd is a composition — the model runtime (neuralos), two sandbox backends (boxlite, microsandbox) and the data toolchain (pydantic, pymysql, openpyxl). When any of them ships a release, there is an automated path to follow.

The check ships as a script, so you can run it yourself:

python scripts/component_versions.py --check      # report drift vs the snapshot
python scripts/component_versions.py --check --record   # update the snapshot
python scripts/component_versions.py --bump patch      # bump neuralosd

Drift is detected by diffing live PyPI against components.json, which is committed — so the state is auditable and a re-run with no changes is a no-op. Each watched component declares a scope:

Scope	A new release triggers
`runtime`	a new PyPI release and rebuilt binaries
`build`	rebuilt binaries only (e.g. a compiler update)

Pre-releases are skipped. An unreachable PyPI is reported as unknown and is never treated as drift — an outage must not mass-bump a release.

In CI this runs daily (.github/workflows/auto-update.yml), and the gate is real: it tests with the current components, upgrades, then tests again against the new ones before anything is published.

Option B — pip (adds the model + sandbox backends)¶

What you need¶

Requirement	Check
macOS Apple Silicon, Linux (KVM), or Windows/WSL2	See below
Python 3.10–3.13 (not 3.14)	`python3.12 --version`
~2 GB disk for packages + sandbox state	`df -h ~`

macOS Apple Silicon¶

# Check virtualization support
sysctl -n kern.hv_support
# Output must be: 1

# Create a virtual environment
python3.12 -m venv ~/neuralos-venv
source ~/neuralos-venv/bin/activate

# Install neuralOS runtime + framework
pip install --upgrade pip
pip install neuralos pydantic

Linux x86_64 / ARM64¶

# Check KVM support
ls -la /dev/kvm
# If missing, enable:
sudo modprobe kvm_intel    # Intel
sudo modprobe kvm_amd      # AMD

# Create venv and install
python3.12 -m venv ~/neuralos-venv
source ~/neuralos-venv/bin/activate
pip install --upgrade pip
pip install neuralos pydantic

Windows (WSL2)¶

wsl --install
# Inside WSL2:
python3.12 -m venv ~/neuralos-venv
source ~/neuralos-venv/bin/activate
pip install neuralos pydantic

Optional: Install sandbox runtimes¶

# BoxLite (Python SDK, recommended for services)
pip install boxlite

# Microsandbox (CLI + runtime)
curl -fsSL https://install.microsandbox.dev | sh

Verify everything works¶

python3 -c "import needle; print('neuralOS', needle.__version__)"
python3 -c "import pydantic; print('pydantic', pydantic.__version__)"
python3 -c "import boxlite; print('boxlite', boxlite.__version__)"  # if installed

2. Verify the Model¶

Before building anything, confirm the 121M model loads and runs:

python3 -c "
import os
os.environ['NEEDLE_TELEMETRY'] = '0'
os.environ['DO_NOT_TRACK'] = '1'
import needle

@needle.tool(triggers=['add two numbers', 'sum'])
def add(a: int, b: int) -> int:
    'Add two numbers.'
    return a + b

agent = needle.Needle(tools=[add])
r = agent.run('add 4 and 5')
print('model OK, result:', r.get('results'))
"
# Expected output: model OK, result: [9]

If this fails, check: - Python version (must be 3.10–3.13) - The neuralos pip package is installed in the active venv - No firewall blocking local execution (the model runs entirely in-process)

3. Build an Instance¶

An "instance" is a named set of probes that answer questions about a specific data source. You build one by profiling your data, generating the models and probes, and verifying the results.

3.0 `neuralosd init` sources¶

neuralosd init --source <file> profiles the source itself (no separate profiler step) and scaffolds a working instance. Supported sources:

Source	Profile
`.csv` / `.tsv`	columns + types + cardinality; count / list / distinct / sum probes
`.xlsx` / `.xlsm`	every sheet (needs `openpyxl`); first sheet drives the menu
`.json` / `.jsonl`	keys profiled; count / list / lookup probes
`.db` / `.sqlite` / `*.sqlite3`	SQLite database — every table profiled with stdlib `sqlite3` (zero extra deps, works in the frozen binary); all tables named in the generated `probes.py`, the first table drives the column probes
`.xlsx` / `.xlsm` (deep, v2.2.0)	every sheet with row/column/formula counts; named ranges profiled and resolvable; `sheets`, `named_ranges` and caged `formula_scan` probes scaffolded; header row excluded from data (like CSV)
`syslog:<path>` (v2.2.0)	syslog text file (RFC3164/RFC5424, stdlib only); normalized to `{ts, host, app, level, message}` events; unparsable lines are counted as `parse_errors` (in the bridge's `LOG_INTEGRITY`), never silently dropped
`journal:[UNIT]` (v2.2.0)	journald via `journalctl -o json`, newest-first, bounded by `NEURALOSD_LOG_CAP` (default 10000); systemd units become a filter arg
`wevtutil:<Channel>` (v2.2.0)	Windows Event Log channel via `wevtutil qe <Channel> /f:XML` — event_id, provider, level, computer, time (read it on the machine that owns the log)
anything else	a contract template the user fills in

neuralosd init --source ledger.db --name ledger --out ./ledger-inst
neuralosd ask --instance-dir ./ledger-inst "how many rows"

3.1 Profile your data¶

The profiler reads your actual data and records every field, type, range, enum value, and sample. It never guesses.

# CSV file
python3 neuralos/scripts/profile_data.py \
    --source ./data/records.csv \
    --out ./profile.json

# MySQL database (requires usql CLI)
python3 neuralos/scripts/profile_data.py \
    --source "mysql://user:pass@host:3306/mydb" \
    --out ./profile.json

# SQLite database
python3 neuralos/scripts/profile_data.py \
    --source "sqlite:///path/to/data.db" \
    --out ./profile.json

# JSON file
python3 neuralos/scripts/profile_data.py \
    --source ./data/events.jsonl \
    --out ./profile.json

# Log file
python3 neuralos/scripts/profile_data.py \
    --source /var/log/app.log \
    --out ./profile.json

# REST API (fetches and profiles the response)
python3 neuralos/scripts/profile_data.py \
    --source https://api.example.com/v1/data \
    --out profile.json

3.2 Read the profile¶

python3 -c "
import json
p = json.load(open('profile.json'))
for t in p.get('source', {}).get('tables', []):
    print(f\"table: {t['name']} ({t.get('rows')} rows)\")
    for f in t.get('fields', []):
        print(f\"  {f['name']}: {f.get('detected_type')} \" +
              f\"(nulls: {f.get('nulls', 0)}, distinct: {f.get('distinct', '?')})\")
"

Check before proceeding: - Fields that should never be null but are - Fields that should be enums but are stored as strings - Dates stored as text - Values with units embedded ("NGN46494.89")

3.3 Generate Pydantic models¶

python3 neuralos/scripts/gen_pydantic.py \
    --profile profile.json \
    --out models.py \
    [--table table_name]

Edit if needed: the generator knows types, but only you know that status = 2 means "In Progress". Add enum labels, rename cryptic columns.

3.4 Generate the full instance¶

python3 neuralos/scripts/gen_needle_instance.py \
    --profile profile.json \
    --models models.py \
    --out ./my_instance \
    [--db-dsn "mysql://user:pass@host/db"] \
    [--runtime python] \
    [--agent-name myfeed] \
    [--table table_name]

This generates the complete instance directory:

my_instance/
├── needle_menu.json       # probe menu: BARE JSON ARRAY of {name, description, parameters, triggers} entries
├── bridge.py              # data layer
├── models.py              # strict Pydantic models
├── instance.py            # agentic agent
├── ask.py                 # MANDATED entry point
├── serve.py               # standard web service
├── golden.json            # golden question bank
├── truth.json             # SQL cross-checks (database sources)
├── invariants.py          # property checks
├── CATALOG.md             # human "what you can ask"
├── openapi.json           # OpenAPI 3.1 spec
├── mcp.json               # MCP tool manifest
├── graph_edges.json       # relationships
├── graph_bridge.py        # graph traversal
├── verify.py              # verification runner
└── verification.txt       # phase-4 evidence

4. Verify the Instance¶

Never skip verification. Every gate must pass before you trust the instance's answers.

cd my_instance

# Model coverage (≥ 95% required)
python3 verify.py

# Selection test (3 phrasings per probe)
python3 verify.py --full

# Truth check (probe vs direct source query)
python3 verify.py --truth

# Golden question bank
python3 verify.py --golden

# Property invariants
python3 verify.py --invariants

# Everything at once
python3 verify.py --full --truth --golden --invariants

If any gate fails: - Coverage < 95% → the model is too strict; widen constraints or clean data - Selection test fails → check triggers; run lint_triggers.py - Truth check fails → the bridge is reading the wrong data - Invariants fail → fix the data or the probe logic

5. Deploy into a Sandbox¶

5.1 BoxLite (recommended for services)¶

import boxlite, asyncio

async def main():
    rt = boxlite.Boxlite.default()

    # Create a persistent box
    got = await rt.get_or_create(
        boxlite.BoxOptions(
            image="python:3.12-slim",
            auto_delete=0,          # disk survives stop/start
            cpus=1, memory_mib=1024,
            disk_size_gb=6,
        ),
        name="my-instance-box")
    box = got[0] if isinstance(got, tuple) else got
    await box.start()

    # Install dependencies
    ex = await box.exec("pip", "install", "--no-cache-dir",
                        "neuralos", "pydantic", "pymysql",
                        timeout_secs=600)
    async for line in ex.stdout(): print(line, end="")

    # Copy instance files
    await box.copy_in("./my_instance", "/opt/instance")

    # Verify inside the box
    ex = await box.exec("python3", ["-c", "import needle; print('OK')"])
    async for line in ex.stdout(): print(line, end="")

    await box.stop()   # disk persists

asyncio.run(main())

5.2 Microsandbox¶

msb create --name my-instance -c 1 -m 1G python:3.12-slim
msb exec my-instance -- pip install neuralos pydantic
msb copy ./my_instance my-instance:/opt/instance

5.3 Clone for scale¶

# BoxLite: clone the template (CoW, near-instant)
venv/bin/python -c "
import boxlite, asyncio
async def main():
    rt = boxlite.Boxlite.default()
    tpl = await rt.get('my-instance-box')
    clone = await tpl.clone_box(name='my-instance-copy')
    print('cloned:', clone.id)
asyncio.run(main())
"

# Microsandbox: fork the running sandbox (RAM-accurate)
msb snapshot create --sandbox my-instance --full -o template.msb
msb snapshot restore template.msb --name my-instance-copy --forked

⚠ Never clone a sandbox that has unique persisted identity (API keys, mesh keypairs). Only fork stateless instances.

6. Ask Questions¶

CLI (one-shot)¶

cd my_instance
python3 ask.py "how many records are in the database"

Python (programmatic)¶

import os, sys
os.chdir("/path/to/my_instance")
sys.path.insert(0, ".")

from ask import ask_one
env = ask_one("how many records are in the database")
print(env["results"])

HTTP (service mode)¶

# Start the service
python3 serve.py --port 8877 &

# Ask via HTTP
curl -X POST http://localhost:8877/ask \
    -H "Content-Type: application/json" \
    -d '{"question": "how many records"}'

What a response looks like¶

{
  "ask_id": "a140e8748ba5",
  "question": "how many records",
  "normalized": "how many records",
  "probe": "count_records",
  "menu_version": "92bfd18ccd6c",
  "mode": "retrieval",
  "confidence": 0.3956,
  "latency_ms": 97,
  "results": [{"count": 3503}]
}

needle_menu.json is a bare JSON array of {name, description, parameters, triggers} entries. This is the canonical shape: the engine's --tools parser, the app's generator, and the fleet guard all expect one.

Instance.export() writes this shape. read_menu() tolerates two legacy wrappings ({"menu": [...]} and {"probes": [...]}) and raises on an unrecognised shape — a bad menu is a startup error, not a silent zero.

If you need instance-level metadata (name, version), put it in a sibling instance.json — do not wrap the menu.

Deploying into a Microsandbox (`--backend msb`)¶

neuralosd deploy --backend msb is only "deployed" when a question asked through the HOST door returns real data. Everything before that is staging, and every step is verified:

neuralosd deploy --instance-dir ./inst --name myinst \
                 --backend msb --port 8920

R0 the host port must be free (a busy port fails loudly) R1 idempotency: an existing sandbox fails without --force; --force removes and redeploys (the instance lives on the host) R2 the sandbox is CREATED with a native msb port forward (msb create -p BIND:HOST:GUEST — create-time only: msb modify cannot add forwards later, so a port change means recreating) R3 the sandbox agent must answer ping R4 neuralosd is pip-installed INSIDE the sandbox (slim ships nothing) and import-verified R5 the instance is staged to /app// — probes.py, bridge.py, golden.json, traps.json — AND the data file, with the bridge's SOURCE rewritten on the host to the staged path R6 neuralosd serve starts via the console script (-m does not exist) R7 self-test INSIDE the VM: a real ask through localhost R8 self-test from the HOST through the forward

Only then: "deployed ... VERIFIED end to end", and the door is recorded in ~/.neuralosd/doors.json.

neuralosd door list                 # every deployed door
neuralosd door stop myinst          # stop the sandbox, forget the door
neuralosd door proxy --name old-sandbox --port 8931
                                    # fallback relay for sandboxes created
                                    # WITHOUT a native forward (one relay
                                    # process per connection; fine for Q&A)

`--backend boxlite`: verified inner, honest host door¶

boxlite 0.10.5's Python binding cannot publish ports: BoxOptions.ports validates tuple/dict entries, then the rust setter rejects them ("object has no attribute 'ports'"), and PublishedPort/NetworkHandle are non-constructible result types. msb modify-style late forwarding does not exist either. So the boxlite deploy pipeline runs R0–R7 (create detached, start, wait for the in-box agent, pip install + import check, stage instance + data with SOURCE rewritten on the host, serve via the console script, inner self-test through a staged file probe), then starts a host-side relay (neuralosd door proxy --backend boxlite) and self-tests through it.

Two measured upstream defects are handled rather than hidden:

the exec channel wedges after a detached serve runs inside the box (every exec then returns exit 0 with empty output, unrecoverable by restart). The pipeline detects it via a staged channel probe, restarts the box once, and fails loudly if the wedge survives.
the runtime lock is exclusive: Boxlite.default() locks ~/.boxlite, so a long-running relay would block every other boxlite process. The door listener therefore holds NO boxlite — each request spawns a short-lived forward child that opens the runtime, serves the one request, and exits.

If the relay still cannot be verified (the exec channel degrades further under some sequences), deploy exits non-zero with everything verified so far named, the exact remediation, and the upstream limitation — never a reachable door that is not. msb remains the verified one-shot production path; an upstream boxlite issue should cover the ports setter and exec-stdout defects.

Sibling calls (VM -> VM) are not wired in this release. Each microVM sits in its own network namespace and cannot address a sibling's inner port. The design: a small resident listener in each VM on a fixed intra-fleet port, with sibling calls relayed THROUGH THE HOST (the only namespace that can see every door). Not gold-plated until someone actually needs it.

The three outcomes: answer, escalate, refuse¶

The router no longer answers whatever scores highest. Every question lands in one of three outcomes:

Outcome	Meaning	Envelope
CONFIDENT	the best match clears the floor, beats the runner-up by the margin, and knows the question's domain nouns	answered as before
UNSURE	a threshold failed and a model fallback is configured	the fallback tries; if it abstains → refusal
REFUSE	no match, thresholds failed with no fallback, or the fallback abstained	`mode: "refused"`, `refusal_reason`, `results: null`, exit 2

Four thresholds, all per-instance configurable via router.json (fitted by neuralosd calibrate) or environment variables:

Threshold	Default	Catches
`floor` (NEURALOSD_FLOOR)	3.0	a best match too weak to trust — usually a generic prefix like "how many"
`margin` (NEURALOSD_MARGIN)	2.0	two probes within a hair of each other, where picking either is a guess
`min_question_coverage` (NEURALOSD_MIN_QUESTION_COVERAGE)	0.5	the question's domain nouns are unknown to the winning probe — it won on generic words
`refuse_disjunction`	true	"more X or Y" comparisons when no probe claims to compare

Plus a dropped-qualifier check: "blocked", "overdue", "closed", "last month"… — if the question carries a status/time qualifier the winning probe's vocabulary does not, the answer would silently ignore it → refused.

Design details that matter:

Coverage is measured over domain nouns, not all content words. "how many worklog entries exist" is five content tokens, three of them generic intent. Measuring over all five let a probe that knew only "how many" pass at 0.6.
Coverage uses the STRONG vocabulary (name + triggers + enum values), not the description. Prose absorbs nearby words; the curated contract is the honest evidence of scope.
Ambiguity requires a coverage disadvantage. A question sharing one generic token with many probes ("top genres") ties on raw score; refusing an answerable question is its own failure. The winner must explain the question strictly better than the runner-up.
An extracted entity bypasses the thresholds. "status of incident INC0000000077173" matched a caged pattern — that extraction IS the confidence signal. Applying prose thresholds there would refuse the questions caged probes exist for.

Every refusal carries its score vector (top-3 probes + scores) and a refusal_reason:

{"mode": "refused",
 "refusal_reason": "no_probe_matches",
 "scores": [{"probe": "open_incidents", "score": 10.0},
            {"probe": "unresolved_incidents", "score": 10.0}],
 "results": null}

refusal_reason ∈ no_probe_matches, below_floor, ambiguous, low_coverage, comparative, dropped_filter, args_not_extractable, out_of_range, model_abstained. Refusals are audited with their score vectors, and neuralosd gaps mines them — a refusal is a menu gap you can fix.

Fitting the thresholds: write a trap bank, then fit:

// traps.json — questions this instance must REFUSE
{"items": [{"q": "how many worklog entries exist", "expect_refusal": true,
            "refusal_reason": "no_probe_matches"}]}

neuralosd calibrate --golden ./inst/golden.json        # traps.json picked up automatically

calibrate searches (floor x margin), punishes a confident wrong answer 5x harder than a false refusal, writes the chosen operating point to router.json, and prints the trade-off curve. golden.json supports negative cases: {"q": "...", "expect_refusal": true, "refusal_reason": "..."} — a confidently answered trap fails the run outright.

Trusting an answer: the `discarded` ledger¶

An answer is only an answer if it is an answer to the question you asked. A router that quietly drops a filter, or a probe that quietly drops rows, produces a number that looks perfectly reasonable and is wrong.

Every envelope therefore carries a discarded block whenever something was left out:

{
  "probe": "breakdown",
  "results": [{"by": "year", "total": 300.0,
               "grand_total": 600.0, "unaccounted": 300.0,
               "rows_in": 3, "rows_counted": 2,
               "skipped": {"missing_year": 1},
               "rows": [{"year": "2012", "Sales": 300.0}]}],
  "discarded": {"rows": {"missing_year": 1}}
}

Two vectors, one ledger:

Key	Meaning
`discarded.terms`	a filter value the question named that no argument consumed
`discarded.rows`	rows the probe excluded from its own computation

A non-empty ledger means the answer is not an answer to your question. Nothing is cached when the ledger is non-empty, because memoizing a partial answer as if it were complete makes it permanent for the whole TTL.

Probes self-account. Every generated aggregate reports rows_in, rows_counted and skipped, and breakdown also reports grand_total and unaccounted — the difference between the sum of its groups and the total over all rows. That difference used to be invisible, and it is exactly how a breakdown can add up internally while disagreeing with the grand total.

Use --strict to refuse instead of returning a partial answer:

neuralosd ask --instance-dir ./inst --strict "sales by year"
# exit 2, error: would have ignored row(s) missing_year=1 — refusing in strict mode

Exit codes¶

Code	Meaning
`0`	answered — `results` is populated
`2`	refused — `results` is `null`, `error` explains why

A refusal prints a JSON envelope, so check the exit code (or results) to tell an answer from a refusal. In Python, a refusal raises NoResults.

Arguments are never coerced¶

An out-of-range argument is refused, not clamped. user 999 against an argument declared 1..10 used to return the record for user 10 — a different record, silently. Now:

user=999 is outside the supported range (1..10) — widen the argument's min/max

What reaches the disk¶

--pii-mask off (or pii_mask=False) governs the response, not storage. The cache expires; ask_audit.jsonl is append-only and permanent. Secrets and PII are therefore masked on write regardless of that setting, by key name and by value shape — a JWT under a key called notes is still a JWT.

neuralosd scrub --instance-dir ./inst --dry-run   # what is on disk today?
neuralosd scrub --instance-dir ./inst             # rewrite it masked
neuralosd scrub --instance-dir ./inst --purge-cache

scrub is the remediation path for records written before a fix; it reports which secret shapes it found without printing them.

When nothing can answer: the reasoning loop¶

Some questions are the wrong SHAPE for the menu, not merely missing from it:

"what is the average shipping cost per unit"   -> shipped cost per line item (3.5x off)
"what share of revenue comes from the top 10"  -> shipped a list of customers, no share
"average number of tracks per album"           -> shipped the album list

Real data, wrong question, and it looks fine. These are ratios, and the router serves the nearest thing it has.

neuralosd reason closes that gap — by installing a derived metric, which is data, not code:

neuralosd reason \
  --instance-dir ~/neuralos-instances/chinook \
  --oracle 0.193764 \
  "what share of total revenue comes from the top 10 customers"

The loop, in order:

1  a DETERMINISTIC gate decides whether to escalate at all      (no model)
2  every probe is called once -> a CLOSED inventory of quantities
3  the reasoning model picks two ids from that list             (nothing else)
4  ids outside the list are REJECTED; the arithmetic is OURS
5  deterministic invariants check it ("a share cannot exceed 1")
6  an optional oracle checks it against a number the loop never sees
7  every neighbouring question is replayed -> routes must not move
8  a derived.json entry is written
9  the same question is now deterministic and the model is gone

The division of labour is the design. A 7B model must never judge: in the run that produced this, it answered "gap" and then "ok" on identical input. It must never compute either: it once proposed dividing two probe names.

the model decides	our code decides
which two quantities to divide	whether to escalate
	whether the ids exist
	the numbers
	whether the answer is a valid share
	whether anything else moved

Nothing a model emits is executed. The output is two ids from a closed list, which is why a prompt injection can make it want to lie and still cannot succeed — the id is rejected, and even an accepted pair goes through our arithmetic.

`derived.json`¶

{"version": 1,
 "metrics": [{"name": "derived_share_total_revenue_comes_from_top",
              "kind": "ratio",
              "numerator": "chinook_top_customers.rows[].total_spend",
              "denominator": "chinook_overview.grand_total_revenue",
              "scale": "percent",
              "triggers": ["what share of total revenue comes from the top 10 customers"],
              "min_coverage": 0.6,
              "provenance": {"model": "deepseek-r1:8b",
                             "verified_against": 0.193764,
                             "checks": ["ids_in_closed_list", "computes",
                                        "expectation_lte_1",
                                        "matches_outside_oracle"]}}]}

Every id is a field some probe really returned — the inventory is observed, not declared. min_coverage is explained under Avoiding hijack below.

Avoiding hijack¶

A new metric must not capture its siblings' questions. In the run that built this, "total revenue" — a 2-token question — started routing to an 8-token derived trigger, because token-overlap scoring happily reports 2-of-8 the same as 2-of-2.

min_coverage fixes it: the question must contain at least that fraction of the metric's trigger. "total revenue" covers 2/8 = 0.25 and is turned away; the real question covers 8/8. Only derived metrics opt in, so no existing probe changes behaviour.

The loop also replays every trigger in the menu before and after installing and refuses to keep a metric that moved anything but its own target.

Reading a derived answer¶

{"probe": "derived_share_total_revenue_comes_from_top",
 "mode": "deterministic",
 "results": [{"percent": 19.3764,
              "numerator":   {"id": "chinook_top_customers.rows[].total_spend", "value": 451.2},
              "denominator": {"id": "chinook_overview.grand_total_revenue", "value": 2328.6},
              "formula": "total_spend / grand_total_revenue",
              "_derived": true}]}

The formula travels with the answer, so a reader can check the arithmetic without trusting the answer.

Flags¶

Flag	Meaning
`--oracle <ratio>`	an independently-computed expected value; the loop never sees where it came from. Without it the check is reported as unchecked, not as passed
`--model`	Ollama model (default `deepseek-r1:8b`)
`--ollama <url>`	default `http://127.0.0.1:11434`
`--dry-run`	show the whole trail, write nothing
`--force`	escalate even when the gate says not to
`--no-stability-check`	ask once instead of twice (faster, less safe)

ollama pull deepseek-r1:8b     # the reasoning model lives on-device

Exit codes¶

Code	Meaning
0	loop closed, or nothing to do
2	no probe returned anything to build an inventory from
3	Ollama unreachable, or the model is not pulled
4	refused — a check failed, or a neighbour moved route
5	installed but the target question still routes elsewhere

An instance built before neuralosd carries its declaration in JSON rather than in @probe decorators. neuralosd serves it directly:

neuralosd ask --instance-dir ~/neuralos-instances/chinook "how many customers"
neuralosd serve --instance-dir ~/neuralos-instances/chinook --port 8878

The menu's own parameters become the caged args, so the same routing and the same accounting apply. A probe whose module cannot import reports why (ModuleNotFoundError: No module named 'pymysql') rather than "0 routable probes", which sends you looking in the wrong place.

6b. Close the loop (v2.3.0)¶

The loop machinery — gaps, feedback, banks, the reason loop — gained eight closing moves:

Refusal clustering — gaps now clusters refusals into PATTERNS:

neuralosd gaps ask_audit.jsonl --out gaps.json

Each cluster carries its cost ("this gap cost you 412 refused queries across 38 phrasings") — you build what users repeatedly want, not what one person asked once.

Federated gap fingerprints — peers on the mesh share anonymized token-pair hashes + counts, never the questions:

neuralosd gaps ask_audit.jsonl --export-fingerprints fp.json   # safe to share
neuralosd gaps ask_audit.jsonl --ingest-fingerprints peer_fp.json

A cluster seen on both edges reports mesh_count — capability discovery, crowd-sourced, privacy-preserving.

Negative feedback → banks — a thumbs-down finally has a path:

neuralosd feedback --instance-dir . --question "units by region" \
    --verdict bad --correction "breakdown of units by region" --by ada
neuralosd feedback --instance-dir . --question "delete all rows" \
    --verdict invalid
neuralosd feedback --instance-dir . --apply      # promote PROVEN candidates

bad (wrong ANSWER) becomes a golden candidate awaiting a verified oracle — never a trap, the question is legitimate. invalid (out-of-scope question) becomes a trap candidate, promoted only when the refusal is proven.

Oracle provenance — golden cases carry receipts:

neuralosd golden --dir . --record --question "total revenue" \
    --expect-probe total_revenue --value 12642501.91 \
    --by olu --source "global_superstore_2016, sum column" --method "SQL cross-check"
neuralosd golden --dir . --receipts     # the evidence chain, for the auditor

Reason-loop consensus + shadow + birth traps:

neuralosd reason --instance-dir . "share of ..." --oracle 0.19 \
    --consensus-model qwen3:8b        # two DIFFERENT models must agree
neuralosd reason --instance-dir . "share of ..." --oracle 0.19 --shadow
neuralosd shadow --instance-dir . --evaluate         # agreement runs
neuralosd shadow --instance-dir . --promote          # promote on agreement

Every non-shadow install now also synthesizes its misuse trap and PROVES the refusal before reporting success — a probe ships guarded or not at all.

Menu hygiene — probes nobody queried get mothballed (kept, reinstatable):

neuralosd mothball --instance-dir . --age-days 180 --apply
neuralosd mothball --instance-dir . --reinstate row_count

Schema-drift re-audit — a changed source flags the probes it breaks BEFORE they fail at runtime:

neuralosd drift --instance-dir . --source ./new_dump.csv   # exit 1 = re-audit

Probe transfer — a schema sibling inherits proven capabilities:

neuralosd transfer --from ~/instances/finacle_a --to ~/instances/finacle_b

Shared probes are reconfirmed with the golden bank translated (re-oracle flagged — the value was proven on ANOTHER estate); missing probes are proposed with their definitions. A proposal is a proposal: nothing installs itself.

6c. Web, APIs, and trust (v2.4.0)¶

The browser is a source. Install the agent-browser CLI with one consent-gated command (checks Node, installs via npm, verifies with a live ping):

neuralosd browser --setup            # add --yes for non-interactive runs

neuralosd init --source browser:https://portal.corp/dashboard --name portal
neuralosd ask --instance-dir ./portal "distinct role"
neuralosd ask --instance-dir ./portal "nodes with role" --help  # caged by role
neuralosd browser --session portal snapshot          # persistent session ops
neuralosd browser --session portal click e12

Authenticated APIs, with the credential caged. The instance stores only the secret REFERENCE — resolved at ask time through scrt / 1Password / env:

neuralosd init --source api:https://api.corp/v1/tickets \
    --header Authorization --secret-ref op:Corp/infra/token

One instance, several sources, a deterministic join:

neuralosd init --source sales.csv --extra-source prices.csv \
    --on region=region --name priced
neuralosd ask --instance-dir ./priced "how many rows in prices"
neuralosd ask --instance-dir ./priced "joined count"

Answers as reproducible evidence:

neuralosd snapshot --instance-dir . --label q3        # hashed observations
neuralosd ask --instance-dir . --as-of q3 "units per row"

Typed oracles — record the expected ANSWER, not just the route:

neuralosd golden --dir . --record --question "how many tickets open" \
    --expect-probe row_count --value 214 --shape count \
    --by olu --source "service-now, saved search" --method "UI cross-check"
neuralosd golden --dir .        # wrong value = FAIL, separately counted

Trust ops:

neuralosd slo --instance-dir . --max-refusal-rate 0.4      # exit 1 on breach
neuralosd fleet ~/instances/*                               # fleet dashboard
neuralosd badge --instance-dir . --out badge.json \
    --actions-out .github/workflows/golden.yml              # README shield
neuralosd synonyms --instance-dir .                         # trigger proposals
ask --instance-dir . --role analyst "units by region"       # RBAC row caging

6d. Natural language builds — pi as the builder (v2.6.0)¶

With the pi coding agent installed (npm install -g @mariozechner/pi-coding-agent) and a provider configured:

neuralosd pi --setup                # verify pi + model + live ping
neuralosd build "average shipping cost per unit by region" \
    --instance-dir ./portal

Pi writes the probe code inside the instance directory. Neuralosd judges: banks byte-identical, instance loads, both banks replay green — anything else restores the pre-pi state. A successful build leaves the new probe with its misuse trap banked; grow the golden bank around it before trusting it (oracles or bust).

Operator mode — natural language over the closed command list:

neuralosd pi ask --instance-dir . "why did refusals go up this week?"

Durable builds — the conversation survives crashes and is resumable:

neuralosd build "average shipping cost per unit by region" \
    --instance-dir ./portal --durable
# crashed mid-build? continue the SAME conversation:
neuralosd build "average shipping cost per unit by region" \
    --instance-dir ./portal --durable --resume
neuralosd pi --instance-dir ./portal durable --status

And one command to check every integration:

neuralosd doctor --instance-dir .      # exit 1 = something needs attention

Default model `z-ai/glm-5.3-flash` via openrouter; override with
`--model`/`--provider` or `$NEURALOSD_PI_MODEL` / `$NEURALOSD_PI_PROVIDER`.

## 7. Run as a Service

### 7.1 Start the standard service

```bash
neuralosd serve --instance-dir ./inst --port 8877
# → http://127.0.0.1:8877

7.1b Token auth (recommended whenever the port is reachable)¶

neuralosd serve --instance-dir ./inst --port 8877 \
                --auth-token "$NEURALOSD_API_TOKEN"

Every endpoint except /healthz and /ready requires Authorization: Bearer <token> (or X-API-Token: <token>).
/healthz stays open on purpose: it exposes only the instance name. /ready requires the token — it executes a real probe, and an unauthenticated readiness check would let anyone run queries.
Token comparison is constant-time.
Equivalent: export NEURALOSD_API_TOKEN=... instead of the flag.
Non-loopback binding without a token still prints the explicit warning.

serve binds loopback by default. /ask exposes the data behind every probe and has no authentication, so binding wide is a deliberate act:

neuralosd serve --instance-dir ./inst --host 0.0.0.0 --port 8877
# WARNING: bound to a non-loopback address, and /ask has NO authentication.
# Put an authenticating proxy in front, or bind 127.0.0.1 and tunnel.

--strict applies the same refusal policy as ask --strict to every request.

7.2 Endpoints¶

Endpoint	Method	Body	Response
`/healthz`	GET	—	`{"ok": true}`
`/ready`	GET	—	`{"ok": true, "data_layer": "ok"}`
`/ask`	POST	`{"question": "…"}`	envelope with results
`/openapi.json`	GET	—	OpenAPI 3.1 spec

7.3 Ask via HTTP¶

curl -X POST http://localhost:8877/ask \
    -H "Content-Type: application/json" \
    -d '{"question": "top customers"}'

7.4 Response shape¶

{
  "ask_id": "a140e8748ba5",
  "question": "top customers",
  "normalized": "top customers",
  "probe": "chinook_top_customers",
  "menu_version": "92bfd18ccd6c",
  "mode": "retrieval",
  "confidence": 0.3956,
  "latency_ms": 97,
  "results": [{"customer": "Helena Holý", "total_spend": 49.62, ...}]
}

8. Monitor and Audit¶

8.0 Instance import audit — `neuralosd audit`¶

The frozen binary loads instance bridge.py / instance.py / probes.py dynamically, so the build cannot know their imports. The audit closes that gap at birth: it AST-parses every instance module and classifies each import against the runtime and the sidecar — empirically (find_spec), never assumed.

# one instance
neuralosd audit --instance-dir ./inst

# a whole fleet (every dir with needle_menu.json under root)
neuralosd audit --root /srv/fleet --host-python /path/to/sidecar/python

# machine-readable (for CI / the instance-creation gate)
neuralosd audit --instance-dir ./inst --json

Classification:

Class	Meaning
`bundled`	importable in the frozen runtime (proven, not assumed)
`host-only`	not bundled; the sidecar/host python has it
`instance-local`	a sibling `.py`/package in the instance dir — resolves via the dir's sys.path entry
`missing-everywhere`	this instance will fail — install the module into the sidecar

Exit code is non-zero when anything is missing-everywhere, so the instance-creation flow can refuse to declare done. neuralosd init runs the audit automatically and warns before an under-provisioned instance ships. Instance-internal cross-imports (probes.py doing import bridge) and sibling modules (models.py, graph_bridge.py) are recognized as instance-local, never false-flagged.

Audit log¶

Every ask is appended to ask_audit.jsonl:

tail -5 ask_audit.jsonl

Each line contains:

{
  "ts": 1790817120.678,
  "ask_id": "a140e8748ba5",
  "question": "top customers",
  "normalized": "top customers",
  "probe": "chinook_top_customers",
  "confidence": 0.3956,
  "latency_ms": 97,
  "mode": "retrieval"
}

Gap log¶

Questions that scored below the confidence gate are logged to menu_gaps.jsonl — these are candidates for new probes:

tail -3 menu_gaps.jsonl

Lint triggers¶

After any menu change:

python3 neuralos/scripts/lint_triggers.py needle_menu.json

9. Manage Lifecycle¶

Upgrade the model¶

pip install --upgrade neuralos
# Re-run the verification suite
python3 verify.py --full --truth --golden

Upgrade the instance¶

Snapshot the current state (rollback point)
Re-profile the data (if the source changed)
Re-generate the instance
Re-run all verification gates
Deploy the new version

Backup¶

# BoxLite: export the box as a portable archive
python3 -c "
import boxlite, asyncio
async def main():
    rt = boxlite.Boxlite.default()
    box = await rt.get('my-instance')
    await box.export(dest='backup.boxlite')
asyncio.run(main())
"

# Microsandbox: full snapshot (disk + RAM + processes)
msb snapshot create --sandbox my-instance --full -o backup.msb

Scale¶

# Clone the template for parallel processing
venv/bin/python -c "
import boxlite, asyncio
async def main():
    rt = boxlite.Boxlite.default()
    tpl = await rt.get('my-instance')
    for i in range(5):
        await tpl.clone_box(name=f'worker-{i}')
asyncio.run(main())
"

10. Multi-Instance Operations¶

When you have multiple instances, the meta-selector routes questions to the right one:

from neuralosd import MetaSelector

selector = MetaSelector({
    "finance": {
        "description": "revenue, invoices, customers, payments",
        "examples": ["top customers", "sales by country"]
    },
    "ops": {
        "description": "incidents, uptime, alerts, SLA",
        "examples": ["open incidents", "P1 escalations"]
    },
})
instance = selector.route("how many open incidents")
# → "ops"

11. Troubleshooting¶

Problem	Cause	Fix
`ModuleNotFoundError: No module named 'needle'`	neuralOS not installed	`pip install neuralos`
`ModuleNotFoundError: No module named 'pydantic'`	Missing dependency	`pip install pydantic`
Wrong probe selected	Selector fumbled	Use `ask.py` (has retrieval + fast path)
Stale answer from previous question	Results gate missing	Use the generated `ask.py`
`NoResults` / empty results	No probe matched	Check the menu; add triggers for the phrasing
Import fails inside sandbox	sys.path not set	`sys.path.insert(0, "/opt/instance")`
Box won't start	Virtualization not enabled	Enable KVM/HVF/WHP
Model slow (> 5s per ask)	Cold start (weights mmap)	Subsequent asks are fast; or use a warm pool
Trigger collision detected	Two probes share a trigger	Reword or remove the duplicate
PII visible in results	Mask disabled	Ensure `NEURALOS_MASK_PII` is not `0`

Sizing reference¶

Deployment	vCPU	RAM	Disk	Measured
neuralOS only	1	1 GB	4 GB	idle 178 MB, peak 357 MB
+ MariaDB source	1	1 GB	6 GB	+150 MB for MariaDB
Multi-tenant (10 forks)	2	4 GB	20 GB	shared base, per-fork delta

neuralOS Usage & Operational Guide¶

Table of Contents¶

1. Install¶

Option A — single-file binary (no Python needed)¶

Beyond what's baked in: the sidecar¶

Provisioning the sidecar (uv)¶

What init generates¶

The model fallback (--model)¶

Keeping up with the components¶

Option B — pip (adds the model + sandbox backends)¶

What you need¶

macOS Apple Silicon¶

Linux x86_64 / ARM64¶

Windows (WSL2)¶

Optional: Install sandbox runtimes¶

Verify everything works¶

2. Verify the Model¶

3. Build an Instance¶

3.0 neuralosd init sources¶

3.1 Profile your data¶

3.2 Read the profile¶

3.3 Generate Pydantic models¶

3.4 Generate the full instance¶

4. Verify the Instance¶

5. Deploy into a Sandbox¶

5.1 BoxLite (recommended for services)¶

5.2 Microsandbox¶

5.3 Clone for scale¶

6. Ask Questions¶

CLI (one-shot)¶

Python (programmatic)¶

HTTP (service mode)¶

What a response looks like¶

The menu file shape¶

Deploying into a Microsandbox (--backend msb)¶

--backend boxlite: verified inner, honest host door¶

The three outcomes: answer, escalate, refuse¶

Trusting an answer: the discarded ledger¶

Exit codes¶

Arguments are never coerced¶

What reaches the disk¶

When nothing can answer: the reasoning loop¶

derived.json¶

Avoiding hijack¶

Reading a derived answer¶

Flags¶

Exit codes¶

Serving a classic needle-menu instance¶

6b. Close the loop (v2.3.0)¶

6c. Web, APIs, and trust (v2.4.0)¶

6d. Natural language builds — pi as the builder (v2.6.0)¶

7.1b Token auth (recommended whenever the port is reachable)¶

7.2 Endpoints¶

7.3 Ask via HTTP¶

7.4 Response shape¶

8. Monitor and Audit¶

8.0 Instance import audit — neuralosd audit¶

Audit log¶

Gap log¶

Lint triggers¶

9. Manage Lifecycle¶

Upgrade the model¶

Upgrade the instance¶

Backup¶

Scale¶

10. Multi-Instance Operations¶

11. Troubleshooting¶

Sizing reference¶

What `init` generates¶

The model fallback (`--model`)¶

3.0 `neuralosd init` sources¶

Deploying into a Microsandbox (`--backend msb`)¶

`--backend boxlite`: verified inner, honest host door¶

Trusting an answer: the `discarded` ledger¶

`derived.json`¶

8.0 Instance import audit — `neuralosd audit`¶