Skip to main content

Troubleshooting

Check one hop at a time: is the server running, can the browser reach it, can the agent reach it, can the agent reach the network it scans. Change one thing, then test again.

Each entry below starts with what you see, then the likely cause and the fix.

Where the logs are​

Start with the log. MiniPAM writes to several places:

  • Console. The server prints INFO lines to standard output and WARNING and above to standard error. With systemd, read them with journalctl.
  • Docker. docker logs minipam, or docker logs -f minipam to follow.
  • Log files, if you turn them on with files.enabled. MiniPAM writes one file per stream in files.directory: api.log, agent.log, and background.log. Each line is JSON.
  • Administration → System → Logs (admins). The recent log of the running server, kept in memory, with filters for stream, level, module, and text. It keeps DEBUG lines even when the console is at INFO. It is cleared when the server restarts. See Logs.

On the Logs page, the System button changes logging without a restart:

  • Temporary levels raise or lower the level of one module (a logger name prefix, or * for all) for 5 minutes, 15 minutes, or 1 hour. They change the console and the log files, not the in-memory view.
  • Temporary exclusions hide noisy lines everywhere, including the Logs page, for up to 24 hours.

Both expire by themselves, and a restart clears them. To keep a level, set module-levels in the configuration file.

To diagnose a problem:

  • Agent or job problems: the full messages exchanged with agents (agent_request_outbound, agent_response_inbound) are logged only at DEBUG. The Logs page shows them already. To get them in the console or log files too, set api.agent.ws and services.job_runtime to DEBUG.
  • Database problems: set db.sql to DEBUG. SQL statements are recorded only while it is at DEBUG, also on the Logs page.
  • Reproduce the problem, then read the first error, not the last one.

The server does not start​

Read the first error in its output. Common causes:

  • The port is in use. Another process listens on 8091. Stop it, or change listen.http.
  • frontend-directory has no index.html. Build the interface first, or remove the setting.
  • The configuration file is missing. A file given with --config or MINIPAM_CONFIG must exist.
  • The database migration failed. MiniPAM restored the safety copy and stopped. See If an upgrade fails.
  • Two versions share one database. Stop every other server that uses the same file.

curl http://localhost:8091/health returns {"status":"ok"} once the server is up.

Database errors at startup​

Any startup failure ends with Application startup failed while applying database migrations, followed by the configuration and database paths. That line is the same for every cause. Scroll up to the first error above it:

  • This is a non-empty database without an Alembic revision. The file comes from a pre-alpha MiniPAM, which has no upgrade path. Start the old version, export your inventory with Data transfer, then start the new version with a fresh database and import the file.
  • Database revision '…' is not supported by this MiniPAM build. The database was migrated by a newer MiniPAM. You cannot go back to an older version on it. Run the newer version again, or restore a backup made before the upgrade. See Upgrades and backups.
  • Timed out waiting for another MiniPAM process to finish database migration work. Another MiniPAM process holds the migration lock (database.db.migration.lock, next to the database). Stop the other server or CLI command, then start again.
  • Fresh database initialization at '…' failed. Creating a new database failed half-way. Delete the incomplete database file named in the message and start again.
  • Database migration from … to … failed. MiniPAM restored the safety copy. See If an upgrade fails.

Inventory looks empty after a restart or upgrade​

You sign in and see no sites or devices, or the Create initial admin form appears again.

Cause: the server opened a different, new database. Usually one of these:

  • The Docker data volume is not mounted, or a different one is: check -v minipam-data:/var/lib/minipam.
  • server.database.location changed. A relative path is resolved from the folder that holds the configuration file, so moving that file moves the database too.
  • The server started with a different configuration file.

Fix: find out which file the server opens: Administration → System → Database usage shows its path, and so does cli.py db usage. Point the configuration or the volume back at your real database and restart. Do not create a new admin on the empty database first.

The page loads, but nothing works​

Open the browser's developer tools, go to the Network tab, and look at one failing request.

  • Requests go to port 8000. The interface was built without VITE_API_BASE_URL. Rebuild it with VITE_API_BASE_URL=/ npm run build. See Install from source.
  • Requests fail with a CORS error. MiniPAM has no CORS support. Serve the interface and the API from the same address, either with frontend-directory or through one reverse proxy.
  • Saving anything fails with E2403 "The CSRF token is missing or invalid". Reading works, but every change is refused. The browser did not send the CSRF cookie that matches the page's token. Reload the page first: signing in again in another tab replaces the token. If it keeps happening behind a proxy, make sure the proxy passes Cookie and Set-Cookie headers unchanged, and that you open MiniPAM with the same scheme the server thinks it uses. With trust-proxy-headers: true and X-Forwarded-Proto: https, cookies are marked Secure, and a browser on plain http:// does not send them back.
  • You are signed out straight after signing in, behind HTTPS. You set cookie-samesite: none. Browsers reject such cookies unless they are marked Secure. Set trust-proxy-headers: true and make sure the proxy sends X-Forwarded-Proto: https, or go back to the default lax.

The error codes are listed in API errors.

There is no "Create initial admin" form​

The form only appears when no admin exists and bootstrap-token is set. Set the token and restart the server, or create the admin from the command line with cli.py auth set-root.

If an admin already exists, the form does not come back, even with a bootstrap token set. The token is single-use. To regain access, reset the admin password. See Create the first admin.

If the form shows up although you already had an admin, the server opened another database. See Inventory looks empty after a restart or upgrade.

Forgotten admin password​

Reset it with cli.py auth set-root --loginname <name>. See Reset the admin password.

An agent stays offline​

On the agent's machine, check the log: journalctl -u minipam-agent -f, or the terminal if you run it by hand.

  • Wrong or missing token. The log shows status_code=403, then the agent exits with failed to connect upstream service. The server refuses the connection before it opens, and logs nothing about it. The token must match the agent's token exactly (choose Reveal on it in Administration → Agents). As a service, the agent restarts every 5 seconds and fails the same way until you fix MINIPAM_KEY in /etc/minipam/agent.env.
  • Wrong address. It must end in /ws/agent, for example ws://minipam.lan:8091/ws/agent. The built-in default is out of date, so always pass --address.
  • The server is down when the agent starts. The first connection is not retried: the agent exits, and the service restarts it every 5 seconds until the server answers.
  • Wrong site. The agent appears under the site it was created in. Check the Active site selector.
  • A proxy in between. It must allow WebSocket upgrades and long-lived connections. Use wss:// when the proxy serves HTTPS.
  • DNS or certificates. From the agent's machine, curl https://minipam.example.com/health should work.

An agent that connects but is missing a module, such as nmap enrichment, has it disabled or lacks what it needs (nmap, root). See What an agent can do.

An agent stopped reconnecting​

The agent was online, the server or network was down for a long time, and the agent never came back.

Cause: after a lost connection, the agent retries 100 times. It waits 3 s, 5 s, 9 s, 20 s, 37 s, 1 min, and 2 min, then starts the cycle again, so it gives up after about an hour. It keeps running, and logs failed to reconnect upstream from ping loop every 5 seconds, so systemd does not restart it.

Fix: restart it: sudo systemctl restart minipam-agent.

Two agents keep replacing each other​

One agent flips between machines, or jobs on it hang.

Cause: two machines use the same token. Each new connection replaces the previous one. Work goes to the newest connection only, and results for requests sent to the other one are dropped.

Fix: create one agent, with its own token, per machine.

A job fails or never finishes​

Open the job under Discovery → Jobs and read its first error. See Read a failed job.

  • Agent 'N' is not currently online. (status start_fail). The job was never sent. MiniPAM does not queue work for offline agents. Bring the agent online, then run the job again.
  • The agent disconnected before the job completed. (status expired). The connection dropped mid-job. Jobs are not resumed. Run it again.
  • Failed to process agent message: … The server could not read a message from the agent. Usually the agent and server are different versions. Update the agent to match the server.
  • The server could not persist the validated agent result. The message was valid, but saving it failed. Look for agent_message_storage_failed in the server log, and for database errors around it.
  • The job stays running after a server restart. The server does not expire jobs that were running when it stopped. Run the job again.

A scan fails or finds little​

Open the job under Discovery → Jobs and read its first error.

  • ARP netscan needs a directly attached network, not a NAT network (see Where to run an agent), and the cap_net_raw and cap_net_admin capabilities, or root. Try sudo minipam-agent run arp-scan --list on the agent's machine.
  • Port probes and nmap enrichment need nmap on the agent's machine. UDP port probes need root. OS detection also needs root and allow-privileged-detail: true.
  • A script was refused. It is not in the agent's allowed-scripts.
  • Ping fails with a permission error. Allow unprivileged ping with sudo sysctl -w net.ipv4.ping_group_range="0 2147483647".

A scan that completes can still have thin evidence. Fix the cause, then use Scan now or Retry scan in the review queue instead of accepting a weak suggestion.

Nothing reaches the review queue​

A netscan finished, but the review queue stays empty or fills slowly.

Cause: netscan evidence is processed in the background, not straight away:

  • Schedules start 30 seconds after the server starts.
  • system.discovery.process_dirty runs every 60 seconds and handles up to 100 devices per run. A netscan that found 500 devices takes about 5 minutes to reach the queue.
  • A device is processed again only when its evidence changed. A scan that finds the same thing adds nothing.
  • Runs missed while the server was down, or while it was starting, are skipped, not caught up.

Device scan (nmap enrichment) results are processed as soon as the scan finishes.

Fix: wait a few minutes, and check that the system.discovery.process_dirty schedule is not paused under Discovery → Jobs → Schedules. See How the pipeline runs.

Devices wait in a device scan​

When you choose Preview device introspection under Discovery → Device Scans, some devices are listed under Waiting devices, each with the reason:

  • No current IP address is known. Run a netscan that sees the device.
  • The current IP address has ambiguous active network scope. The address is in more than one network. Fix the overlapping networks.
  • Waiting for a capable online agent with a route to this address. Bring an agent online that reaches the address and has nmap enrichment enabled.
  • This agent does not support the … profile. Choose another depth, or update the agent.
  • This agent cannot run the privileged Deep profile. Use Standard, or turn on allow-privileged-detail on the agent.
  • Waiting for scan capacity on this agent. Wait for running scans to finish, or raise the concurrency under Scheduling policy.
  • Introspection is excluded for this device., This device is outside the networks selected for introspection., or The current address is in an excluded network. The site's introspection policy leaves the device out. These lists have no page yet; admins read and change them through /v1/api/discovery/introspection/policy?site_id=….

A template is Blocked or Unprotected​

  • Blocked: a require failed. Open the template's Checks tab, read the failing message, fix the inventory or the requirement, and choose Run checks. The last approved output keeps being served in the meantime.
  • Unprotected: nothing guards the template. It still works, but automatic publication needs at least one require or linked check.
  • A render fails with a limit error. The template queried or produced too much. See Limits.

A template render or download fails​

  • Template rendering exceeded the 3-second safety limit. A render may take 3 seconds at most. Narrow the queries, and move repeated lookups out of loops.
  • The template renderer is busy. Try again in a moment. At most two renders run at once. Wait and retry.
  • E1404 "This template has no approved output." Nothing is published yet. A cron job that fetches …/outputs/download gets this until you approve the first output. See Templates.
  • "Withdraw the approved output before deleting…" You cannot delete an approved output, or a template that has one. Choose Withdraw on the output first.

The database is locked​

The log or an error message shows database is locked.

Cause: another process held a long write on the database. MiniPAM runs SQLite in WAL mode, so reading never blocks, but only one writer works at a time. A writer waits up to 5 seconds for the lock, then fails. Routine writes take milliseconds, so a longer wait usually means one of these:

  • cli.py db vacuum is running. It blocks writers until it finishes.
  • A large Data transfer import is running.
  • A second MiniPAM server, or another tool, writes to the same database.db.
  • The database is on a network share (NFS, SMB), where SQLite's file locking does not work reliably.

Fix: wait for the vacuum or import to finish and retry. Run db vacuum with the server stopped. Stop any other server or tool that opens the same file. Move the database to a local disk or Docker volume, set server.database.location, and keep database.db-wal and database.db-shm with it: move all three with the server stopped.

The database keeps growing​

Look at Administration → System → Database usage, or run cli.py db usage, to see which tables grow. Then lower the matching retention setting under System → Settings: jobs and job logs, discovery activity, template revisions, or template outputs.

Deleted rows leave free space that SQLite reuses, so the file does not shrink by itself. To give that space back to the disk, see Compact the database.

Asking for help​

Include the MiniPAM version, what you did, what you expected, what happened, the time it happened, any job ID, and the first specific error from the log. Leave out tokens, API keys, cookies, and unredacted exports.

Last updated on