Failover from backups

When a host goes down for good (hardware failure, lost data center), NexoVirt helps you bring its guests back on another host from their newest Proxmox Backup Server backups. Failover is assisted, never automatic: the panel detects the outage and prepares everything, but only an admin with the hosts.manage permission starts it.

Caution

A failover restores from the last backup. Everything written after that backup is not on the new copy. For a host that is still reachable, use migration instead.

Requirements

• The guests are backed up to a PBS datastore that is attached to the failed host.
• The same PBS server is attached to the target host, so the target can read those backups.
• The target host has room for the guests (RAM and a storage with enough space).

When a host counts as down

A host is down only when both of these have been silent for the configured time:

• its agent heartbeat, and
• its newest node inventory row (the window is at least twice the inventory interval).

Hosts in maintenance mode and hosts that were never inventoried are never treated as down. This check uses the panel’s own data only, so a short network blip or a single missed heartbeat does not trigger it.

When a host is down:

• admins with hosts.manage see a red banner on every panel page, “Node name is unreachable since HH:MM.”, with an Open failover button,
• a host_down alert fires through the alerting engine,
• the down host no longer blocks the panel through the agent health check.

Backup protection

Each guest’s protection is based on its newest PBS backup:

Protection	Meaning
Protected	A backup exists and is newer than the maximum backup age.
Stale	The newest backup is older than the maximum backup age.
No backup	No backup of this guest was found on the PBS servers attached to its host.

A guest’s page shows its Failover protection in the details card. On the Failover page you can Preview any host, even a healthy one, to check the protection of all its guests before you need it. A host that is up can only be previewed, not failed over.

Running a failover

1. Click Open failover in the banner, or open Failover in the sidebar. With several down hosts, switch between them with the buttons in the card header.
2. Review the guest list: Last backup, Protection, Target and Pre-flight per guest.
3. Choose the Target: Automatic (best node with the PBS server attached) or a specific node. Hosts in maintenance or unreachable are listed but disabled.
4. Choose the IP zone on the target: Keep the IP when possible, or a zone to take new addresses from. Turn on Drop port-forwards if forwards should not move along.
5. Select the guests and click Fail over selected.

The pre-flight runs per guest and refuses, for example, guests without a backup, targets without room, or targets that do not have the PBS server attached. The target storage is chosen automatically.

What happens per guest

You land on the run page (Failover of host), which refreshes every 5 seconds. Guests are restored one at a time. For each guest the panel:

1. checks again that the source host is still down. If it is reachable again, that guest is aborted with “The source node is reachable again. Use migration instead.”,
2. keeps the guest’s VMID on the target when it is free, otherwise takes the next free one,
3. restores the chosen backup on the target (it never overwrites an existing guest),
4. applies the IP plan, moves port-forwards, moves the panel records (owner, plan and so on) to the restored guest,
5. starts the guest if it was running before.

Problems after the restore (IP, port-forwards, records, start) finish the guest with warnings instead of failing it; check the restored guest’s IP, port-forwards, owner and power state. If the restore itself fails, only the guest this failover created is removed. Stop on the run page skips the guests that have not started yet.

Starting a failover does not put the failed host into maintenance mode.

The source copies

On the failed host, each failed-over guest is now a source copy. Its page shows a Failed over notice with the new location and backup time. Every action on it is refused, and it is left out of evacuations, alerts and plan enforcement.

When the host comes back

When the failed host returns, its guests may start again on their own (for example through autostart). NexoVirt checks every inventory run:

• A source copy found running is stopped immediately, and a failover_conflict alert is raised in the inbox. This alert is not resolved automatically: review the situation, then resolve it yourself.
• The Failover page lists these copies under Returned nodes with their status and where the guest now runs. Select them and click Delete source copies, or use Delete source copy on a copy’s page.

Tip

Delete the source copies once you have confirmed the restored guests work. Until then they stay stopped and blocked, so they cannot clash with the restored guests.

Settings

The failover settings are under Settings → Placement → Failover:

• Node down after (minutes): how long the agent and inventory must both be silent. Default 5.
• Max backup age (hours): backups older than this count as Stale. Default 24.

Failover runs are executed by the migrate:work worker, like migrations. See Migration & balancing.

Next steps

• Backups (PBS) to set up backup jobs and attach PBS to every host.
• Alerts & webhooks to route host_down and backup alerts.