NetOvi: One Intent, Four Network Operating Systems

In the last NetOvi post, everything still ran through an AI chat client and a handful of MCP tools, against eleven Cisco routers. A web portal was only an idea in its last paragraph, not built yet.

This post is the first time that portal is shown. It’s built, it runs, and the lab behind it is no longer eleven Cisco routers: it’s sixteen devices from four vendors.

The rule didn’t change. Every change to a device is proposed, shown as a diff, confirmed with a literal YES, applied, and verified, in code. What changed is how many ways there are to get that wrong.

What NetOvi is now

A web portal an engineer logs into (Keycloak single sign-on, users and groups from FreeIPA). Nineteen change types, from NTP and syslog to VLANs, OSPF, BGP and ACLs, each with its own form. Pick the devices, and the portal:

checks the change number against ServiceNow (approved, and inside its window) and against change freezes; these checks fail closed;
reads the device’s current state, renders the candidate from NetBox, and diffs the two;
shows the diff and waits for a literal YES;
applies, verifies, and writes a work note back to the ServiceNow change.

The NetOvi dashboard: 16 devices in NetBox, 1 site, 0 drift, 16 reachable — The dashboard after login: what NetBox knows, what the network answers, and whether anything has drifted.

Around that: a drift check that asks every device every half hour whether it still matches NetBox, opening a ServiceNow incident for every difference and resolving it by itself once the device matches again, config backups, scheduled changes, and a read-only AI assistant (a local model through Ollama) for troubleshooting. The assistant can query NetBox, run show commands, and read metrics and logs. It has no configuration tool at all, not even a diff-only one. Letting it propose changes again would be easy now, and the reason it doesn’t is the same one as in the last post: a confident, wrong answer about a device’s configuration is damage too.

The Automation page with the nineteen change types grouped by category — Nineteen change types, one form each.

A change form with a device selected, a ServiceNow change number and an optional run time — A form: devices from NetBox, the ServiceNow change number, and an optional time to run it later.

The lab: sixteen devices, four operating systems

Built in EVE-NG from generated configs, so the whole lab can be rebuilt from scratch:

Role	Platform	How the portal talks to it
Campus core and access	Cisco IOS (IOL), HSRP, rapid-PVST	SSH (NAPALM `ios`)
Data centre core	Cisco NX-OS 10.5, vPC	NX-API (NAPALM `nxos`)
DC fabric, spine and leaf	Arista EOS 4.29, EVPN-VXLAN, MLAG	eAPI (NAPALM `eos`)
MPLS PE	Juniper vMX 24.4, VRF over LDP	NETCONF (NAPALM `junos`)
Firewalls	Juniper vSRX cluster, FortiGate HA pair	not yet

Lab topology: HQ campus with FortiGates and Cisco core/access, an MPLS provider of two vMX, a branch with a vSRX cluster, and a data centre with Nexus cores and an Arista spine-leaf fabric — The lab in EVE-NG. Click for full size.

NetBox device list: the sixteen MVL devices with role, manufacturer, type and location — The same sixteen devices in NetBox, each with a role, a platform and a location. That's all the portal needs to find them.

One intent, four syntaxes

The design choice that made the rest possible: NetBox describes what, never how. The intent for NTP is “these servers, in the management VRF”, not an IOS line.

NetBox config context JSON with ntp servers 162.159.200.123 and 162.159.200.1 in VRF MGMT — The intent for NTP in a NetBox config context: two servers, the management VRF. Nothing vendor-specific.

Each platform gets its own renderer and its own collector (the code that reads what’s on the device now), both reading that same intent. The management VRF is the one thing that differs per platform, so it’s a NetBox config context scoped to the site and the platform. The same intent then renders as:

IOS     ntp server vrf MGMT 162.159.200.123
NX-OS   ntp server 162.159.200.123 use-vrf management
EOS     ntp server vrf MGMT 162.159.200.123
Junos   set system ntp server 162.159.200.123 routing-instance mgmt_junos

Where a platform simply doesn’t have a feature (NAT on a vEOS, spanning tree on a router), there’s no renderer, and the portal says “not supported for this platform” instead of guessing.

The demo: one change, four platforms

Adding a second NTP server is the smallest real change there is, which makes it a good one to watch. One line in the NetBox config context above, then the NTP form for one device of each platform:

The review page with a diff per device: IOS, EOS, NX-OS and Junos each get the new NTP server in their own syntax — The review page: one diff per device, each in its own syntax, and nothing applied yet. The AI line under each diff is a read-only health check of that device, for the reviewer.

Apply all with YES typed: Applied 4 of 4, and the first card says Applied and verified and recorded as a work note — One YES for all four, after reading every diff. Each device still keeps its own token and its own stale-diff check. Applied 4 of 4, each verified.

And on the devices themselves, the same intent in four syntaxes:

Terminal output on IOS, NX-OS, EOS and Junos showing both NTP servers configured — Read on the devices after the change: both servers, four syntaxes.

ServiceNow work note on the change: device, who typed YES, engine, result Applied and verified, and the reviewed diff — Every apply ends as a work note on the ServiceNow change: who confirmed it, the engine, the result and the diff that was reviewed.

SNMP form message: no changes needed, the device already matches NetBox — Running the same form again: nothing left to change.

The guardrails, in the screenshots

A YES means YES. Not Y, not yes please:

Two review cards: with Y typed the Confirm button stays disabled, with YES typed it turns green — Top: `Y`, Confirm stays disabled. Bottom: `YES`, and only then.

ServiceNow is checked before anything is read from a device, and fails closed. Here the change number was approved, but its planned window had passed:

Two error banners: the change is past its planned window, not allowed — Approved is not enough: the change window has to cover the moment it runs.

The third guard showed up without being staged. The two vSRX nodes form a chassis cluster that shares one configuration. A change committed on the first node was synced to the second, so the second node’s proposal no longer matched the device, and the portal refused to apply it: “live config changed since this was proposed; refusing to apply a stale diff.” Exactly the case that check exists for, on a device where I hadn’t expected it.

Drift, ServiceNow incidents, and the two switches I forgot

This part wasn’t planned either. When I first rolled the new NTP server out, the evening before the demo, I selected twelve devices and left out the two access switches. The next drift check found them:

MVL-C-ACC-1 running config shows only one NTP server — MVL-C-ACC-1 still had only the old server.

Drift page: 2 devices differ from NetBox, MVL-C-ACC-1 and MVL-C-ACC-2, ntp, each with a new incident number — The drift check: two devices out of line, the missing line as a diff, and an incident for each.

Every difference the drift check finds now opens a ServiceNow incident, one per device and change type, with the diff and the two ways to resolve it: run the change form (NetBox is right), or fix NetBox (the device is right). A difference that stays across runs keeps pointing at the same incident instead of opening a new one every half hour, and new incidents land in the network team’s queue.

ServiceNow incident: Drift, ntp on MVL-C-ACC-1 does not match NetBox, with the diff and how to resolve it in the description — The incident the drift check opened, written for whoever picks it up.

ServiceNow incident list: the two new drift incidents for the access switches — Both in ServiceNow, state New.

The fix went through the portal like any other change: the NTP form, the two access switches, the diff, YES.

Applying the fix to both access switches, three times the real speed: Apply all, both verified, a work note each, and the AI's health check arriving afterwards.

The next drift check found both switches matching NetBox again, and resolved the incidents itself. Nobody touched them in ServiceNow:

Drift page: 0 drifted, 304 checks, 0 errors, and resolved this run: INC0010003 and INC0010002 — Zero drift, and the two incidents resolved by this run.

ServiceNow incident list: the two drift incidents now Resolved — In ServiceNow: Resolved, with close notes saying when and why.

The read-only assistant

The assistant answers questions about the network from NetBox, show commands, Prometheus and Loki. I asked it in Dutch, which a local model handles fine:

AI assistant: asked whether any OSPF neighbors are down, it queried Prometheus and answered that all neighbors are Full — Are any OSPF neighbours down? One Prometheus query, and the answer names the devices it checked.

AI assistant: an OSPF neighbour table across IOS, NX-OS and Junos, built from one show command per platform — All OSPF neighbours in a table: one show command per platform, parsed into named fields before the model sees them.

AI assistant refusing to add an NTP server and pointing to the change forms — Asked to add an NTP server, it refuses and points to the change forms. It has no tool that could do otherwise.

Parsing the output before the model sees it mattered more than the prompt. Every vendor orders its OSPF columns differently (Junos puts the neighbour’s address first and its router ID fourth), and the model mixed them up as long as it had to map raw text itself.

Observability

Telegraf polls every device over SNMP, Promtail collects syslog, and Grafana shows both. The assistant reads the same data.

Grafana panels: interface traffic and interface error rate over the last hour — Interface traffic and error rate from the SNMP poll.

What I learned building it

Each vendor surprised me once. Every one of these only showed up when a real change hit a real device:

Arista EOS said “nothing to change” while the switch had actually rejected the line. Switching to Arista’s API (eAPI) made the error visible.
Cisco NX-OS combines lines. Changing a setting for one VLAN quietly changed it for another VLAN on the same line.
Juniper Junos fails the whole change if you ask it to delete something that isn’t there. The portal now only deletes what it has actually seen on the device.
Cisco IOS reported success but stored a password in a different format than asked.

“Applied” is not the same as “done”. So after every change, the portal reads the device again and compares it with NetBox once more. Only when nothing is left to change does it say verified. That check is what caught the IOS password and the NX-OS VLAN.

Everything was tested on real devices. Every change type, on every platform: applied, verified, checked again, and rolled back. 142 of 145 tests passed. Two failed because the virtual Arista switch has no QoS hardware, and one exposed a bug I had made earlier that day. The testing also found bugs in older code that had never run against real hardware.

NetBox has to be right. The portal treats NetBox as the truth, so a mistake in NetBox becomes a change on the network. The discovery tool I used to fill NetBox (Diode) also copied in things that were observed, not intended: a test address, and port settings a switch reports by itself. The drift check showed all of it. After cleaning NetBox up: 16 devices, 304 checks, zero drift.

What’s next

The FortiGate firewalls. They are in the lab and in NetBox, but the portal doesn’t change them yet: there is no NAPALM driver for FortiOS that is good enough to build on. The next phase looks at another engine for them, most likely Ansible’s FortiOS modules, with the same rules: the intent from NetBox, a diff first, a literal YES, and a check afterwards.

More testing. Every change type works on every platform now, but the lab can do more than one change at a time. Next: scheduled changes run end to end on a real device, more failure cases on purpose (a device that goes away mid-change, a ServiceNow change that is withdrawn), and whatever those turn up.

Resources

NAPALM: the ios, nxos, eos and junos drivers, merge candidates
NetBox config contexts: how contexts merge by weight
NetBox Diode: discovery into NetBox
Netmiko
EVE-NG