Skip to content

SONiC E2E: provision NetBox with docker compose - #2532

Draft
ideaship wants to merge 19 commits into
mainfrom
sonic-e2e-compose
Draft

SONiC E2E: provision NetBox with docker compose#2532
ideaship wants to merge 19 commits into
mainfrom
sonic-e2e-compose

Conversation

@ideaship

@ideaship ideaship commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Purpose

Draft, opened to get a CI timing number for the compose migration. It extends the SONiC E2E harness from:

with four scenario commits and then replaces the kind + Helm NetBox fixture with docker compose. Not for review yet — see Relationship to #2483 below.

Why

python-osism-sonic-e2e timed out (buildset 8ea91237, 2457s). It was aborted mid-generation, ~2 minutes short of finishing.

The job did not hang. Zuul's timeout is one budget spanning pre-run + run, and the job normally consumed 80–92% of its 2400s: seven successive runs of identical code took 1919–2212s pre+run. What tipped it over was node-local — ensure-pip : Update package lists took 301s instead of the usual 2–6s, adding ~409s to the pre-run. Not shared-mirror contention: the sibling jobs in the same buildset ran that identical task in 5s, 2s and 2s.

The underlying problem was that ~1650s of a ~2100s run was fixed setup: helm install --wait 704–792s and seeding 888–1011s.

What was measured

Kubernetes contributed nothing the test used, while the chart's defaults were a large part of the cost: PostgreSQL capped at 150m CPU, and a netbox-worker whose wait-for-backend init container hit CrashLoopBackOff and extended helm --wait. Image pulls were never the issue (netbox, 243 MB, pulled in 20.6s).

Comparable phases (provisioning + base example seeding), all measured on one 16-core host:

fixture provisioning base seeding total
kind + Helm, chart defaults (what CI runs today) 460s 433s 893s
kind + Helm, resource caps removed, worker off 349s 302s 651s
docker compose 241s 272s ~504s

Also measured: NetBox's expensive startup is one-time per fresh database — the image's entrypoint guards migrate/trace_paths/reindex behind migrate --check, so first boot took 339s and subsequent boots 50s and 60s.

Changes

  • nodeset: ubuntu-noble-large — the job ran a Kubernetes control plane, a four-container NetBox stack and the Ansible client on SCS-2V-4-10 (2 vCPU / 4 GB). Node size alone was worth 1.53× on the deploy and 2.19× on seeding.
  • tests/e2e/compose.yaml — postgres 17.10 / valkey 8.1 / netbox v4.5.10, with healthchecks and depends_on: service_healthy. NetBox is configured entirely through the environment variables its own baked configuration.py reads, so no config file is authored; the Helm chart had been overriding that file. API_TOKEN_PEPPERS is deliberately unset, because the image's bootstrap only ever creates a v2 token — which is why a v1 token still has to be minted for pynetbox / netbox.netbox.
  • tests/e2e/deploy_netbox.sh — docker compose up --wait replaces cluster creation, chart install and readiness polling. No port-forward: the API is published on 127.0.0.1:8080 directly, removing that process and its liveness guard.
  • tests/e2e/run.sh — keeps ownership of the lifecycle trap and teardown; diagnostics now capture the NetBox application log (which was unavailable when diagnosing the original failure — the cause had to be inferred from Kubernetes probe events).
  • playbooks/pre-sonic-e2e.yml — kind, kubectl and Helm installs removed along with their six pinned digests. ensure-docker already provides docker-compose-plugin. The accept_ra=2 sysctl is retained deliberately.

timeout stays at 2400.

Verification

Full local run: make sonic-e2e exit 0, OK: exports match the golden files. All 10 device configs byte-identical with no --regenerate — same NetBox version in, same configs out. End to end 9m34s.

What this PR is for is the CI number: pre+run against the ~2100s baseline. The projection is ~1250–1350s, extrapolated from 16 cores onto 8 vCPU and therefore unverified.

Relationship to #2483

This branch contains #2483's commits plus four scenario additions plus this migration. It is a draft opened to obtain CI timing; once green, the intent is to fold the result back rather than keep two overlapping PRs open.

Not included

  • --parallel 4 seeding (measured 302s → 167s) — deferred deliberately, since it is the only change that could alter NetBox state non-deterministically, and keeping it separate makes a golden mismatch unambiguously attributable.
  • netbox-manager is untouched and keeps its kind harness, so netbox-manager-e2e cannot regress.
  • Removing the accept_ra workaround, and pinning postgres/valkey by digest.

Known follow-ups

  • The harness seeds every file in tests/e2e/scenario/resources/, tracked or not. Harmless in CI's clean checkout, but local WIP silently joins the fixture.
  • ubuntu-noble-large has no min-ready (ubuntu-noble has 2), so this job now pays a cold VM boot per run — queue latency, outside the build timeout.

🤖 Generated with Claude Code

ideaship added 13 commits July 28, 2026 06:41
@ideaship
Make the SONiC port_config path configurable
976f932 
The directory holding the per-HWSKU port_config .ini files was
hardcoded to /etc/sonic/port_config, which only exists inside the
container image (the Dockerfile copies files/sonic/port_config there).
Running the config generator outside a container -- for local
development or an E2E test harness that points it at the in-repo
files -- required root to plant files under /etc/sonic.

Introduce a SONIC_PORT_CONFIG_PATH setting in osism.settings,
following the existing SONIC_* environment variable convention, and
wire constants.PORT_CONFIG_PATH to it. The default is unchanged, and
the value is read at import time like every other setting, so the
container behavior is identical.

Tests cover the default, the environment override, and the
settings-to-constants wiring.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add E2E golden-test harness for SONiC configs
0cace17 
Unit tests verify the SONiC config generator against fixture values
that encode the author's assumptions, so a class of regressions --
such as mixed kbps/Mbps interface speeds -- can pass the unit suite
while producing broken switch configurations. Nothing exercised the
full pipeline: real NetBox data in, complete config_db.json out.

Add a golden-file E2E harness that provisions NetBox on a local kind
cluster (reusing the deploy script from a netbox-manager checkout),
seeds it with the bundled example data, runs sync_sonic(), and
compares every exported config_db file against goldens committed
under tests/e2e/golden/. Any generator behavior change then surfaces
as a reviewable golden diff instead of shipping silently.

tests/e2e/generate.py drives the generation. sync_sonic() returns
only a device-to-config dict and logs-and-swallows per-device
failures, so the driver asserts success itself: the returned device
set must match the expectation derived from the golden file names,
empty configs fail, and a loguru sink turns any ERROR-level record
into a failure. --no-expect skips the device-set check for the
initial golden bootstrap.

tests/e2e/compare.py enforces exact file-set equality (missing,
extra, and mismatched exports are reported as distinct categories)
and reports mismatches both as structural paths such as
PORT|Ethernet4.speed and as a unified diff of the canonicalized
JSON. Canonicalization sorts dictionary keys only; array order is
part of the compared contract. --regenerate rewrites the goldens
canonically for review. Both modules are pure logic and covered by
the regular unit suite under tests/unit/e2e/.

tests/e2e/run.sh and the Makefile targets sonic-e2e, sonic-e2e-regen,
sonic-e2e-up and sonic-e2e-down tie the phases together. The exports
use hostname identifiers (the seed devices carry no serial numbers)
and the generator reads the in-repo port_config files via
SONIC_PORT_CONFIG_PATH. netbox-manager is installed from the checkout
rather than PyPI so a coordinated cross-repository change is tested
against the changed code and data.

Golden files are not yet committed; the initial set is bootstrapped
with make sonic-e2e-regen and reviewed by hand. A Zuul job and seed
data for known regression scenarios are follow-up work.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Build the SNMP vault only when secrets exist
97bd56c 
_add_snmp_configuration created a VaultLib instance for every device
before looking at the secrets custom field. get_vault() reads the key
file at /share/ansible_vault_password.key and fetches the encrypted
vault password from Redis, so outside a worker container -- local
development, or an E2E run against a plain NetBox -- it fails and
logs three ERROR-level messages per synced device, even though there
was nothing to decrypt in the first place.

Defer vault construction until the device actually carries a
non-empty secrets custom field. Devices with encrypted secrets are
decrypted exactly as before; devices without secrets no longer touch
the key file or Redis at all.

Found by the E2E harness error guard, which treats any ERROR-level
log record during config generation as a failure.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Isolate the E2E seeding tool in its own venv
d9128d8 
The harness previously installed netbox-manager into the project
venv. netbox-manager pins its own versions of packages that
python-osism pins exactly -- the install mutated five of them,
including pynetbox -- so a local E2E run silently left the developer
venv diverged from Pipfile.lock.

Install netbox-manager into a dedicated venv (.venv-sonic-e2e,
gitignored, reused across runs, overridable via SEED_VENV) instead.
The venv's bin directory is prepended to PATH because netbox-manager
drives Ansible through ansible-runner, which resolves ansible-playbook
via PATH rather than relative to its own interpreter.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add golden files for the SONiC E2E test
d80365e 
Bootstrap the golden set from the netbox-manager example data via
make sonic-e2e-regen: the four testbed switches plus the OOB switch,
generated by sync_sonic() against a seeded NetBox 4.5.10.

The files pin current generator behavior as a change detector; they
were cross-checked against their independent sources rather than
reviewed line by line:

- hostname, hwsku, management IP, loopback addresses (v4/v6) and the
  eth0 MAC match the seed resources for each device
- all 34 PORT entries of the AS7726-32X devices carry exactly the
  speeds of files/sonic/port_config/Accton-AS7726-32X.ini (the seed
  sets no NetBox speed overrides)
- BGP ASNs follow the documented rule (4200 prefix + loopback-derived
  suffix, e.g. 192.168.16.27 -> 4200016027) and the interconnected
  spine pair shares the group minimum 4200016029
- a second seed-and-generate cycle against a fresh NetBox deployment
  reproduced all five files byte-for-byte (determinism)

The VLAN tables are empty on every device: the example data assigns
VLANs only to the switches' own management interfaces, never to data
ports, so the generator's VLAN paths are not yet exercised. Coverage
for those (and for the breakout regression scenarios) comes with the
planned scenario seed data.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add the SONiC E2E golden test to Zuul
43a451d 
Wire the E2E golden test into CI as python-osism-sonic-e2e: provision
NetBox on a kind cluster, seed it with the netbox-manager example
data, run sync_sonic() and compare the exported config_db files
against tests/e2e/golden/ via tests/e2e/run.sh.

The netbox-manager checkout comes from required-projects, so the job
consumes its seed data and CLI at tip-of-main and a Depends-On change
is tested against the changed code and data. The pre-run playbook is
adapted from netbox-manager's pre-e2e.yml (same pinned kind, kubectl
and Helm versions with SHA256-verified downloads, same IPv6 accept_ra
workaround for SLAAC-only CI nodes) and must stay in sync with it
until the setup is factored into a shared zuul-jobs role. On top of
that it installs curl, openssl and python3-venv for the harness
script and its seeding venv.

In check the job carries a files matcher so it only runs for changes
that can alter the generated output: conductor code, settings, the
port_config files, the harness itself, the CI playbooks, and the
dependency pins. periodic-daily runs it unconditionally, bounding
silent drift of the netbox-manager seed data to a day.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Install the ansible extra before E2E generation
23faef3 
The CI run of python-osism-sonic-e2e failed at the generation step:
importing sync_sonic pulls in the whole conductor package, whose
utils module does `from ansible import constants` at import time, and
ansible-core was not present in the job's venv.

The gap is easy to miss because every other environment masks it: the
unit tests stub the ansible modules in tests/unit/conftest.py (so the
unit job and a local pytest run stay green without ansible-core), and
the container image installs requirements.ansible.txt explicitly.
The E2E driver is the first consumer of the real import chain in a
plain pipenv environment.

Have run.sh install the project's own [ansible] extra (which resolves
to requirements.ansible.txt, the same pin the container uses) into the
project venv before generating. The step is idempotent and covers
local runs from a fresh venv as well as CI.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add a breakout scenario to the SONiC E2E test
2f8e71f 
The bundled netbox-manager example models no breakout ports and sets
no explicit interface speeds, so breakout detection and the
kbps-to-Mbps speed normalisation are never exercised by the base
golden test -- exactly the generation logic behind the recent
customer-visible breakout speed regression.

Add a scenario overlay applied as a second seeding pass. It defines a
minimal synthetic device type (hwsku Accton-AS9726-32D) and two
devices that each break the 8-lane 400G master Eth1/1 into a 4x100G
group. The three non-master sub-ports are absent from the port_config
.ini and so are generated by _add_missing_breakout_ports. The two
devices differ only in where the sub-port speed comes from:

  e2e-breakout-derived   speed derived from the interface type; no
                         explicit NetBox speed. This is how real
                         deployments model breakouts.
  e2e-breakout-explicit  the same sub-ports with an explicit NetBox
                         speed in kbps -- the other unit the
                         collection step must normalise.

Both must yield sub-port speed 100000. A tagged-VLAN port is added on
the derived device for VLAN/VLAN_MEMBER coverage the base example
lacks.

The seed data is synthetic; real customer NetBox configs were used
only as a modelling reference for interface naming and speed
conventions, with no customer identifiers copied.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add golden files for the breakout scenario
8ae22d3 
Goldens for the two breakout scenario devices, generated against a
seeded NetBox. Both devices produce sub-port speed 100000 for the
4x100G group -- the same value whether the speed is derived from the
interface type or set explicitly in kbps.

Verified the guard by reverting the fix locally against these goldens:

  - Restoring the mixed-unit collection (explicit speed left in kbps)
    together with the downstream division turns the derived golden red
    (Ethernet0/2/4/6 speed 100000 -> 100, the customer-visible
    symptom) while the explicit golden stays green.
  - Dropping the division instead turns the explicit golden red
    (speed 100000 -> 100000000, raw kbps) while the derived golden
    stays green.

The current code keeps both green, so either regression would now
fail the E2E test with a speed diff on the affected device.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add a port-channel scenario to the SONiC E2E test
31c8d53 
The base example models no LAGs, so PORTCHANNEL, PORTCHANNEL_MEMBER
and PORTCHANNEL_INTERFACE are emitted-but-empty. Add a device that
bonds two data ports into PortChannel1, modeled the way NetBox and
real deployments express a LAG: a type=lag interface plus the member
ports referencing it via their lag field.

Introduces a shared minimal device type (edgecore-7726-32x-e2e,
hwsku Accton-AS7726-32X) with a management port and two data ports.
The golden pins the resulting PORTCHANNEL / PORTCHANNEL_MEMBER /
PORTCHANNEL_INTERFACE entries. Seed data is synthetic; the base
example switches and the existing scenario goldens are unaffected.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add an EVPN/VXLAN/VRF scenario to the SONiC E2E test
57d6c31 
The base example assigns no VRFs, so the whole EVPN/VXLAN/VRF
subsystem is emitted-but-empty: VRF, VXLAN_TUNNEL, VXLAN_TUNNEL_MAP,
VXLAN_EVPN_NVO, the L2VPN-EVPN BGP_GLOBALS_AF and
BGP_GLOBALS_ROUTE_ADVERTISE. Generation keys purely on NetBox VRF
objects assigned to interfaces (interface.vrf) -- no config context
or tags are involved.

Add a leaf device with a Loopback0 (its primary IP becomes the VXLAN
tunnel source) and two VRFs assigned to data ports:

  VrfStorage  rd 2001 (a pure number) -> VNI 2001, which populates all
              of the tables above (route-target auto-derived as 2001:1)
              plus the synthesized Vlan2001 and its VLAN_INTERFACE.
  vrf42       no rd -> the table_id-only branch: VRF Vrf42 with
              vrf_table_id 42 and no EVPN/VXLAN.

The golden pins the resulting tables. Seed data is synthetic and
reuses the shared device type from the port-channel scenario; the
base switches and existing goldens are unaffected.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
sonic: add explicit breakout_mode declaration
31bf367 
Give SONiC breakout detection an explicit, authoritative signal instead
of inferring breakout structure from incidental NetBox artifacts. A
device-level custom field `sonic_parameters.breakout` maps a master port
(NetBox `Eth1/N` or canonical `EthernetN`, normalized via the hwsku
port_config) to a mode string `NxSpeedG`.

When a declaration is present for a resolvable master it is
authoritative and fail-closed: the master is claimed into
`suppressed_masters` before validation, so the inference path is
suppressed for it even when the declared mode is invalid or two keys
collide. The mode is validated structurally against the port_config lane
count (L % N == 0); children and their exact per-child lane slices are
computed from the mode (fixing 2x*/8x* which the count-based inference
never handled), the physical port comes from the port_config index
(correct on mixed-lane platforms), and config_generator uses the
declared per-child speed and lanes ahead of any NetBox-derived value.

Absent a `breakout` map, behaviour is unchanged by construction: the
declared pass no-ops, `suppressed_masters` stays empty, and the
inference branches (which only consult the set, never populate it) and
their existing dedup are untouched. Structural validation only; the
platform (platform.json) may still reject a structurally-valid mode.

Adds unit coverage for the parsers, the resolver, mode emission across
4x/2x/8x/4x100G, key normalization, mixed-layout port index, collision
and invalid/unresolvable/single-lane/malformed declarations, and the
declared-child downstream precedence.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Add declared-breakout scenario to SONiC E2E
71e40e6 
Add e2e-breakout-declared to the SONiC E2E golden test: an
Accton-AS9726-32D switch carrying an authoritative device-level
sonic_parameters.breakout map (no sub-ports modelled in NetBox), so a
golden diff proves the explicit declared-mode path rather than the
inference fallback.

The map covers three splits on the 8-lane platform: Ethernet0 4x100G
(2 lanes/child), Ethernet8 2x50G (4 lanes/child, the case count-based
inference never handled), and physical key Eth1/9 -> Ethernet64 8x50G
(1 lane/child, exercising key normalization). The committed golden
records the expected BREAKOUT_CFG (MANUAL owner, mode string, port_config
index), BREAKOUT_PORTS (child -> {master} only), and per-child PORT
lanes/speed.

Regenerating the goldens left all existing files byte-unchanged,
confirming the merged explicit-mode feature does not perturb the
inference path.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
This was referenced Jul 30, 2026
Closed
Closed
Closed
Closed
ideaship added 4 commits July 30, 2026 11:19
@ideaship
tests/e2e: add compose NetBox fixture
9e87f46 
The SONiC E2E golden test only needs a NetBox REST API, its
PostgreSQL and its Valkey -- it does not need Kubernetes. Replace
the kind cluster and Helm chart with a docker compose stack
(postgres, valkey, netbox) defined in tests/e2e/compose.yaml, plus
a deploy_netbox.sh that starts the stack and mints the API token.

NetBox is configured entirely through the environment variables its
own baked /etc/netbox/config/configuration.py reads, so no
configuration file needs to be mounted. API_TOKEN_PEPPERS is
deliberately left unset: the image's super_user.py only creates an
API token when a pepper is configured, and then only a "v2" one,
which pynetbox / netbox.netbox cannot use. deploy_netbox.sh instead
mints a deterministic v1 token for the superuser via `manage.py
shell`, which NetBox still accepts through v4.6. The NetBox version
is pinned to v4.5.10 because the golden files were generated against
it; postgres and valkey are pinned to a patch level.

compose.yaml's only interpolated value is NETBOX_PORT (default
8080), so `docker compose down`, `ps` and `logs` all work from an
environment where it is unset. The secret key, superuser password
and DB password are fixed literals rather than ${VAR:?} because this
stack is ephemeral, loopback-only, and thrown away after every run.

This commit is purely additive: tests/e2e/run.sh still calls
netbox-manager's kind-based deploy script, so the existing E2E
harness keeps working unchanged. Wiring run.sh to the new compose
stack is the next commit.

Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
tests/e2e: use compose instead of kind
d0eed6f 
The SONiC E2E job spent roughly 1650s of its ~2100s pre+run time on
the kind cluster bring-up, the Helm-installed NetBox chart, and
seeding through a port-forward, while Kubernetes itself contributed
nothing the test exercises -- it only added resource caps and worker
backoff behind that slowness. tests/e2e/compose.yaml now runs NetBox
(postgres, valkey, netbox) directly on the host via docker compose.

run.sh is rewired accordingly: phase 1 now calls the local
tests/e2e/deploy_netbox.sh, which starts the stack and mints the API
token; the port-forward and its readiness poll are gone because
`up --wait` already establishes readiness via healthchecks and the
API is published on 127.0.0.1:NETBOX_PORT directly. Cluster-name and
namespace variables are replaced by a `compose()` helper bound to
compose.yaml. Diagnostics on failure now dump `compose ps` and the
NetBox application log instead of kubectl node/pod/event output, and
cleanup tears the stack down with `compose down --volumes
--remove-orphans` instead of deleting a kind cluster.

playbooks/pre-sonic-e2e.yml no longer installs kind, kubectl or Helm
(the pinned versions/checksums and the six related tasks are
removed); ensure-docker already installs docker-compose-plugin
alongside docker-ce, so the node needs no new package. The accept_ra
IPv6 workaround stays, since the CI node's SLAAC default route is
still at risk once Docker enables forwarding.

The Makefile targets are repointed the same way: sonic-e2e/-regen no
longer pass CLUSTER_NAME, sonic-e2e-up runs the local
deploy_netbox.sh, and sonic-e2e-down calls `docker compose ... down`
instead of `kind delete cluster`.

netbox-manager itself is untouched and still provisions its own E2E
NetBox on kind; only this repo's fixture moved to compose.

Verified with a full local `make sonic-e2e`: NetBox boots, is seeded
with the netbox-manager example data plus the three scenario overlay
files, 10 SONiC device configs are generated, and the exports match
tests/e2e/golden/ unchanged.

Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
tests/e2e: polish fixes from final review
bf69685 
Five fixes found in the final review of the docker compose NetBox
fixture for the SONiC E2E golden test:

- run.sh: cap dump_diagnostics' `compose logs` at the last 200 lines
  (--tail 200). Uncapped, it fires on the most common failure -- a
  golden mismatch in phase 4, where compare.py has already printed the
  useful diff -- and buries that diff under megabytes of NetBox
  first-boot migration and postgres per-connection logs in the Zuul
  console. 200 lines still covers a genuine boot failure.

- pre-sonic-e2e.yml: rewrite the accept_ra comment, which still
  justified the guard by citing `kind create cluster` even though kind
  was removed from this playbook in an earlier commit. It now explains
  the guard in terms of the current setup (compose's IPv4-only default
  bridge probably won't trigger IPv6 forwarding) while noting the
  guard is kept deliberately as cheap insurance, and that removing it
  should be its own experiment.

- pre-sonic-e2e.yml: correct the "Install required packages" comment,
  which claimed curl is used by run.sh directly; it is now only used
  by the NetBox container's healthcheck.

- .zuul.yaml: update the stale job comment saying NetBox is provisioned
  "on kind" to describe the docker compose stack.

- deploy_netbox.sh: validate NETBOX_TOKEN is exactly 40 hex characters
  right after it is defaulted. This guards against a caller-supplied
  token breaking out of the Python string literal it is interpolated
  into (a quote or newline), and against a malformed token being
  silently rejected by NetBox much later, far from the actual cause.

Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
tests/e2e: seed NetBox in parallel
219205b 
Seeding was the largest remaining cost: 555s of a 1335s pre+run. Files
are grouped by leading number and only files within one group run
concurrently, so 000 -> 100 -> 200 -> 300 ordering still holds.
Measured in CI: the base example pass dropped from 555s to 250s
(2.22x), taking pre+run from 1335s to 1025s.

The overlay pass gets the same setting for consistency but does not
benefit -- its files are in distinct numeric groups and serialise
regardless (49.8s parallel vs 47.4s serial, i.e. unchanged).

SEED_PARALLEL is the escape hatch, defaulting to 4. Set it to 1 to
reproduce the original strictly sequential behaviour, either locally
or, via the sonic_e2e_seed_parallel job var, in CI. That matters
because parallel seeding is the one change here that could in
principle alter NetBox state non-deterministically, so a suspicious
failure needs to be cheap to re-test serially.

A race is much likelier to surface as a spurious failure than a false
pass: interleaving would change the generated configs and the golden
comparison would catch it. Static analysis supports that it should not
race at all -- the 16 files in the 300- group touch 251 distinct
objects with no overlap between files.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship force-pushed the sonic-e2e-compose branch from a747863 to 219205b Compare July 30, 2026 09:22
ideaship added 2 commits July 30, 2026 11:44
@ideaship
tests/e2e: seed serially by default
2c6c765 
Concurrent seeding deadlocks. Reverting the default to 1 while keeping
the knob.

--parallel 4 measured 555s -> 250s (2.22x) and passed twice, so it
looked safe. The static argument for it was that no two resource files
write the same object -- 251 distinct objects, zero overlap across the
16 files in the 300- group. That analysis was not sufficient.

The files share foreign key parents. Every node file creates cables
terminating on the shared switches, and inserting a row that references
a device takes a KEY SHARE lock on that device's row for the FK check.
Two transactions taking those parent locks in opposite orders deadlock:

  deadlock detected ... while locking tuple (1,3) in relation
  "dcim_device"
  SELECT 1 FROM ONLY "dcim_device" x WHERE "id" = $1 FOR KEY SHARE OF x

Six files were in flight, including two node files and three switch
files. Two CI runs passed and the third failed this way -- roughly a
one-in-three flake rate, unusable as a default.

It fails safely: a deadlock aborts the run rather than silently
reordering writes, so the golden files were never at risk. That matches
the expectation that a race here would show up as a spurious failure
rather than a false pass.

SEED_PARALLEL=4 opts back in. Worth revisiting only if netbox-manager
gains deadlock retry, because the ~300s saving is real.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
tests/e2e: rename run.sh to sonic_golden_test.sh
def9860 
"run.sh" says nothing about what the script does and is unusable as a
reference without its full path. Both matter here.

python-osism already has a /run.sh with an unrelated meaning: the
osism-ansible container entrypoint that the task wrappers shell out to
under a Redlock. Grepping for run.sh in this repo therefore returned
production code paths mixed with this test harness.

It was also inconsistent with its own directory, where every sibling is
named for what it does: compare.py, compose.yaml, deploy_netbox.sh,
generate.py.

The name was inherited by copying netbox-manager's tests/e2e/run.sh.
That convention is sound for container entrypoints -- one per image,
referenced only from the Dockerfile -- but for a test harness it is
filler that relies on the directory to carry all the meaning.

Renamed with git mv so history follows, and updated the two invocations
plus the surrounding comments. The unrelated /run.sh references in
osism/tasks/ and tests/unit/ are deliberately untouched.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

Human Board
Status: New

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ideaship @osism-agent