BGP w/OVN: nexthop value not synchronised across cluster nodes #14531
Comments
|
In case it helps, it looks like restarting the LXD snap causes those values to re-sync: jsimpso@mc-002:~$ lxc query /internal/testing/bgp{
"peers": [
{
"address": "10.0.25.1",
"asn": 64513,
"count": 1,
"holdtime": 0,
"password": ""
}
],
"prefixes": [
{
"nexthop": "10.0.10.3",
"owner": "network_3",
"prefix": "172.16.0.128/28"
},
{
"nexthop": "0.0.0.0",
"owner": "network_4",
"prefix": "172.16.0.0/28"
},
{
"nexthop": "0.0.0.0",
"owner": "network_5",
"prefix": "172.16.1.0/25"
}
],
"server": {
"address": "10.0.25.11",
"asn": 64512,
"router_id": "10.0.25.11",
"running": true
}
}jsimpso@mc-002:~$ sudo snap restart lxd
2024-11-28T11:07:48+08:00 INFO Waiting for "snap.lxd.daemon.service" to stop.
Restarted.
jsimpso@mc-002:~$ lxc query /internal/testing/bgp{
"peers": [
{
"address": "10.0.25.1",
"asn": 64513,
"count": 1,
"holdtime": 0,
"password": ""
}
],
"prefixes": [
{
"nexthop": "10.0.10.4",
"owner": "network_4",
"prefix": "172.16.0.0/28"
},
{
"nexthop": "10.0.10.5",
"owner": "network_5",
"prefix": "172.16.1.0/25"
},
{
"nexthop": "10.0.10.3",
"owner": "network_3",
"prefix": "172.16.0.128/28"
}
],
"server": {
"address": "10.0.25.11",
"asn": 64512,
"router_id": "10.0.25.11",
"running": true
}
} |
|
Thanks @jsimpso this is high on my list to work on. |
|
We should also consider here whether we can relax the requirements that all LXD cluster members are online when modifying OVN load balancers and forwards as currently the only reason for that is to update the local per-member BGP exporters. |
When you did this, were the networks that didn't have their MTU changed also affected (WRT to their BGP nexthop) or was it only the network that was being edited that was affected? |
|
Please can you show |
|
An initial attempt to reproduce this issue hasn't been successful: I observed the nexthop address did not go to 0 on updating the ovn network's bridge.mtu on any of the cluster members. |
Do you see the same issue if editing the network's config using |
Only the network that had a configuration change was affected
Before:jsimpso@mc-001:~$ lxc network show dev-test
name: dev-test
description: ""
type: ovn
managed: true
status: Created
config:
bridge.mtu: "1442"
ipv4.address: 172.16.0.1/28
ipv4.nat: "false"
ipv6.address: none
network: UPLINK
volatile.network.ipv4.address: 10.0.10.4
used_by:
- /1.0/profiles/default?project=dev-test
locations:
- mc-001
- mc-002
- mc-003BGP status ( mc-001
{
"nexthop": "10.0.10.4",
"owner": "network_4",
"prefix": "172.16.0.0/28"
}
mc-002
{
"nexthop": "10.0.10.4",
"owner": "network_4",
"prefix": "172.16.0.0/28"
}
mc-003
{
"nexthop": "10.0.10.4",
"owner": "network_4",
"prefix": "172.16.0.0/28"
}After changing
|
Just tested making the same change through the CLI and no, that doesn't seem to cause the same issue. When the network is changed through the CLI each member shows the expected nexthop value. FWIW It also doesn't seem to matter what field is modified when the change is made through the UI, I've just reproduced it by changing only the network's description.
The record I'm using to access the UI resolves to the IP of all three cluster members I'll test overriding it to resolve a single host and see if the behaviour is different |
|
The active chassis for this network is
|
|
Confirmed this matches on all three members |
|
@jsimpso would you be able to run cc @edlerd I wont transfer this to lxd-ui repo just yet as not 100% sure its a UI bug yet. |
tomponline
added
Bug
From the UI code, it seems to be a |
Interesting here, that the nexthop is correct for one of the three members, but null for the other two. Which host is the one that serves the UI and its API to you? |
I tried to cut this down to what's relevant without pulling out anything useful, but I can grab a more verbose sample if needed: The target this time was |
@edlerd it appears that whichever host I target to serve the UI is the one that has the correct nexthop value after the change. I'll run |
|
Here's the output from another node (all I'm doing to trigger this now is flipping the network description between |
|
thanks for this @jsimpso |
|
@jsimpso now please could you get the same |
|
We can see the request from the UI here and the notification to the other member here: |
|
Were they the same requests @jsimpso as I see a difference between "test" and "testing" in the two requests at the end. |
Those were indeed two different requests, I was switching the value back and forth between "test" and "testing" to trigger the issue. I'll re-run that and capture the same time period for all three nodes when changing via the UI, and then repeat for changing via the CLI |
Changing network configuration via web UIPre-test
Making the change
ResultChecking the nexthop values after the change shows that the other two nodes ( jsimpso@mc-001:~$ lxc query /internal/testing/bgp | jq '.prefixes.[] | select(.prefix=="172.16.0.0/28") | .nexthop'
"10.0.10.4"
jsimpso@mc-002:~$ lxc query /internal/testing/bgp | jq '.prefixes.[] | select(.prefix=="172.16.0.0/28") | .nexthop'
"0.0.0.0"
jsimpso@mc-003:~$ lxc query /internal/testing/bgp | jq '.prefixes.[] | select(.prefix=="172.16.0.0/28") | .nexthop'
"0.0.0.0"Outputmc-001jsimpso@mc-001:~$ lxc monitor --pretty
DEBUG [2024-12-18T10:22:43+08:00] Event listener server handler started id=6b51a3ea-f15b-4c25-aed6-1abc74a90d6b local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2024-12-18T10:22:44+08:00] Handling API request ip="10.0.37.136:48824" method=PUT protocol=tls url="/1.0/networks/dev-test?project=dev-test" username=5273a69d19633c2b9aa19daf7e4dc2ee4384c05c7640fa5fe8c7d901e32095b2
DEBUG [2024-12-18T10:22:44+08:00] Matched trusted cert fingerprint=5273a69d19633c2b9aa19daf7e4dc2ee4384c05c7640fa5fe8c7d901e32095b2 subject="CN=jsimpso@nautilus,O=LXD"
DEBUG [2024-12-18T10:22:44+08:00] Update clientType=normal driver=ovn network=dev-test newNetwork="{map[bridge.mtu:1442 ipv4.address:172.16.0.1/28 ipv4.nat:false ipv6.address:none network:UPLINK] changed-via-ui}" project=dev-test
DEBUG [2024-12-18T10:22:44+08:00] Notify node 10.0.25.11:8443 of state changes
DEBUG [2024-12-18T10:22:44+08:00] Notify node 10.0.25.12:8443 of state changes
DEBUG [2024-12-18T10:22:44+08:00] Connecting to a remote LXD over HTTPS url="https://10.0.25.12:8443"
DEBUG [2024-12-18T10:22:44+08:00] Connecting to a remote LXD over HTTPS url="https://10.0.25.11:8443"
DEBUG [2024-12-18T10:22:44+08:00] Sending request to LXD etag= method=PUT url="https://10.0.25.11:8443/1.0/networks/dev-test?project=dev-test"
DEBUG [2024-12-18T10:22:44+08:00] Sending request to LXD etag= method=PUT url="https://10.0.25.12:8443/1.0/networks/dev-test?project=dev-test"
INFO [2024-12-18T10:22:44+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:22:44+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
DEBUG [2024-12-18T10:22:44+08:00] Setting up network driver=ovn network=dev-test project=dev-test
DEBUG [2024-12-18T10:22:44+08:00] Stable MAC generated driver=ovn hwAddr="00:16:3e:1f:44:31" network=dev-test project=dev-test seed=37ec2b324056d390fc3562e98362abee62523bb6d4f99484621c844b90dbd22a.0.4
DEBUG [2024-12-18T10:22:46+08:00] Matched trusted cert fingerprint=676ee46ad7a0347fc2bc6e65a2aac7f57c487cfccfc199c8f5e67bce92439d51 subject="CN=root@mc-002,O=LXD"
DEBUG [2024-12-18T10:22:46+08:00] Replace current raft nodes raftMembers="[{{1 10.0.25.10:8443 voter} mc-001} {{2 10.0.25.11:8443 voter} mc-002} {{3 10.0.25.12:8443 voter} mc-003}]"
INFO [2024-12-18T10:22:47+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: tls/5273a69d19633c2b9aa19daf7e4dc2ee4384c05c7640fa5fe8c7d901e32095b2 (10.0.37.136:48824)
INFO [2024-12-18T10:22:47+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: tls/5273a69d19633c2b9aa19daf7e4dc2ee4384c05c7640fa5fe8c7d901e32095b2 (10.0.37.136:48824)
DEBUG [2024-12-18T10:22:47+08:00] Matched trusted cert fingerprint=5273a69d19633c2b9aa19daf7e4dc2ee4384c05c7640fa5fe8c7d901e32095b2 subject="CN=jsimpso@nautilus,O=LXD"
DEBUG [2024-12-18T10:22:47+08:00] Handling API request ip="10.0.37.136:48824" method=GET protocol=tls url="/1.0/networks/dev-test?project=dev-test" username=5273a69d19633c2b9aa19daf7e4dc2ee4384c05c7640fa5fe8c7d901e32095b2
DEBUG [2024-12-18T10:22:47+08:00] Matched trusted cert fingerprint=efa460eca20a32479bbb6b3d415d6e9340196ea717e93213a784686c82430c52 subject="CN=metrics.local"
DEBUG [2024-12-18T10:22:47+08:00] Handling API request ip="10.0.25.24:45208" method=GET protocol=tls url=/1.0/metrics username=efa460eca20a32479bbb6b3d415d6e9340196ea717e93213a784686c82430c52
DEBUG [2024-12-18T10:22:47+08:00] Connecting to a VM agent over a VM socket
DEBUG [2024-12-18T10:22:47+08:00] Sending request to LXD etag= method=GET url="https://custom.socket/1.0"
DEBUG [2024-12-18T10:22:47+08:00] Got response struct from LXD
DEBUG [2024-12-18T10:22:47+08:00] Sending request to LXD etag= method=GET url="https://custom.socket/1.0/metrics"
DEBUG [2024-12-18T10:22:47+08:00]
{
"config": null,
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"backup_compression",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode",
"resources_pci_vpd",
"qemu_raw_conf",
"storage_cephfs_fscache",
"network_load_balancer",
"vsock_api",
"instance_ready_state",
"network_bgp_holdtime",
"storage_volumes_all_projects",
"metrics_memory_oom_total",
"storage_buckets",
"storage_buckets_create_credentials",
"metrics_cpu_effective_total",
"projects_networks_restricted_access",
"storage_buckets_local",
"loki",
"acme",
"internal_metrics",
"cluster_join_token_expiry",
"remote_token_expiry",
"init_preseed",
"storage_volumes_created_at",
"cpu_hotplug",
"projects_networks_zones",
"network_txqueuelen",
"cluster_member_state",
"instances_placement_scriptlet",
"storage_pool_source_wipe",
"zfs_block_mode",
"instance_generation_id",
"disk_io_cache",
"amd_sev",
"storage_pool_loop_resize",
"migration_vm_live",
"ovn_nic_nesting",
"oidc",
"network_ovn_l3only",
"ovn_nic_acceleration_vdpa",
"cluster_healing",
"instances_state_total",
"auth_user",
"security_csm",
"instances_rebuild",
"numa_cpu_placement",
"custom_volume_iso",
"network_allocations",
"storage_api_remote_volume_snapshot_copy",
"zfs_delegate",
"operations_get_query_all_projects",
"metadata_configuration",
"syslog_socket",
"event_lifecycle_name_and_project",
"instances_nic_limits_priority",
"disk_initial_volume_configuration",
"operation_wait",
"cluster_internal_custom_volume_copy",
"disk_io_bus",
"storage_cephfs_create_missing",
"instance_move_config",
"ovn_ssl_config",
"init_preseed_storage_volumes",
"metrics_instances_count",
"server_instance_type_info",
"resources_disk_mounted",
"server_version_lts",
"oidc_groups_claim",
"loki_config_instance",
"storage_volatile_uuid",
"import_instance_devices",
"instances_uefi_vars",
"instances_migration_stateful",
"container_syscall_filtering_allow_deny_syntax",
"access_management",
"vm_disk_io_limits",
"storage_volumes_all",
"instances_files_modify_permissions",
"image_restriction_nesting",
"container_syscall_intercept_finit_module",
"device_usb_serial",
"network_allocate_external_ips",
"explicit_trust_token"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"auth_user_name": "",
"auth_user_method": "",
"environment": {
"addresses": null,
"architectures": null,
"certificate": "",
"certificate_fingerprint": "",
"driver": "",
"driver_version": "",
"instance_types": null,
"firewall": "",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": null,
"kernel_version": "5.15.0-1069-kvm",
"lxc_features": null,
"os_name": "",
"os_version": "",
"project": "",
"server": "lxd-agent",
"server_clustered": false,
"server_event_mode": "",
"server_name": "juju-32f1c2-0",
"server_pid": 369,
"server_version": "5.21.2",
"server_lts": false,
"storage": "",
"storage_version": "",
"storage_supported_drivers": null
}
}
^Cmc-002jsimpso@mc-002:~$ lxc monitor --pretty
DEBUG [2024-12-18T10:22:43+08:00] Event listener server handler started id=ada1f916-366b-4d4d-9424-9f29313c56ec local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2024-12-18T10:22:44+08:00] Handling API request fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf ip="10.0.25.10:38612" method=PUT protocol=cluster url="/1.0/networks/dev-test?project=dev-test"
DEBUG [2024-12-18T10:22:44+08:00] Matched trusted cert fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf subject="CN=root@mc-001,O=LXD"
DEBUG [2024-12-18T10:22:44+08:00] Update clientType=notifier driver=ovn network=dev-test newNetwork="{map[bridge.mtu:1442 ipv4.address:172.16.0.1/28 ipv4.nat:false ipv6.address:none network:UPLINK] changed-via-ui}" project=dev-test
INFO [2024-12-18T10:22:44+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:22:44+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:22:44+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
DEBUG [2024-12-18T10:22:46+08:00] Starting heartbeat round local="10.0.25.11:8443" mode=normal
DEBUG [2024-12-18T10:22:46+08:00] Heartbeat updating local raft members members="[{{1 10.0.25.10:8443 voter} mc-001} {{2 10.0.25.11:8443 voter} mc-002} {{3 10.0.25.12:8443 voter} mc-003}]"
DEBUG [2024-12-18T10:22:46+08:00] Sending heartbeat request address="10.0.25.10:8443"
DEBUG [2024-12-18T10:22:46+08:00] Successful heartbeat remote="10.0.25.10:8443"
INFO [2024-12-18T10:22:47+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: tls/5273a69d19633c2b9aa19daf7e4dc2ee4384c05c7640fa5fe8c7d901e32095b2 (10.0.37.136:48824)
DEBUG [2024-12-18T10:22:50+08:00] Sending heartbeat request address="10.0.25.12:8443"
DEBUG [2024-12-18T10:22:50+08:00] Successful heartbeat remote="10.0.25.12:8443"
DEBUG [2024-12-18T10:22:50+08:00] Rebalancing member roles in heartbeat local="10.0.25.11:8443"
DEBUG [2024-12-18T10:22:50+08:00] Completed heartbeat round duration=4.302598901s local="10.0.25.11:8443"
^Cmc-003jsimpso@mc-003:~$ lxc monitor --pretty
DEBUG [2024-12-18T10:22:43+08:00] Event listener server handler started id=973b0266-9a62-41ba-bf51-35738a0909b7 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2024-12-18T10:22:44+08:00] Handling API request fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf ip="10.0.25.10:39326" method=PUT protocol=cluster url="/1.0/networks/dev-test?project=dev-test"
DEBUG [2024-12-18T10:22:44+08:00] Matched trusted cert fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf subject="CN=root@mc-001,O=LXD"
DEBUG [2024-12-18T10:22:44+08:00] Update clientType=notifier driver=ovn network=dev-test newNetwork="{map[bridge.mtu:1442 ipv4.address:172.16.0.1/28 ipv4.nat:false ipv6.address:none network:UPLINK] changed-via-ui}" project=dev-test
DEBUG [2024-12-18T10:22:48+08:00] Handling API request ip="10.0.25.24:41736" method=GET protocol=tls url=/1.0/metrics username=efa460eca20a32479bbb6b3d415d6e9340196ea717e93213a784686c82430c52
DEBUG [2024-12-18T10:22:48+08:00] Matched trusted cert fingerprint=efa460eca20a32479bbb6b3d415d6e9340196ea717e93213a784686c82430c52 subject="CN=metrics.local"
DEBUG [2024-12-18T10:22:48+08:00] Sending request to LXD etag= method=GET url="https://custom.socket/1.0"
DEBUG [2024-12-18T10:22:48+08:00] Connecting to a VM agent over a VM socket
DEBUG [2024-12-18T10:22:48+08:00] Got response struct from LXD
DEBUG [2024-12-18T10:22:48+08:00] Sending request to LXD etag= method=GET url="https://custom.socket/1.0/metrics"
DEBUG [2024-12-18T10:22:48+08:00]
{
"config": null,
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"backup_compression",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode",
"resources_pci_vpd",
"qemu_raw_conf",
"storage_cephfs_fscache",
"network_load_balancer",
"vsock_api",
"instance_ready_state",
"network_bgp_holdtime",
"storage_volumes_all_projects",
"metrics_memory_oom_total",
"storage_buckets",
"storage_buckets_create_credentials",
"metrics_cpu_effective_total",
"projects_networks_restricted_access",
"storage_buckets_local",
"loki",
"acme",
"internal_metrics",
"cluster_join_token_expiry",
"remote_token_expiry",
"init_preseed",
"storage_volumes_created_at",
"cpu_hotplug",
"projects_networks_zones",
"network_txqueuelen",
"cluster_member_state",
"instances_placement_scriptlet",
"storage_pool_source_wipe",
"zfs_block_mode",
"instance_generation_id",
"disk_io_cache",
"amd_sev",
"storage_pool_loop_resize",
"migration_vm_live",
"ovn_nic_nesting",
"oidc",
"network_ovn_l3only",
"ovn_nic_acceleration_vdpa",
"cluster_healing",
"instances_state_total",
"auth_user",
"security_csm",
"instances_rebuild",
"numa_cpu_placement",
"custom_volume_iso",
"network_allocations",
"storage_api_remote_volume_snapshot_copy",
"zfs_delegate",
"operations_get_query_all_projects",
"metadata_configuration",
"syslog_socket",
"event_lifecycle_name_and_project",
"instances_nic_limits_priority",
"disk_initial_volume_configuration",
"operation_wait",
"cluster_internal_custom_volume_copy",
"disk_io_bus",
"storage_cephfs_create_missing",
"instance_move_config",
"ovn_ssl_config",
"init_preseed_storage_volumes",
"metrics_instances_count",
"server_instance_type_info",
"resources_disk_mounted",
"server_version_lts",
"oidc_groups_claim",
"loki_config_instance",
"storage_volatile_uuid",
"import_instance_devices",
"instances_uefi_vars",
"instances_migration_stateful",
"container_syscall_filtering_allow_deny_syntax",
"access_management",
"vm_disk_io_limits",
"storage_volumes_all",
"instances_files_modify_permissions",
"image_restriction_nesting",
"container_syscall_intercept_finit_module",
"device_usb_serial",
"network_allocate_external_ips",
"explicit_trust_token"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"auth_user_name": "",
"auth_user_method": "",
"environment": {
"addresses": null,
"architectures": null,
"certificate": "",
"certificate_fingerprint": "",
"driver": "",
"driver_version": "",
"instance_types": null,
"firewall": "",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": null,
"kernel_version": "6.8.0-49-generic",
"lxc_features": null,
"os_name": "",
"os_version": "",
"project": "",
"server": "lxd-agent",
"server_clustered": false,
"server_event_mode": "",
"server_name": "u1",
"server_pid": 453,
"server_version": "5.21.2",
"server_lts": false,
"storage": "",
"storage_version": "",
"storage_supported_drivers": null
}
}
DEBUG [2024-12-18T10:22:50+08:00] Matched trusted cert fingerprint=676ee46ad7a0347fc2bc6e65a2aac7f57c487cfccfc199c8f5e67bce92439d51 subject="CN=root@mc-002,O=LXD"
DEBUG [2024-12-18T10:22:50+08:00] Replace current raft nodes raftMembers="[{{3 10.0.25.12:8443 voter} mc-003} {{1 10.0.25.10:8443 voter} mc-001} {{2 10.0.25.11:8443 voter} mc-002}]"
^C
|
Changing network configuration via CLIPre-test
Making the change
ResultChecking the nexthop values after the change shows that all nodes are still in agreement: jsimpso@mc-001:~$ lxc query /internal/testing/bgp | jq '.prefixes.[] | select(.prefix=="172.16.0.0/28") | .nexthop'
"10.0.10.4"
jsimpso@mc-002:~$ lxc query /internal/testing/bgp | jq '.prefixes.[] | select(.prefix=="172.16.0.0/28") | .nexthop'
"10.0.10.4"
jsimpso@mc-003:~$ lxc query /internal/testing/bgp | jq '.prefixes.[] | select(.prefix=="172.16.0.0/28") | .nexthop'
"10.0.10.4"Outputmc-001jsimpso@mc-001:~$ lxc monitor --pretty
DEBUG [2024-12-18T10:40:33+08:00] Event listener server handler started id=7275977c-610c-4112-8427-f8a0e157da1e local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2024-12-18T10:40:34+08:00] Handling API request ip=@ method=PUT protocol=unix url="/1.0/networks/dev-test?project=dev-test" username=jsimpso
DEBUG [2024-12-18T10:40:34+08:00] Update clientType=normal driver=ovn network=dev-test newNetwork="{map[bridge.mtu:1442 ipv4.address:172.16.0.1/28 ipv4.nat:false ipv6.address:none network:UPLINK volatile.network.ipv4.address:10.0.10.4] changed-via-cli}" project=dev-test
DEBUG [2024-12-18T10:40:34+08:00] Connecting to a remote LXD over HTTPS url="https://10.0.25.12:8443"
DEBUG [2024-12-18T10:40:34+08:00] Notify node 10.0.25.12:8443 of state changes
DEBUG [2024-12-18T10:40:34+08:00] Connecting to a remote LXD over HTTPS url="https://10.0.25.11:8443"
DEBUG [2024-12-18T10:40:34+08:00] Notify node 10.0.25.11:8443 of state changes
DEBUG [2024-12-18T10:40:34+08:00] Sending request to LXD etag= method=PUT url="https://10.0.25.12:8443/1.0/networks/dev-test?project=dev-test"
DEBUG [2024-12-18T10:40:34+08:00] Sending request to LXD etag= method=PUT url="https://10.0.25.11:8443/1.0/networks/dev-test?project=dev-test"
INFO [2024-12-18T10:40:34+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:40:35+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:40:35+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: unix/jsimpso (@)
INFO [2024-12-18T10:40:35+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: unix/jsimpso (@)
DEBUG [2024-12-18T10:40:37+08:00] Matched trusted cert fingerprint=676ee46ad7a0347fc2bc6e65a2aac7f57c487cfccfc199c8f5e67bce92439d51 subject="CN=root@mc-002,O=LXD"
DEBUG [2024-12-18T10:40:37+08:00] Replace current raft nodes raftMembers="[{{3 10.0.25.12:8443 voter} mc-003} {{1 10.0.25.10:8443 voter} mc-001} {{2 10.0.25.11:8443 voter} mc-002}]"
^Cmc-002jsimpso@mc-002:~$ lxc monitor --pretty
DEBUG [2024-12-18T10:40:33+08:00] Event listener server handler started id=a3449fbd-29a7-4d96-a0a2-fe10ca05c5ad local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2024-12-18T10:40:34+08:00] Handling API request fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf ip="10.0.25.10:57312" method=PUT protocol=cluster url="/1.0/networks/dev-test?project=dev-test"
DEBUG [2024-12-18T10:40:34+08:00] Matched trusted cert fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf subject="CN=root@mc-001,O=LXD"
DEBUG [2024-12-18T10:40:34+08:00] Update clientType=notifier driver=ovn network=dev-test newNetwork="{map[bridge.mtu:1442 ipv4.address:172.16.0.1/28 ipv4.nat:false ipv6.address:none network:UPLINK volatile.network.ipv4.address:10.0.10.4] changed-via-cli}" project=dev-test
INFO [2024-12-18T10:40:34+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:40:34+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:40:35+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: / ()
INFO [2024-12-18T10:40:35+08:00] Action: network-updated, Source: /1.0/networks/dev-test?project=dev-test, Requestor: unix/jsimpso (@)
DEBUG [2024-12-18T10:40:36+08:00] Heartbeat updating local raft members members="[{{1 10.0.25.10:8443 voter} mc-001} {{2 10.0.25.11:8443 voter} mc-002} {{3 10.0.25.12:8443 voter} mc-003}]"
DEBUG [2024-12-18T10:40:36+08:00] Starting heartbeat round local="10.0.25.11:8443" mode=normal
DEBUG [2024-12-18T10:40:37+08:00] Sending heartbeat request address="10.0.25.10:8443"
DEBUG [2024-12-18T10:40:37+08:00] Successful heartbeat remote="10.0.25.10:8443"
^C
mc-003jsimpso@mc-003:~$ lxc monitor --pretty
DEBUG [2024-12-18T10:40:33+08:00] Event listener server handler started id=a9403c3a-4dcb-42ff-bc0c-4af06b099a33 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2024-12-18T10:40:33+08:00] Handling API request ip="10.0.25.24:34834" method=GET protocol=tls url=/1.0/metrics username=efa460eca20a32479bbb6b3d415d6e9340196ea717e93213a784686c82430c52
DEBUG [2024-12-18T10:40:33+08:00] Matched trusted cert fingerprint=efa460eca20a32479bbb6b3d415d6e9340196ea717e93213a784686c82430c52 subject="CN=metrics.local"
DEBUG [2024-12-18T10:40:33+08:00] Connecting to a VM agent over a VM socket
DEBUG [2024-12-18T10:40:33+08:00] Sending request to LXD etag= method=GET url="https://custom.socket/1.0"
DEBUG [2024-12-18T10:40:33+08:00] Got response struct from LXD
DEBUG [2024-12-18T10:40:33+08:00]
{
"config": null,
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"backup_compression",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode",
"resources_pci_vpd",
"qemu_raw_conf",
"storage_cephfs_fscache",
"network_load_balancer",
"vsock_api",
"instance_ready_state",
"network_bgp_holdtime",
"storage_volumes_all_projects",
"metrics_memory_oom_total",
"storage_buckets",
"storage_buckets_create_credentials",
"metrics_cpu_effective_total",
"projects_networks_restricted_access",
"storage_buckets_local",
"loki",
"acme",
"internal_metrics",
"cluster_join_token_expiry",
"remote_token_expiry",
"init_preseed",
"storage_volumes_created_at",
"cpu_hotplug",
"projects_networks_zones",
"network_txqueuelen",
"cluster_member_state",
"instances_placement_scriptlet",
"storage_pool_source_wipe",
"zfs_block_mode",
"instance_generation_id",
"disk_io_cache",
"amd_sev",
"storage_pool_loop_resize",
"migration_vm_live",
"ovn_nic_nesting",
"oidc",
"network_ovn_l3only",
"ovn_nic_acceleration_vdpa",
"cluster_healing",
"instances_state_total",
"auth_user",
"security_csm",
"instances_rebuild",
"numa_cpu_placement",
"custom_volume_iso",
"network_allocations",
"storage_api_remote_volume_snapshot_copy",
"zfs_delegate",
"operations_get_query_all_projects",
"metadata_configuration",
"syslog_socket",
"event_lifecycle_name_and_project",
"instances_nic_limits_priority",
"disk_initial_volume_configuration",
"operation_wait",
"cluster_internal_custom_volume_copy",
"disk_io_bus",
"storage_cephfs_create_missing",
"instance_move_config",
"ovn_ssl_config",
"init_preseed_storage_volumes",
"metrics_instances_count",
"server_instance_type_info",
"resources_disk_mounted",
"server_version_lts",
"oidc_groups_claim",
"loki_config_instance",
"storage_volatile_uuid",
"import_instance_devices",
"instances_uefi_vars",
"instances_migration_stateful",
"container_syscall_filtering_allow_deny_syntax",
"access_management",
"vm_disk_io_limits",
"storage_volumes_all",
"instances_files_modify_permissions",
"image_restriction_nesting",
"container_syscall_intercept_finit_module",
"device_usb_serial",
"network_allocate_external_ips",
"explicit_trust_token"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"auth_user_name": "",
"auth_user_method": "",
"environment": {
"addresses": null,
"architectures": null,
"certificate": "",
"certificate_fingerprint": "",
"driver": "",
"driver_version": "",
"instance_types": null,
"firewall": "",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": null,
"kernel_version": "6.8.0-49-generic",
"lxc_features": null,
"os_name": "",
"os_version": "",
"project": "",
"server": "lxd-agent",
"server_clustered": false,
"server_event_mode": "",
"server_name": "u1",
"server_pid": 482,
"server_version": "5.21.2",
"server_lts": false,
"storage": "",
"storage_version": "",
"storage_supported_drivers": null
}
}
DEBUG [2024-12-18T10:40:33+08:00] Sending request to LXD etag= method=GET url="https://custom.socket/1.0/metrics"
DEBUG [2024-12-18T10:40:34+08:00] Matched trusted cert fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf subject="CN=root@mc-001,O=LXD"
DEBUG [2024-12-18T10:40:34+08:00] Handling API request fingerprint=485db7985f483d2fe7ecc0acc076f2363e40dce2844da4ece5db02f15a57a6cf ip="10.0.25.10:33296" method=PUT protocol=cluster url="/1.0/networks/dev-test?project=dev-test"
DEBUG [2024-12-18T10:40:35+08:00] Update clientType=notifier driver=ovn network=dev-test newNetwork="{map[bridge.mtu:1442 ipv4.address:172.16.0.1/28 ipv4.nat:false ipv6.address:none network:UPLINK volatile.network.ipv4.address:10.0.10.4] changed-via-cli}" project=dev-test
^C
|
|
So it looks like The newNetwork.map:
- bridge.mtu: 1442
- ipv4.address: 172.16.0.1/28
- ipv4.nat: false
- ipv6.address: none
- network: UPLINK Vs the CLI: newNetwork.map:
- bridge.mtu: 1442
- ipv4.address: 172.16.0.1/28
- ipv4.nat: false
- ipv6.address: none
- network: UPLINK
- volatile.network.ipv4.address: 10.0.10.4 |
|
@edlerd is this something you can look into? |
The UI preserves any fields of a network that it doesn't present to the user in a web form. I don't fully understand the cause of the problem, as the volatile.* keys should be preserved. I'll have a closer look into it, the cluster case might be special in this regard. |
|
The UI explicitly ignores @tomponline Are there circumstances where the volatile keys should be ignored for networks? If not, I will remove this volatile filter from the UI and that should fix the behaviour reported above. |
|
I see we are not doing any filtering for volatile keys on other entities, like instances, profile, storage or alike. |
edlerd
added a commit
to edlerd/lxd-ui
that referenced
this issue
Dec 18, 2024
…rk. see canonical/lxd#14531 Signed-off-by: David Edler <[email protected]>
When submitting a PUT request an entity, the configuration submitted will replace the existing configuration. This is why its important to ensure that all config that is read via GET is submitted back during the PUT request. The other option is to use the PATCH method and only submit the changed keys. |
|
@edlerd do you know if there is anywhere else in the UI where other keys are being stripped on return? |
We do preserve keys on the top level of the object and nested under config.* for instances, profiles, storage pools and volumes, networks and projects. |
The problem with PATCH is, that if a user unsets a field, there is no way to express it in the payload. |
Using empty string for the key is the way to express this: However the drawback is that it also clears any non-config fields in the struct, such as |
|
Although the immediate issue will be fixed by canonical/lxd-ui#1036 (thanks @edlerd !) I will keep this issue open as removing the volatile key should consistently update the nexthop address, so this needs to be looked into why its not happening. |
|
For OVN networks, if there is no volatile IP or |
|
@edlerd once you have a fix for this in the LXD UI, please can you create a release tag and then we can include it in an interim release of LXD 6.2, and then we can also include it in the forthcoming LTS release of LXD 5.21.3 which is going to include the UI version from LXD 6.2. |
edlerd
added a commit
to canonical/lxd-ui
that referenced
this issue
Dec 18, 2024
## Done - Ensure volatile keys are preserved, when saving a network Fixes canonical/lxd#14531 ## QA 1. Run the LXD-UI: - On the demo server via the link posted by @webteam-app below. This is only available for PRs created by collaborators of the repo. Ask @mas-who or @edlerd for access. - With a local copy of this branch, [build and run as described in the docs](../CONTRIBUTING.md#setting-up-for-development). 2. Perform the following QA steps: - edit a network, that has settings under config.volatile.* - modify the network in the UI, ensure the save call contains all config.volatile keys in the payload
Crafted a 0.15 tag and updated it in |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Required information
Issue description
This is a MicroCloud deployed cross 3 physical nodes
I've configured BGP between LXD and the physical router in this lab network
The router is successfully receiving routes for the three networks I've created so far. However, one of those networks is showing different values for the next hop:
Running
lxc query internal/testing/bgpacross the three nodes confirms that only one of them has the nexthop value set to the OVN router, the other two list "0.0.0.0".:mc-001:
{ "peers": [ { "address": "10.0.25.1", "asn": 64513, "count": 1, "holdtime": 0, "password": "" } ], "prefixes": [ { "nexthop": "10.0.10.3", "owner": "network_3", "prefix": "172.16.0.128/28" }, { "nexthop": "10.0.10.4", "owner": "network_4", "prefix": "172.16.0.0/28" }, { "nexthop": "0.0.0.0", "owner": "network_5", "prefix": "172.16.1.0/25" } ], "server": { "address": "10.0.25.10", "asn": 64512, "router_id": "10.0.25.10", "running": true } }mc-002:
{ "peers": [ { "address": "10.0.25.1", "asn": 64513, "count": 1, "holdtime": 0, "password": "" } ], "prefixes": [ { "nexthop": "10.0.10.3", "owner": "network_3", "prefix": "172.16.0.128/28" }, { "nexthop": "10.0.10.4", "owner": "network_4", "prefix": "172.16.0.0/28" }, { "nexthop": "0.0.0.0", "owner": "network_5", "prefix": "172.16.1.0/25" } ], "server": { "address": "10.0.25.11", "asn": 64512, "router_id": "10.0.25.11", "running": true } }mc-003:
{ "peers": [ { "address": "10.0.25.1", "asn": 64513, "count": 1, "holdtime": 0, "password": "" } ], "prefixes": [ { "nexthop": "10.0.10.3", "owner": "network_3", "prefix": "172.16.0.128/28" }, { "nexthop": "10.0.10.4", "owner": "network_4", "prefix": "172.16.0.0/28" }, { "nexthop": "10.0.10.5", "owner": "network_5", "prefix": "172.16.1.0/25" } ], "server": { "address": "10.0.25.12", "asn": 64512, "router_id": "10.0.25.12", "running": true } }Steps to reproduce
I seem to be able to trigger this behaviour by changing the configuration of the network. This issue occurred after I changed the MTU for the network from
1442to1500for some unrelated troubleshooting. Changing the configuration back to its original state doesn't appear to have any positive effect.I've just reproduced this by replicating that same change (editing the network's YAML configuration via the LXD UI) on
network_4from the above output, and now see the same situation there:Router
mc-001:
{ "peers": [ { "address": "10.0.25.1", "asn": 64513, "count": 1, "holdtime": 0, "password": "" } ], "prefixes": [ { "nexthop": "0.0.0.0", "owner": "network_5", "prefix": "172.16.1.0/25" }, { "nexthop": "10.0.10.3", "owner": "network_3", "prefix": "172.16.0.128/28" }, { "nexthop": "0.0.0.0", "owner": "network_4", "prefix": "172.16.0.0/28" } ], "server": { "address": "10.0.25.10", "asn": 64512, "router_id": "10.0.25.10", "running": true } }mc-002:
{ "peers": [ { "address": "10.0.25.1", "asn": 64513, "count": 1, "holdtime": 0, "password": "" } ], "prefixes": [ { "nexthop": "10.0.10.3", "owner": "network_3", "prefix": "172.16.0.128/28" }, { "nexthop": "0.0.0.0", "owner": "network_4", "prefix": "172.16.0.0/28" }, { "nexthop": "0.0.0.0", "owner": "network_5", "prefix": "172.16.1.0/25" } ], "server": { "address": "10.0.25.11", "asn": 64512, "router_id": "10.0.25.11", "running": true } }mc-003:
{ "peers": [ { "address": "10.0.25.1", "asn": 64513, "count": 1, "holdtime": 0, "password": "" } ], "prefixes": [ { "nexthop": "10.0.10.3", "owner": "network_3", "prefix": "172.16.0.128/28" }, { "nexthop": "10.0.10.4", "owner": "network_4", "prefix": "172.16.0.0/28" }, { "nexthop": "10.0.10.5", "owner": "network_5", "prefix": "172.16.1.0/25" } ], "server": { "address": "10.0.25.12", "asn": 64512, "router_id": "10.0.25.12", "running": true } }The text was updated successfully, but these errors were encountered: