Loading
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Assignees
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Assignees
Labels
Incomplete
Waiting on more information from reporter
Comments
Required information
- Distribution: Ubuntu
- Distribution version: 20.04
- The output of «lxc info» or if that fails:
- Kernel version: 5.4.0-109-generic
- LXC version:
- LXD version: 5.1
- Storage backend in use: ZFS
Issue description
A brief description of the problem. Should include what you were
attempting to do, what you did, what happened and what you expected to
see happen.
I developend an application with pylxd before and did not run into any issues with it. Now I am migrating my codebase to go and using the lxd client library.
I ran into in issue with timeouts
Post "https://127.0.0.1:8443/1.0/networks?project=sample-project": net/http: timeout awaiting response headers
In this case, the python code uses a timeout of None which means that it will wait indefinitely until it gets a response back.
https://github.com/lxc/pylxd/blob/a44deea30cb2dce3b1fd34240155c5af6479f1a2/pylxd/client.py#L333
On network creation, the function is blocked until it was created
https://github.com/lxc/pylxd/blob/a44deea30cb2dce3b1fd34240155c5af6479f1a2/pylxd/models/network.py#L112
In the go codebase, there is a timeout of 30 seconds for response headers
| ResponseHeaderTimeout: time.Second * 30, |
That is the problem, my network creation request takes more than 30 seconds to complete.
Possible fixes are to increase the timeout or provide a way for the programmer to increase it manually.
Note: The network gets created in the background even after the timeout, but I want a way to block my application until the request is completed successfully. I do not want to write a loop checking the state of the network until it gets created 
Steps to reproduce
- Step one
Deploy a slow LXD server - Step two
Write a go application to create a new OVN network - Step three
After 30 seconds, the CreateNetwork function thorws a timeout error
Information to attach
- Any relevant kernel output (
dmesg) - Container log (
lxc info NAME --show-log) - Container configuration (
lxc config show NAME --expanded) - Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
- Output of the client with —debug
- Output of the daemon with —debug (alternatively output of
lxc monitorwhile reproducing the issue)
What is the contents and url of your request?
We added timeouts recently (as all network programming should have timeouts) but we may need to increase it to cover all common workload types (although our ovn tests are completing in under that time) so would be good to understand why its taking so long in your case.
How long does it take with your pylxd app?
Copy link
Contributor
Author
import time from pylxd import Client client = Client( verify=False, cert=("lxd.crt", "lxd.key"), endpoint="https://remote-ip:8443", project="sample-project", ) tic = time.perf_counter() client.networks.create( "ovn-network", type="ovn", config={ "ipv4.nat": "true", "ipv6.address": "none", "network": "parent-network", }, ) toc = time.perf_counter() print(f"Request took {toc - tic:0.4f} seconds")
Request took 103.0310 seconds
My LXD server is running dozens of containers/vms/networks, that might be why it takes almost 2 minutes to create that network
Please can you run ‘lxc monitor —type=logging —pretty’ on the server while trying to create the network so we can see which steps take time.
I recently starting getting this timeout error too copying an image from a local to a remote (my ISP upload is around 18-20mbps & the lxd nodes are connected with wireguard) — the image size is 179mb
lxc image copy -v --copy-aliases --mode=push local:ubuntu-minio-22.04 myremote:
the image copies but the alias does not get applied
Please can you run the command with —debug output as I’m surprised that an operation like copying an image isn’t done with an async operation.
Please can you also provide the exact error. We did not limit request timeout. But rather just the initial header response timeout. So I suspect something else aside from bandwidth limitations is at play here (it may be firewall or multi ip issue).
lxc image copy --debug -v --copy-aliases --mode=push local:ubuntu-minio-22.04 iacimg:
DEBUG [2022-05-08T06:46:57+01:00] Connecting to a local LXD over a Unix socket
DEBUG [2022-05-08T06:46:57+01:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
DEBUG [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG [2022-05-08T06:46:57+01:00]
{
"config": {
"core.https_address": "10.201.0.1"
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"candid_authentication",
"backup_compression",
"candid_config",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"candid_config_key",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"rbac",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"10.201.0.1:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "-----BEGIN CERTIFICATE-----nMIICCzCCAZKgAwIBAgIQUI0swBDs6vJwp2PlwqkJazAKBggqhkjOPQQDAzA3MRwwnGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRcwFQYDVQQDDA5yb290QGVuZGVhndm91cjAeFw0yMjA0MjUxNTU0NDlaFw0zMjA0MjIxNTU0NDlaMDcxHDAaBgNVBAoTnE2xpbnV4Y29udGFpbmVycy5vcmcxFzAVBgNVBAMMDnJvb3RAZW5kZWF2b3VyMHYwnEAYHKoZIzj0CAQYFK4EEACIDYgAEgjrXWznR94gEKBONGs4smRBLo7qqyaMwkiTqn/2J7iMKF5nsZgqwNv1UJnw1AlJnLbx+M5bkwA0AtCTyeSbtNRnIgFgs8pjmR6n7qnHuV428RLPl4JMlrdsQ9SIdQx3dSVo2MwYTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lnBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAsBgNVHREEJTAjggllbmRlYXZvndXKHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwQvRCnwVy+aWeg0rUYSOboOVy7rYzQEdDNlfgJWe2HNwBHhywtpab7+ooZgZfU4mXMAjBWn8w9kCHwLh3/bBwO7TNmV7t/k2ZHB+GpBZiRCNpEHUpdfio20Wjm+7954uyd6g6Y=n-----END CERTIFICATE-----n",
"certificate_fingerprint": "0532a6edbf8bdd51b97faeb9b2669191ed7f5aa1ece9234a4d1e4c7abf3b3c34",
"driver": "lxc | qemu",
"driver_version": "4.0.12 | 6.1.1",
"firewall": "nftables",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": {
"idmapped_mounts": "true",
"netnsid_getifaddrs": "true",
"seccomp_listener": "true",
"seccomp_listener_continue": "true",
"shiftfs": "false",
"uevent_injection": "true",
"unpriv_fscaps": "true"
},
"kernel_version": "5.17.5-hardened1-1-hardened",
"lxc_features": {
"cgroup2": "true",
"core_scheduling": "true",
"devpts_fd": "true",
"idmapped_mounts_v2": "true",
"mount_injection_file": "true",
"network_gateway_device_route": "true",
"network_ipvlan": "true",
"network_l2proxy": "true",
"network_phys_macvlan_mtu": "true",
"network_veth_router": "true",
"pidfd": "true",
"seccomp_allow_deny_syntax": "true",
"seccomp_notify": "true",
"seccomp_proxy_send_notify_fd": "true"
},
"os_name": "EndeavourOS",
"os_version": "",
"project": "default",
"server": "lxd",
"server_clustered": false,
"server_event_mode": "full-mesh",
"server_name": "endeavour",
"server_pid": 13740,
"server_version": "5.1",
"storage": "zfs",
"storage_version": "2.1.4-1",
"storage_supported_drivers": [
{
"Name": "zfs",
"Version": "2.1.4-1",
"Remote": false
},
{
"Name": "ceph",
"Version": "15.2.14",
"Remote": true
},
{
"Name": "btrfs",
"Version": "5.4.1",
"Remote": false
},
{
"Name": "cephfs",
"Version": "15.2.14",
"Remote": true
},
{
"Name": "dir",
"Version": "1",
"Remote": false
},
{
"Name": "lvm",
"Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
"Remote": false
}
]
}
}
DEBUG [2022-05-08T06:46:57+01:00] Connecting to a remote LXD over HTTPS
DEBUG [2022-05-08T06:46:57+01:00] Sending request to LXD etag= method=GET url="https://10.203.0.1:8443/1.0"
DEBUG [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG [2022-05-08T06:46:57+01:00]
{
"config": {
"core.https_address": "10.203.0.1"
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"candid_authentication",
"backup_compression",
"candid_config",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"candid_config_key",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"rbac",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"10.203.0.1:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "-----BEGIN CERTIFICATE-----nMIICAjCCAYmgAwIBAgIQN4N01EkRTWt3KmQdy46muDAKBggqhkjOPQQDAzA0MRwwnGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGJhY2t1ncDAeFw0yMjA1MDcyMjI2MzNaFw0zMjA1MDQyMjI2MzNaMDQxHDAaBgNVBAoTE2xpnbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAYmFja3VwMHYwEAYHKoZInzj0CAQYFK4EEACIDYgAERizSl/gpUEExmNc27xXKpHIlqQOOWNh0oNhj+4Bgf0SfnHV6/ggxBIKGxzZvwGp/shDU2PONlwnATOt2OUUGVVVlKcZLleP5DV9Ku82ty1Y8kntbRYE4nGq55NRLBBXMvno2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYInKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZiYWNrdXCHBH8AAAGHnEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwUZ1NZCLyKl6mfWjgnZdRl6Ga7UwXVxzaw+YVmHFvZdDDslJWDODpRfHktYOcwmarWAjAmHKRMdv6Z2Ro3nQgYeHZ1DkVsK7XX9HKHmHuUNmmafeeVmyyuKiztkKSl1h/8ty5U=n-----END CERTIFICATE-----n",
"certificate_fingerprint": "2aca9ad8acf0160e3bc16302ae31515d1aeff4a6d3ad030bff619a49c58562ac",
"driver": "qemu | lxc",
"driver_version": "6.1.1 | 4.0.12",
"firewall": "nftables",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": {
"idmapped_mounts": "true",
"netnsid_getifaddrs": "true",
"seccomp_listener": "true",
"seccomp_listener_continue": "true",
"shiftfs": "false",
"uevent_injection": "true",
"unpriv_fscaps": "true"
},
"kernel_version": "5.15.0-27-generic",
"lxc_features": {
"cgroup2": "true",
"core_scheduling": "true",
"devpts_fd": "true",
"idmapped_mounts_v2": "true",
"mount_injection_file": "true",
"network_gateway_device_route": "true",
"network_ipvlan": "true",
"network_l2proxy": "true",
"network_phys_macvlan_mtu": "true",
"network_veth_router": "true",
"pidfd": "true",
"seccomp_allow_deny_syntax": "true",
"seccomp_notify": "true",
"seccomp_proxy_send_notify_fd": "true"
},
"os_name": "Ubuntu",
"os_version": "22.04",
"project": "default",
"server": "lxd",
"server_clustered": false,
"server_event_mode": "full-mesh",
"server_name": "backup",
"server_pid": 1154,
"server_version": "5.1",
"storage": "zfs",
"storage_version": "2.1.2-1ubuntu3",
"storage_supported_drivers": [
{
"Name": "btrfs",
"Version": "5.4.1",
"Remote": false
},
{
"Name": "cephfs",
"Version": "15.2.14",
"Remote": true
},
{
"Name": "dir",
"Version": "1",
"Remote": false
},
{
"Name": "lvm",
"Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
"Remote": false
},
{
"Name": "zfs",
"Version": "2.1.2-1ubuntu3",
"Remote": false
},
{
"Name": "ceph",
"Version": "15.2.14",
"Remote": true
}
]
}
}
DEBUG [2022-05-08T06:46:57+01:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/images/aliases/ubuntu-minio-22.04"
DEBUG [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG [2022-05-08T06:46:57+01:00]
{
"description": "",
"target": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
"name": "ubuntu-minio-22.04",
"type": "container"
}
DEBUG [2022-05-08T06:46:57+01:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0"
DEBUG [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG [2022-05-08T06:46:57+01:00]
{
"auto_update": false,
"properties": {
"description": "IAC image: ubuntu-minio-22.04"
},
"public": false,
"expires_at": "2022-05-26T08:51:17+01:00",
"profiles": [
"default"
],
"aliases": [
{
"name": "ubuntu-minio-22.04",
"description": ""
}
],
"architecture": "x86_64",
"cached": false,
"filename": "",
"fingerprint": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
"size": 188141771,
"type": "container",
"created_at": "2022-05-07T17:34:21.514701392Z",
"last_used_at": "2022-05-07T17:34:45.49451231Z",
"uploaded_at": "2022-05-07T17:34:21.516895766Z"
}
DEBUG [2022-05-08T06:46:57+01:00] Connected to the websocket: wss://10.203.0.1:8443/1.0/events
DEBUG [2022-05-08T06:46:57+01:00] Sending request to LXD etag= method=POST url="https://10.203.0.1:8443/1.0/images"
DEBUG [2022-05-08T06:46:57+01:00]
{
"auto_update": false,
"properties": {
"description": "IAC image: ubuntu-minio-22.04"
},
"public": false,
"expires_at": "2022-05-26T08:51:17+01:00",
"profiles": null,
"filename": "",
"source": {
"alias": "",
"certificate": "",
"protocol": "",
"server": "",
"image_type": "",
"mode": "push",
"type": "",
"url": "",
"name": "",
"fingerprint": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
"secret": "",
"project": ""
},
"compression_algorithm": "",
"aliases": null
}
DEBUG [2022-05-08T06:46:57+01:00] Got operation from LXD
DEBUG [2022-05-08T06:46:57+01:00]
{
"id": "452b81cf-ec74-4d9c-883e-29e299ac4fac",
"class": "token",
"description": "Image download token",
"created_at": "2022-05-08T06:46:57.896860241+01:00",
"updated_at": "2022-05-08T06:46:57.896860241+01:00",
"status": "Running",
"status_code": 103,
"resources": {
"images": [
"/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0"
]
},
"metadata": {
"aliases": null,
"expires_at": "2022-05-26T08:51:17+01:00",
"properties": {
"description": "IAC image: ubuntu-minio-22.04"
},
"public": false,
"secret": "fbc575da2c7bc2c17eb6fa742796997998146a9500d7db1c3699775bc681cbe8"
},
"may_cancel": true,
"err": "",
"location": "none"
}
DEBUG [2022-05-08T06:46:57+01:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG [2022-05-08T06:46:57+01:00] Sending request to LXD etag= method=POST url="http://unix.socket/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0/export"
DEBUG [2022-05-08T06:46:57+01:00]
{
"target": "https://10.203.0.1:8443",
"secret": "fbc575da2c7bc2c17eb6fa742796997998146a9500d7db1c3699775bc681cbe8",
"certificate": "-----BEGIN CERTIFICATE-----nMIICAjCCAYmgAwIBAgIQN4N01EkRTWt3KmQdy46muDAKBggqhkjOPQQDAzA0MRwwnGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGJhY2t1ncDAeFw0yMjA1MDcyMjI2MzNaFw0zMjA1MDQyMjI2MzNaMDQxHDAaBgNVBAoTE2xpnbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAYmFja3VwMHYwEAYHKoZInzj0CAQYFK4EEACIDYgAERizSl/gpUEExmNc27xXKpHIlqQOOWNh0oNhj+4Bgf0SfnHV6/ggxBIKGxzZvwGp/shDU2PONlwnATOt2OUUGVVVlKcZLleP5DV9Ku82ty1Y8kntbRYE4nGq55NRLBBXMvno2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYInKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZiYWNrdXCHBH8AAAGHnEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwUZ1NZCLyKl6mfWjgnZdRl6Ga7UwXVxzaw+YVmHFvZdDDslJWDODpRfHktYOcwmarWAjAmHKRMdv6Z2Ro3nQgYeHZ1DkVsK7XX9HKHmHuUNmmafeeVmyyuKiztkKSl1h/8ty5U=n-----END CERTIFICATE-----n",
"aliases": [
{
"name": "ubuntu-minio-22.04",
"description": ""
}
],
"project": "default",
"profiles": null
}
DEBUG [2022-05-08T06:46:58+01:00] Got operation from LXD
DEBUG [2022-05-08T06:46:58+01:00]
{
"id": "f3472697-cdb2-4489-9761-3f9ae2ca8d1f",
"class": "task",
"description": "Downloading image",
"created_at": "2022-05-08T06:46:58.185805989+01:00",
"updated_at": "2022-05-08T06:46:58.185805989+01:00",
"status": "Running",
"status_code": 103,
"resources": null,
"metadata": null,
"may_cancel": false,
"err": "",
"location": "none"
}
DEBUG [2022-05-08T06:46:58+01:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/operations/f3472697-cdb2-4489-9761-3f9ae2ca8d1f"
DEBUG [2022-05-08T06:46:58+01:00] Got response struct from LXD
DEBUG [2022-05-08T06:46:58+01:00]
{
"id": "f3472697-cdb2-4489-9761-3f9ae2ca8d1f",
"class": "task",
"description": "Downloading image",
"created_at": "2022-05-08T06:46:58.185805989+01:00",
"updated_at": "2022-05-08T06:46:58.185805989+01:00",
"status": "Running",
"status_code": 103,
"resources": null,
"metadata": null,
"may_cancel": false,
"err": "",
"location": "none"
}
DEBUG [2022-05-08T06:48:19+01:00] Sending request to LXD etag= method=DELETE url="https://10.203.0.1:8443/1.0/operations/452b81cf-ec74-4d9c-883e-29e299ac4fac"
Error: Post "https://10.203.0.1:8443/1.0/images?project=default": net/http: timeout awaiting response headers
Does using relay mode help by the way? Ive tried reproducing using tc to simulate a slower network so the transfer takes over 30s and have been able to reproduce.
yes relay mode completes without errors
Is it possible to get the output of ‘lxc monitor —type=logging —pretty’ on source and target servers when you run the command. I’d like to see where the delay is occurring.
SOURCE MONITOR:
DEBUG [2022-05-09T13:39:21+01:00] Event listener server handler started listener=6facecee-8fc1-42d7-9f39-0f7e4d3c15c9 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-09T13:39:31+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=stuart
DEBUG [2022-05-09T13:39:31+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/images/aliases/archlinux-base-current username=stuart
DEBUG [2022-05-09T13:39:31+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/images/4d142148795f0eb41fd6f638c613e12ca5914c189127f918b763ab381947960b username=stuart
DEBUG [2022-05-09T13:39:32+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=stuart
DEBUG [2022-05-09T13:39:32+01:00] Event listener server handler started listener=9bfd2080-d2a0-4f0b-981c-e4490723c241 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-09T13:39:32+01:00] Handling API request ip=@ method=POST protocol=unix url=/1.0/images/4d142148795f0eb41fd6f638c613e12ca5914c189127f918b763ab381947960b/export username=stuart
DEBUG [2022-05-09T13:39:32+01:00] Connecting to a remote LXD over HTTPS
DEBUG [2022-05-09T13:39:32+01:00] Sending request to LXD etag= method=GET url="https://10.203.0.1:8443/1.0"
DEBUG [2022-05-09T13:39:32+01:00] Got response struct from LXD
DEBUG [2022-05-09T13:39:32+01:00]
{
"config": null,
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"candid_authentication",
"backup_compression",
"candid_config",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"candid_config_key",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"rbac",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "untrusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": null,
"architectures": null,
"certificate": "",
"certificate_fingerprint": "",
"driver": "",
"driver_version": "",
"firewall": "",
"kernel": "",
"kernel_architecture": "",
"kernel_features": null,
"kernel_version": "",
"lxc_features": null,
"os_name": "",
"os_version": "",
"project": "",
"server": "",
"server_clustered": false,
"server_event_mode": "",
"server_name": "",
"server_pid": 0,
"server_version": "",
"storage": "",
"storage_version": "",
"storage_supported_drivers": null
}
}
DEBUG [2022-05-09T13:39:32+01:00] New task Operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e
DEBUG [2022-05-09T13:39:32+01:00] Started task operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e
DEBUG [2022-05-09T13:39:32+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/2313a3e6-77a1-4270-95c2-4483f85caa4e username=stuart
DEBUG [2022-05-09T13:43:12+01:00] Failure for task operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e: Post "https://10.203.0.1:8443/1.0/images?project=default": net/http: timeout awaiting response headers
DEBUG [2022-05-09T13:43:13+01:00] Event listener server handler stopped listener=9bfd2080-d2a0-4f0b-981c-e4490723c241 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DESTINATION MONITOR:
DEBUG [2022-05-09T13:39:24+01:00] Event listener server handler started listener=06f44bbc-1d78-46f0-af18-20369e26bf10 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-09T13:39:31+01:00] Matched trusted cert fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG [2022-05-09T13:39:31+01:00] Matched trusted cert fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG [2022-05-09T13:39:31+01:00] Handling API request ip="10.0.0.1:37056" method=GET protocol=tls url=/1.0 username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG [2022-05-09T13:39:31+01:00] Matched trusted cert fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG [2022-05-09T13:39:31+01:00] Handling API request ip="10.0.0.1:37058" method=GET protocol=tls url=/1.0/events username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG [2022-05-09T13:39:31+01:00] Event listener server handler started listener=4b20a751-b8fa-4b9e-b721-295a62d577d8 local="10.203.0.1:8443" remote="10.0.0.1:37058"
DEBUG [2022-05-09T13:39:32+01:00] Matched trusted cert fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG [2022-05-09T13:39:32+01:00] Handling API request ip="10.0.0.1:37060" method=POST protocol=tls url=/1.0/images username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG [2022-05-09T13:39:32+01:00] Matched trusted cert fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG [2022-05-09T13:39:32+01:00] New token Operation: 61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG [2022-05-09T13:39:32+01:00] Started token operation: 61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG [2022-05-09T13:39:32+01:00] Allowing untrusted GET ip="10.0.0.1:37062" url=/1.0
DEBUG [2022-05-09T13:39:32+01:00] Allowing untrusted POST ip="10.0.0.1:37064" url="/1.0/images?project=default"
DEBUG [2022-05-09T13:39:32+01:00] Cancelling operation class=token operation=61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG [2022-05-09T13:39:32+01:00] Cancelled operation class=token operation=61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG [2022-05-09T13:43:13+01:00] Matched trusted cert fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG [2022-05-09T13:43:13+01:00] Handling API request ip="10.0.0.1:37066" method=DELETE protocol=tls url=/1.0/operations/61a3897d-c899-476c-85db-3ceee2c14e15 username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG [2022-05-09T13:43:13+01:00] Database error err="Operation not found"
DEBUG [2022-05-09T13:43:13+01:00] Event listener server handler stopped listener=4b20a751-b8fa-4b9e-b721-295a62d577d8 local="10.203.0.1:8443" remote="10.0.0.1:37058"
10.0.0.1 is the source wireguard interface (with routing rules in /etc/systemd/network/99-wg0.network)
10.203.0.0/24 is the destination LXD subnet
10.201.0.0/24 is the source LXD subnet
Only trusted ip’s are allowed to connect to the remote wireguard (allowed at the ingress hook level in nftables so there shouldn’t be any mangling by the firewall)
Hi, I’m seeing some issues with similar errors that I’ve never had before when creating lxd container on Fedora 34. Started in the last few days: «timeout awaiting response headers»
Creating oel84d10
Error: Failed instance creation: Post «http://unix.socket/1.0/instances»: net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate «local:oel84d10»: Instance not found
Creating oel84d10
Error: Failed instance creation: Post «http://unix.socket/1.0/instances»: net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate «local:oel84d10»: Instance not found
Creating oel84d10
Error: Failed instance creation: Post «http://unix.socket/1.0/instances»: net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate «local:oel84d10»: Instance not found
Creating oel84d10
Error: Failed instance creation: Post «http://unix.socket/1.0/instances»: net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate «local:oel84d10»: Instance not found
Creating oel84d10
@gstanden can you rerun command with —debug and paste output please.
[orabuntu@f34sv1 ~]$ lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix --debug
DEBUG [2022-05-09T20:03:51-05:00] Connecting to a local LXD over a Unix socket
DEBUG [2022-05-09T20:03:51-05:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
DEBUG [2022-05-09T20:03:51-05:00] Got response struct from LXD
DEBUG [2022-05-09T20:03:51-05:00]
{
"config": {
"cluster.https_address": "10.209.53.1:8443",
"core.https_address": "10.209.53.1:8443",
"core.trust_password": true
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"candid_authentication",
"backup_compression",
"candid_config",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"candid_config_key",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"rbac",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"10.209.53.1:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "-----BEGIN CERTIFICATE-----nMIICAzCCAYmgAwIBAgIQW1WMY5wWjGGh84OVrZUNxTAKBggqhkjOPQQDAzA0MRwwnGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2nMTAeFw0yMjA1MDkyMjI0NDhaFw0zMjA1MDYyMjI0NDhaMDQxHDAaBgNVBAoTE2xpnbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZjM0c3YxMHYwEAYHKoZInzj0CAQYFK4EEACIDYgAE4p63DAbe7w7iED5wf+Ra54b9WEStQbvB/bwnSq9tXOiznqIVoqWxxxAho8HHASCVAVMdkY0vVGtffqiFAe7de+zH+FFGKZysqHNXoigDOtTVOnHY3E5IMAMCfG4YNesffao2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYInKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZmMzRzdjGHBH8AAAGHnEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAwZQIwErX8iCV4YL6/gLyAnSCDaVqTkge0PPsvlHVCgrzes569GZlPDndA9W9X3yDItHcAiAjEAkbMZOaB61PfRnwAbUZ7YbNbVw8FSFGXMfVNdowhwyTod+oz+CstDbGd5QHnDHgHW/n-----END CERTIFICATE-----n",
"certificate_fingerprint": "4797e3981caf2829d8c7be9fc1768f5ebc6d7bfc49e6bacd3fe20fcc678c07f3",
"driver": "lxc",
"driver_version": "4.0.12",
"firewall": "nftables",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": {
"idmapped_mounts": "true",
"netnsid_getifaddrs": "true",
"seccomp_listener": "true",
"seccomp_listener_continue": "true",
"shiftfs": "false",
"uevent_injection": "true",
"unpriv_fscaps": "true"
},
"kernel_version": "5.17.5-100.fc34.x86_64",
"lxc_features": {
"cgroup2": "true",
"core_scheduling": "true",
"devpts_fd": "true",
"idmapped_mounts_v2": "true",
"mount_injection_file": "true",
"network_gateway_device_route": "true",
"network_ipvlan": "true",
"network_l2proxy": "true",
"network_phys_macvlan_mtu": "true",
"network_veth_router": "true",
"pidfd": "true",
"seccomp_allow_deny_syntax": "true",
"seccomp_notify": "true",
"seccomp_proxy_send_notify_fd": "true"
},
"os_name": "Fedora",
"os_version": "34",
"project": "default",
"server": "lxd",
"server_clustered": true,
"server_event_mode": "full-mesh",
"server_name": "f34sv1",
"server_pid": 160714,
"server_version": "5.1",
"storage": "zfs | dir",
"storage_version": "2.1.4-1 | 1",
"storage_supported_drivers": [
{
"Name": "zfs",
"Version": "2.1.4-1",
"Remote": false
},
{
"Name": "ceph",
"Version": "15.2.14",
"Remote": true
},
{
"Name": "btrfs",
"Version": "5.4.1",
"Remote": false
},
{
"Name": "cephfs",
"Version": "15.2.14",
"Remote": true
},
{
"Name": "dir",
"Version": "1",
"Remote": false
},
{
"Name": "lvm",
"Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
"Remote": false
}
]
}
}
Creating oel
DEBUG [2022-05-09T20:03:51-05:00] Connecting to a remote simplestreams server URL="https://images.linuxcontainers.org"
DEBUG [2022-05-09T20:03:51-05:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG [2022-05-09T20:03:51-05:00] Sending request to LXD etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG [2022-05-09T20:03:51-05:00]
{
"architecture": "",
"config": {},
"devices": {},
"ephemeral": false,
"profiles": [
"olxc_sx1a"
],
"stateful": false,
"description": "",
"name": "oel",
"source": {
"type": "image",
"certificate": "",
"alias": "oracle//amd64",
"server": "https://images.linuxcontainers.org",
"protocol": "simplestreams",
"mode": "pull",
"allow_inconsistent": false
},
"instance_type": "",
"type": "container"
}
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
[orabuntu@f34sv1 ~]$
Thanks @gstanden can you also provide output of lxc monitor --type=logging --pretty on the server when you’re running that command. I’m trying to see where the hold up in response headers are. Thanks
@gstanden is it possible to get access to this server?
@itoffshore is it possible to get access to this server for testing?
@tomponline here you go … also, it worked fine this time! of course — it always does when it «knows» we are watching 
… did you make any changes … if so they seem to be working. As before, this is the same Fedora 34 server (VM).
[orabuntu@f34sv1 ~]$ lxc monitor --type=logging --pretty
DEBUG [2022-05-10T08:41:23-05:00] Event listener server handler started listener=28f116df-c67a-48a6-bd00-b940aeb74256 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:41:30-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:41:30-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:41:30-05:00] Completed heartbeat round duration=28.118492ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:41:33-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG [2022-05-10T08:41:33-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG [2022-05-10T08:41:33-05:00] Event listener server handler started listener=ed32291c-37e0-45de-bac2-83eebdccb9da local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:41:33-05:00] Handling API request ip=@ method=POST protocol=unix url=/1.0/instances username=orabuntu
DEBUG [2022-05-10T08:41:33-05:00] Responding to instance create
DEBUG [2022-05-10T08:41:33-05:00] Connecting to a remote simplestreams server URL="https://images.linuxcontainers.org"
DEBUG [2022-05-10T08:41:35-05:00] New task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:35-05:00] Started task operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:35-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b username=orabuntu
DEBUG [2022-05-10T08:41:35-05:00] Connecting to a remote simplestreams server URL="https://images.linuxcontainers.org"
DEBUG [2022-05-10T08:41:35-05:00] Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG [2022-05-10T08:41:35-05:00] Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG [2022-05-10T08:41:35-05:00] Database error err="Image not found"
DEBUG [2022-05-10T08:41:35-05:00] Database error err="Image not found"
INFO [2022-05-10T08:41:35-05:00] Downloading image alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=baa997f7-c06a-40e6-a27f-01396220ba6b server="https://images.linuxcontainers.org" trigger=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:35-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:36-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:36-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:37-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:37-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:39-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:39-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:40-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:40-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:41:40-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:41:40-05:00] Completed heartbeat round duration=5.473403ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:41:40-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:43-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:43-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:44-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:44-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:45-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:45-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:46-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:46-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:47-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:47-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:48-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:48-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:49-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:49-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:50-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:50-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:41:50-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:41:50-05:00] Completed heartbeat round duration=21.990183ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:41:50-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:53-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:53-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:54-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:54-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:56-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:56-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:58-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:59-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:41:59-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:00-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:42:00-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:42:00-05:00] Completed heartbeat round duration=22.543699ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:01-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:01-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:03-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:03-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:04-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:04-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:06-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:06-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:07-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:07-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:08-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:08-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:09-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:09-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:10-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:42:10-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:42:10-05:00] Completed heartbeat round duration=23.069247ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:11-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:11-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:12-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:12-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:15-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:16-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:16-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:18-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:18-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:20-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:42:20-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:42:20-05:00] Completed heartbeat round duration=10.655645ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:42:20-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:20-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:21-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:21-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
INFO [2022-05-10T08:42:22-05:00] Image downloaded alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=baa997f7-c06a-40e6-a27f-01396220ba6b server="https://images.linuxcontainers.org" trigger=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:22-05:00] Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG [2022-05-10T08:42:22-05:00] Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG [2022-05-10T08:42:22-05:00] Instance operation lock created action=create instance=oel84d10 project=default reusable=false
INFO [2022-05-10T08:42:22-05:00] Creating container ephemeral=false instance=oel84d10 instanceType=container project=default
DEBUG [2022-05-10T08:42:22-05:00] Database error err="Network not found"
DEBUG [2022-05-10T08:42:22-05:00] Database error err="Network not found"
DEBUG [2022-05-10T08:42:22-05:00] Adding device device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG [2022-05-10T08:42:22-05:00] Adding device device=root instance=oel84d10 instanceType=container project=default type=disk
INFO [2022-05-10T08:42:22-05:00] Created container ephemeral=false instance=oel84d10 instanceType=container project=default
DEBUG [2022-05-10T08:42:22-05:00] CreateInstanceFromImage started instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:22-05:00] EnsureImage started fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG [2022-05-10T08:42:22-05:00] Database error err="Storage pool volume not found"
INFO [2022-05-10T08:42:22-05:00] Image unpack started imageFile=/var/snap/lxd/common/lxd/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG [2022-05-10T08:42:22-05:00] Mounted ZFS dataset dev=olxd-0933/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local
DEBUG [2022-05-10T08:42:22-05:00] Running filler function dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local
DEBUG [2022-05-10T08:42:22-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
INFO [2022-05-10T08:42:25-05:00] Image unpack stopped imageFile=/var/snap/lxd/common/lxd/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG [2022-05-10T08:42:30-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:42:30-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:42:30-05:00] Completed heartbeat round duration=16.878987ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:42:31-05:00] Unmounted ZFS dataset dev=olxd-0933/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG [2022-05-10T08:42:31-05:00] EnsureImage finished fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG [2022-05-10T08:42:31-05:00] Checking volume size instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:31-05:00] Set new volume size instance=oel84d10 project=default size=
DEBUG [2022-05-10T08:42:31-05:00] Mounted ZFS dataset dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG [2022-05-10T08:42:31-05:00] Unmounted ZFS dataset dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG [2022-05-10T08:42:31-05:00] UpdateInstanceBackupFile started instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:31-05:00] CreateInstanceFromImage finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:31-05:00] Mounted ZFS dataset dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG [2022-05-10T08:42:32-05:00] Instance operation lock finished action=create err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG [2022-05-10T08:42:32-05:00] Unmounted ZFS dataset dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG [2022-05-10T08:42:32-05:00] UpdateInstanceBackupFile finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:32-05:00] Success for task operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG [2022-05-10T08:42:32-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG [2022-05-10T08:42:32-05:00] Handling API request ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG [2022-05-10T08:42:32-05:00] New task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG [2022-05-10T08:42:32-05:00] Instance operation lock created action=start instance=oel84d10 project=default reusable=false
DEBUG [2022-05-10T08:42:32-05:00] Start started instance=oel84d10 instanceType=container project=default stateful=false
DEBUG [2022-05-10T08:42:32-05:00] Started task operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG [2022-05-10T08:42:32-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/3b33fee1-63ac-4fa8-893f-d2ce36852ae6 username=orabuntu
DEBUG [2022-05-10T08:42:32-05:00] MountInstance started instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:32-05:00] MountInstance finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:32-05:00] Mounted ZFS dataset dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG [2022-05-10T08:42:32-05:00] Updated metadata for task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG [2022-05-10T08:42:32-05:00] Container idmap changed, remapping instance=oel84d10 instanceType=container project=default
DEBUG [2022-05-10T08:42:33-05:00] Updated metadata for task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG [2022-05-10T08:42:33-05:00] Database error err="Network not found"
DEBUG [2022-05-10T08:42:33-05:00] Starting device device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG [2022-05-10T08:42:33-05:00] Starting device device=root instance=oel84d10 instanceType=container project=default type=disk
DEBUG [2022-05-10T08:42:33-05:00] UpdateInstanceBackupFile started instance=oel84d10 project=default
INFO [2022-05-10T08:42:33-05:00] Starting container action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC"
DEBUG [2022-05-10T08:42:33-05:00] UpdateInstanceBackupFile finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:33-05:00] Skipping unmount as in use driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG [2022-05-10T08:42:33-05:00] Handling API request ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstart?project=default" username=root
DEBUG [2022-05-10T08:42:33-05:00] Scheduler: container oel84d10 started: re-balancing
DEBUG [2022-05-10T08:42:33-05:00] Success for task operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
INFO [2022-05-10T08:42:33-05:00] Started container action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC"
DEBUG [2022-05-10T08:42:33-05:00] Start finished instance=oel84d10 instanceType=container project=default stateful=false
DEBUG [2022-05-10T08:42:33-05:00] Instance operation lock finished action=start err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG [2022-05-10T08:42:33-05:00] Event listener server handler stopped listener=ed32291c-37e0-45de-bac2-83eebdccb9da local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:40-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:42:40-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:42:40-05:00] Completed heartbeat round duration=46.256365ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG [2022-05-10T08:42:48-05:00] Event listener server handler started listener=f94d1b77-620e-4b9e-bd7a-309f7cafbfcc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=POST protocol=unix url=/1.0/instances/oel84d10/exec username=orabuntu
DEBUG [2022-05-10T08:42:48-05:00] New websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Waiting for exec websockets to connect
DEBUG [2022-05-10T08:42:48-05:00] Started websocket operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=46adcf39146218420742abbaeec8c24cd50f3ec5aa1884dc4094620474ea653b" username=orabuntu
DEBUG [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=ca9356d7c0ea567b1a811df478974372876d21629215f0491112e824e84419f0" username=orabuntu
DEBUG [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=89b3ab2d10db416808cf3d7f2f952be88d061b1a546ff278e86a6eaf7c2e167e" username=orabuntu
DEBUG [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=3925d5e121c6dbd82d84cc4f6d2ef318bc7f16936b84a39d17374ab86942db07" username=orabuntu
DEBUG [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:48-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Retrieved PID of executing child process attachedPid=110357 instance=oel84d10 instanceType=container project=default
DEBUG [2022-05-10T08:42:49-05:00] Exec mirror websocket started PID=110357 instance=oel84d10 interactive=false number=1 project=default
DEBUG [2022-05-10T08:42:49-05:00] Exec control handler started PID=110357 instance=oel84d10 interactive=false project=default
DEBUG [2022-05-10T08:42:49-05:00] Exec mirror websocket started PID=110357 instance=oel84d10 interactive=false number=0 project=default
DEBUG [2022-05-10T08:42:49-05:00] WebsocketRecvStream got message barrier
DEBUG [2022-05-10T08:42:49-05:00] Exec mirror websocket finished PID=110357 instance=oel84d10 interactive=false number=0 project=default
DEBUG [2022-05-10T08:42:49-05:00] Instance process started PID=110357 instance=oel84d10 interactive=false project=default
DEBUG [2022-05-10T08:42:49-05:00] Exec mirror websocket started PID=110357 instance=oel84d10 interactive=false number=2 project=default
DEBUG [2022-05-10T08:42:49-05:00] Exec control handler finished PID=110357 instance=oel84d10 interactive=false project=default
DEBUG [2022-05-10T08:42:49-05:00] Success for websocket operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:49-05:00] Instance process stopped PID=110357 exitStatus=0 instance=oel84d10 interactive=false project=default
DEBUG [2022-05-10T08:42:49-05:00] Exec mirror websocket finished PID=110357 instance=oel84d10 interactive=false number=1 project=default
DEBUG [2022-05-10T08:42:49-05:00] Exec mirror websocket finished PID=110357 instance=oel84d10 interactive=false number=2 project=default
DEBUG [2022-05-10T08:42:49-05:00] Updated metadata for websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG [2022-05-10T08:42:49-05:00] Event listener server handler stopped listener=f94d1b77-620e-4b9e-bd7a-309f7cafbfcc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Event listener server handler started listener=40b2f8fd-916b-41f7-985a-31f7e94f2cec local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] New task Operation: f30078f7-3d05-4f79-9312-5e6fc9889839
DEBUG [2022-05-10T08:42:49-05:00] Instance operation lock created action=update instance=oel84d10 project=default reusable=false
DEBUG [2022-05-10T08:42:49-05:00] Started task operation: f30078f7-3d05-4f79-9312-5e6fc9889839
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/f30078f7-3d05-4f79-9312-5e6fc9889839 username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Database error err="Network not found"
DEBUG [2022-05-10T08:42:49-05:00] UpdateInstanceBackupFile started instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:49-05:00] UpdateInstanceBackupFile finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:49-05:00] Skipping unmount as in use driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG [2022-05-10T08:42:49-05:00] Success for task operation: f30078f7-3d05-4f79-9312-5e6fc9889839
DEBUG [2022-05-10T08:42:49-05:00] Instance operation lock finished action=update err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG [2022-05-10T08:42:49-05:00] Event listener server handler stopped listener=40b2f8fd-916b-41f7-985a-31f7e94f2cec local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Event listener server handler started listener=d170a28f-c83a-4d6d-a7b7-adbf52695cdc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Started task operation: c7e0508c-3d04-4351-8b79-78b744f9fad7
DEBUG [2022-05-10T08:42:49-05:00] Instance operation lock created action=stop instance=oel84d10 project=default reusable=true
DEBUG [2022-05-10T08:42:49-05:00] Shutdown started instance=oel84d10 instanceType=container project=default timeout=-1s
DEBUG [2022-05-10T08:42:49-05:00] New task Operation: c7e0508c-3d04-4351-8b79-78b744f9fad7
DEBUG [2022-05-10T08:42:49-05:00] Shutdown request sent to instance instance=oel84d10 instanceType=container project=default
INFO [2022-05-10T08:42:49-05:00] Shutting down container action=shutdown created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default timeout=-1s used="2022-05-10 13:42:33.633858363 +0000 UTC"
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/c7e0508c-3d04-4351-8b79-78b744f9fad7 username=orabuntu
DEBUG [2022-05-10T08:42:49-05:00] Handling API request ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstopns?netns=%2Fproc%2F109702%2Ffd%2F4&project=default&target=stop" username=root
DEBUG [2022-05-10T08:42:49-05:00] Database error err="Network not found"
DEBUG [2022-05-10T08:42:49-05:00] Stopping device device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG [2022-05-10T08:42:50-05:00] Handling API request ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstop?project=default&target=stop" username=root
DEBUG [2022-05-10T08:42:50-05:00] Container stopped, cleaning up instance=oel84d10 instanceType=container project=default
DEBUG [2022-05-10T08:42:50-05:00] Stopping device device=root instance=oel84d10 instanceType=container project=default type=disk
DEBUG [2022-05-10T08:42:50-05:00] UnmountInstance started instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:50-05:00] UnmountInstance finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:50-05:00] Unmounted ZFS dataset dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG [2022-05-10T08:42:50-05:00] Success for task operation: c7e0508c-3d04-4351-8b79-78b744f9fad7
DEBUG [2022-05-10T08:42:50-05:00] Scheduler: container oel84d10 stopped: re-balancing
DEBUG [2022-05-10T08:42:50-05:00] Instance operation lock finished action=stop err="<nil>" instance=oel84d10 project=default reusable=true
DEBUG [2022-05-10T08:42:50-05:00] Shutdown finished instance=oel84d10 instanceType=container project=default timeout=-1s
DEBUG [2022-05-10T08:42:50-05:00] Event listener server handler stopped listener=d170a28f-c83a-4d6d-a7b7-adbf52695cdc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:50-05:00] Heartbeat updating local raft members members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG [2022-05-10T08:42:50-05:00] Starting heartbeat round local="10.209.53.1:8443" mode=normal
DEBUG [2022-05-10T08:42:50-05:00] Completed heartbeat round duration=8.130839ms local="10.209.53.1:8443"
DEBUG [2022-05-10T08:42:55-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG [2022-05-10T08:42:55-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG [2022-05-10T08:42:55-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG [2022-05-10T08:42:55-05:00] Event listener server handler started listener=f090c5fd-f5fa-4922-b45b-7c7317a5823d local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2022-05-10T08:42:55-05:00] Handling API request ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG [2022-05-10T08:42:55-05:00] New task Operation: 454defe6-7771-4d4b-8941-2bbb260ea16c
DEBUG [2022-05-10T08:42:55-05:00] Started task operation: 454defe6-7771-4d4b-8941-2bbb260ea16c
DEBUG [2022-05-10T08:42:55-05:00] Start started instance=oel84d10 instanceType=container project=default stateful=false
DEBUG [2022-05-10T08:42:55-05:00] Instance operation lock created action=start instance=oel84d10 project=default reusable=false
DEBUG [2022-05-10T08:42:55-05:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/454defe6-7771-4d4b-8941-2bbb260ea16c username=orabuntu
DEBUG [2022-05-10T08:42:55-05:00] MountInstance started instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:55-05:00] MountInstance finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:55-05:00] Mounted ZFS dataset dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG [2022-05-10T08:42:55-05:00] Updated metadata for task Operation: 454defe6-7771-4d4b-8941-2bbb260ea16c
DEBUG [2022-05-10T08:42:55-05:00] Container idmap changed, remapping instance=oel84d10 instanceType=container project=default
DEBUG [2022-05-10T08:42:56-05:00] Updated metadata for task Operation: 454defe6-7771-4d4b-8941-2bbb260ea16c
DEBUG [2022-05-10T08:42:56-05:00] Database error err="Network not found"
DEBUG [2022-05-10T08:42:56-05:00] Starting device device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG [2022-05-10T08:42:56-05:00] Starting device device=root instance=oel84d10 instanceType=container project=default type=disk
DEBUG [2022-05-10T08:42:56-05:00] UpdateInstanceBackupFile started instance=oel84d10 project=default
INFO [2022-05-10T08:42:56-05:00] Starting container action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="2022-05-10 13:42:33.633858363 +0000 UTC"
DEBUG [2022-05-10T08:42:56-05:00] UpdateInstanceBackupFile finished instance=oel84d10 project=default
DEBUG [2022-05-10T08:42:56-05:00] Skipping unmount as in use driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG [2022-05-10T08:42:56-05:00] Handling API request ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstart?project=default" username=root
DEBUG [2022-05-10T08:42:56-05:00] Scheduler: container oel84d10 started: re-balancing
DEBUG [2022-05-10T08:42:56-05:00] Success for task operation: 454defe6-7771-4d4b-8941-2bbb260ea16c
DEBUG [2022-05-10T08:42:56-05:00] Start finished instance=oel84d10 instanceType=container project=default stateful=false
DEBUG [2022-05-10T08:42:56-05:00] Instance operation lock finished action=start err="<nil>" instance=oel84d10 project=default reusable=false
INFO [2022-05-10T08:42:56-05:00] Started container action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="2022-05-10 13:42:33.633858363 +0000 UTC"
DEBUG [2022-05-10T08:42:56-05:00] Event listener server handler stopped listener=f090c5fd-f5fa-4922-b45b-7c7317a5823d local=/var/snap/lxd/common/lxd/unix.socket remote=@
^C
[orabuntu@f34sv1 ~]$
No changes made
Perhaps worth deleting the downloaded image and retrying?
I have to set aside my non-paying opensource work on Orabuntu-LXC atm, and head off to my paying job shortly, but I will definitely do a retry later tonight or tomorrow morning and try to capture a timed-out session and post here. EDIT: In fact what I will do is add some code to Orabuntu-LXC that automatically starts up a logging service that runs «lxc monitor —type=logging —pretty» when lxc launch runs so that this gets captured on every run of the software. It will be a nice logging addition to the software anyway.
ok finally captured the below two log files using this service:
[Unit]
Description=lxc-monitor Service
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=orabuntu
RemainAfterExit=yes
ExecStart=/var/lib/snapd/snap/bin/lxc monitor --type=logging --pretty
ExecStop=ps -aux | grep pretty | grep -v grep | grep monitor | awk '{print $2}' | xargs kill -9
StandardOutput=append:/home/orabuntu/lxc-monitor.log
StandardError=append:/home/orabuntu/lxc-monitor.err
[Install]
WantedBy=multi-user.target
The Orabuntu-LXC software starts this service just before it tries to create the first Oracle Linux LXD container. The logs of two runs that gave the error «Error: Failed instance creation: Post «http://unix.socket/1.0/instances»: net/http: timeout awaiting response headers» are below:
lxc-monitor.seed.log
lxc-monitor.run1.log
Now here is another «feature» of this issue; if I then immediately try to create an LXD container of Oracle Linux 8 by just incrementing my «$SeedPostfix» variable (increment from «d10» to «d11») the LXD container creates perfectly fine:
lxc-monitor.run3.log
So something is happening only with that first container attempt to create just after LXD has been setup.
The cluster looks ok:
[orabuntu@f34sv1 system]$ lxc cluster list
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| f34sv1 | https://10.209.53.1:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
[orabuntu@f34sv1 system]$ lxc storage list
+------------+--------+------------------+---------+---------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
+------------+--------+------------------+---------+---------+
| containerd | dir | | 3 | CREATED |
+------------+--------+------------------+---------+---------+
| docker | dir | | 3 | CREATED |
+------------+--------+------------------+---------+---------+
| kubelet | dir | | 3 | CREATED |
+------------+--------+------------------+---------+---------+
| local | zfs | f34sv1-olxd-f2fc | 5 | CREATED |
+------------+--------+------------------+---------+---------+
[orabuntu@f34sv1 system]$ sudo zpool list
[sudo] password for orabuntu:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
olxd-f2fc 960M 578M 382M - - 13% 60% 1.00x ONLINE -
[orabuntu@f34sv1 system]$
Are you sure «seed» was the same error the logs show something different:
time="2022-05-11T01:02:02-05:00" level=debug msg="Database error" err="FOREIGN KEY constraint failed"
time="2022-05-11T01:02:02-05:00" level=debug msg="Database error" err="Storage pool volume not found"
time="2022-05-11T01:02:02-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T01:02:02-05:00" level=debug msg="CreateInstanceFromImage finished" instance=oel84d10 project=default
time="2022-05-11T01:02:02-05:00" level=debug msg="Removing device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T01:02:02-05:00" level=debug msg="Removing device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T01:02:02-05:00" level=debug msg="Failure for task operation: 9221fa4c-af35-4801-91b6-3cc32f2097c3: Failed creating instance from image: Failed to set apply_template volatile key: Failed to set volatile config: FOREIGN KEY constraint failed"
time="2022-05-11T01:02:02-05:00" level=debug msg="Database error" err="Query deleted 0 rows instead of 1"
When my variables are set to create «oel84d10» (the first LXD container created after LXD has been installed and configured) I get the error.
If I increment to create «oel84d11» everything creates fine. It’s always just that first LXD container creation attempt after LXD has been installed and configured.
Here’s the log for oel84d11
lxc-monitor.run3.log
Looks like you’re trying to create the same instance concurrently:
time="2022-05-11T01:01:48-05:00" level=info msg="Image downloaded" alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=9221fa4c-af35-4801-91b6-3cc32f2097c3 server="https://images.linuxcontainers.org" trigger=/1.0/operations/9221fa4c-af35-4801-91b6-3cc32f2097c3
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Image already exists in the DB" fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
time="2022-05-11T01:01:48-05:00" level=debug msg="Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Image already exists in the DB" fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066""
time="2022-05-11T01:01:48-05:00" level=debug msg="Instance operation lock created" action=create instance=oel84d10 project=default reusable=false
time="2022-05-11T01:01:48-05:00" level=debug msg="Database error" err="Add instance info to the database: This "instances" entry already exists"
time="2022-05-11T01:01:48-05:00" level=debug msg="Failure for task operation: 5ded6b69-dcd6-4fa4-9b5b-46eacff7b6fd: Failed creating instance record: Add instance info to the database: This "instances" entry already exists"
time="2022-05-11T01:01:48-05:00" level=debug msg="Failure for task operation: f00877e3-dcf3-42b3-b1c7-6a45f5d287b9: Failed creating instance record: Add instance info to the database: This "instances" entry already exists"
time="2022-05-11T01:01:48-05:00" level=debug msg="Database error" err="Add instance info to the database: This "instances" entry already exists"
time="2022-05-11T01:01:48-05:00" level=info msg="Creating container" ephemeral=false instance=oel84d10 instanceType=container project=default
I never got these problems before in the lxd 4.x series. This really seems to be a new issue that was introduced in lxd 5.x I ran the following just now with no issue:
[orabuntu@f34sv1 ~]$ lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix
Creating oel84d10
Error: Failed instance creation: Failed creating instance from image: Failed to run: zfs clone olxd-f2fc/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066@readonly olxd-f2fc/containers/oel84d10: cannot create 'olxd-f2fc/containers/oel84d10': dataset already exists
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
[sudo] password for orabuntu:
NAME USED AVAIL REFER MOUNTPOINT
olxd-f2fc 582M 250M 96K legacy
olxd-f2fc/containers 4.09M 250M 96K legacy
olxd-f2fc/containers/oel84d10 88K 250M 575M legacy
olxd-f2fc/containers/oel84d11 3.91M 250M 573M legacy
olxd-f2fc/custom 96K 250M 96K legacy
olxd-f2fc/deleted 480K 250M 96K legacy
olxd-f2fc/deleted/containers 96K 250M 96K legacy
olxd-f2fc/deleted/custom 96K 250M 96K legacy
olxd-f2fc/deleted/images 96K 250M 96K legacy
olxd-f2fc/deleted/virtual-machines 96K 250M 96K legacy
olxd-f2fc/images 575M 250M 96K legacy
olxd-f2fc/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 575M 250M 575M legacy
olxd-f2fc/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066@readonly 0B - 575M -
olxd-f2fc/virtual-machines 96K 250M 96K legacy
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-f2fc/containers/oel84d10
[orabuntu@f34sv1 ~]$ lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix
Creating oel84d10
Starting oel84d10
[orabuntu@f34sv1 ~]$
I guess I will just put some code into Orabuntu-LXC to just do the «zfs destroy» step if this «timeout awaiting response headers» error is detected.
I never got these problems before in the lxd 4.x series. This really seems to be a new issue that was introduced in lxd 5.x I ran the following just now with no issue:
The timeouts are in LXD 5.1, not LXD 5.0. My point is that we are trying to figure out why its taking over 30s to get the headers, and in this case it seems like you have multiple ongoing requests for creating the same instance, which is likely to incur delays due to locking.
I’m trying to understand your workflow, as I’m only getting part of the picture each time (either the server side logs or the client side logs), but never together with the command requests initiated. I need to build up a full picture of what you are doing to reproduce this issue.
My suggestion would be to clear up any partially created containers and then re-run your script (showing the specific commands you are running) along with the --debug log of those commands and the server-side debug log, so we can see what is going on specifically.
This is the code that was added when this problem started apparently with 5.1:
LaunchCode=1
n=0
while [ $LaunchCode -ne 0 ] && [ $n -le 5 ]
do
eval echo "'/var/lib/snapd/snap/bin/lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
LaunchCode=`echo $?`
if [ $LaunchCode -ne 0 ]
then
eval echo "'/var/lib/snapd/snap/bin/lxc stop -f oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
eval echo "'/var/lib/snapd/snap/bin/lxc delete oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
sudo zfs destroy $LXDStoragePoolName/containers/oel$OracleRelease$SeedPostfix
sleep 5
fi
n=$((n+1))
done
Prior to that everything «just worked» and there was no retry code. I actually think this is really a good thing (especially once it’s resolved) because there should be some retry code in Orabuntu-LXC to address the case that the container creation fails for whatever reason — so this has at least resulted it making Orabuntu-LXC more robust which is a good thing.
But prior to 5.1 this lxc launch step «just worked» always everytime — which is not proof that 5.1 is the culprit, but there is a possiblity that it is. The «zfs destroy» step was added just now.
I also added «—debug» to the initial «launch» command for the next test of the entire Orabuntu-LXC deploy run:
LaunchCode=1
n=0
while [ $LaunchCode -ne 0 ] && [ $n -le 5 ]
do
eval echo "'/var/lib/snapd/snap/bin/lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix --debug' | sg lxd $CGROUP_SUFFIX"
LaunchCode=`echo $?`
if [ $LaunchCode -ne 0 ]
then
eval echo "'/var/lib/snapd/snap/bin/lxc stop -f oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
eval echo "'/var/lib/snapd/snap/bin/lxc delete oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
sudo zfs destroy $LXDStoragePoolName/containers/oel$OracleRelease$SeedPostfix
sleep 5
fi
n=$((n+1))
done
Does this still occur if you run lxc list first before running your script (to allow LXD to start first).
Prior to that everything «just worked» and there was no retry code. I actually think this is really a good thing (especially once it’s resolved) because there should be some retry code in Orabuntu-LXC to address the case that the container creation fails for whatever reason — so this has at least resulted it making Orabuntu-LXC more robust which is a good thing.
Can I also get the logs without the retry code, as I think that might be why the confusing logs are showing concurrent instance creation.
First thank you so much for your valuable time on this. Really appreciated!
Doing a full run of Orabuntu-LXC now and will capture all the things you have requested. The code prior to the to the «lxc launch» is now as follows:
echo "=============================================="
echo "List LXD Cluster ... "
echo "=============================================="
echo ''
eval echo "'/var/lib/snapd/snap/bin/lxc cluster list' | sg lxd $CGROUP_SUFFIX"
sleep 5
clear
echo ''
echo "=============================================="
echo "Done: List LXD Cluster. "
echo "=============================================="
echo ''
sleep 5
clear
echo ''
echo "=============================================="
echo "List LXD Containers... "
echo "=============================================="
echo ''
eval echo "'/var/lib/snapd/snap/bin/lxc list' | sg lxd $CGROUP_SUFFIX"
sleep 5
clear
echo ''
echo "=============================================="
echo "Done: List LXD Containers. "
echo "=============================================="
echo ''
sleep 5
clear
The «—debug» on the inital «lxc launch» should capture logging prior to the retry code.
The lxc-monitor service will log retries.
Does it all meet your requirements ?
Also just FYI, regarding a side issue, in case you might be wondering: The $CGROUP_SUFFIX after the «sg lxd» is currently an empty null enviro variable. It was used previously for earlier issues on Fedora described below and was used to suppress errors during the run of Orabuntu-LXC on Fedora related to this issue (but this went away apparently due to snapd improvements so that variable is now empty). The issues on Fedora with snapd were related to the use of cgroupv2 and nftables
echo ''
echo "=============================================="
echo "On $LinuxFlavor $RedHatVersion the WARNING: "
echo " "
echo "WARNING: cgroup v2 is not fully supported yet "
echo "proceeding with partial confinement. "
echo " "
echo "can be safely IGNORED. "
echo "This is a snapd issue not an LXD issue. "
echo " "
echo "This specific warning has been suppressed "
echo "during this install of Orabuntu-LXC. "
echo " "
echo " More info here: "
echo " "
echo "https://discuss.linuxcontainers.org/t/lxd-cgroup-v2-support/10455"
echo "https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1850667"
echo " "
echo "=============================================="
Here is the —debug from the first attempt:
installed: 5.1-1f6f485
==============================================
Launch Oracle LXD Seed Container...
==============================================
==============================================
(takes a minute or two ... patience) ...
==============================================
DEBUG [2022-05-11T07:59:45-05:00] Connecting to a local LXD over a Unix socket
DEBUG [2022-05-11T07:59:45-05:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
DEBUG [2022-05-11T07:59:45-05:00] Got response struct from LXD
DEBUG [2022-05-11T07:59:45-05:00]
{
"config": {
"cluster.https_address": "10.209.53.1:8443",
"core.https_address": "10.209.53.1:8443",
"core.trust_password": true
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"candid_authentication",
"backup_compression",
"candid_config",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"candid_config_key",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"rbac",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"10.209.53.1:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "-----BEGIN CERTIFICATE-----nMIICBDCCAYqgAwIBAgIRAK7JizW5VTRBuz/K4S0mS+YwCgYIKoZIzj0EAwMwNDEcnMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEUMBIGA1UEAwwLcm9vdEBmMzRzndjEwHhcNMjIwNTExMTI1ODIwWhcNMzIwNTA4MTI1ODIwWjA0MRwwGgYDVQQKExNsnaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2MTB2MBAGByqGnSM49AgEGBSuBBAAiA2IABHwpvgG9f4rExLy1cvCCzHEHpJ7WV2uMHSxHRPJX7XP/nVntjCOXvS1PvF78r+kUkd1bBbkVOdTGQuNQozTc8hTqm8uYazzfEDNgOoHTyOKs3n2AOj82BrVDkKna5afEv/AqNgMF4wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGnCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0RBCIwIIIGZjM0c3YxhwR/AAABnhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMQDZUSLGDShkMb46nqqb3sKmjq4bs9TdlEOFbjZ1t1zTAlwDVbsFvjmVtQ9Y2lBH0TYcCMCzv5gbzBmTOnvWM8QmuCtpnI2fYlOi2CVdJdXVqj3NS2mDSPHqRCs+gJcUpoVkEzGA==n-----END CERTIFICATE-----n",
"certificate_fingerprint": "e1975887f497482483ff68da74d24020a25b3b34451928c591348e549c794614",
"driver": "lxc",
"driver_version": "4.0.12",
"firewall": "nftables",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": {
"idmapped_mounts": "true",
"netnsid_getifaddrs": "true",
"seccomp_listener": "true",
"seccomp_listener_continue": "true",
"shiftfs": "false",
"uevent_injection": "true",
"unpriv_fscaps": "true"
},
"kernel_version": "5.17.5-100.fc34.x86_64",
"lxc_features": {
"cgroup2": "true",
"core_scheduling": "true",
"devpts_fd": "true",
"idmapped_mounts_v2": "true",
"mount_injection_file": "true",
"network_gateway_device_route": "true",
"network_ipvlan": "true",
"network_l2proxy": "true",
"network_phys_macvlan_mtu": "true",
"network_veth_router": "true",
"pidfd": "true",
"seccomp_allow_deny_syntax": "true",
"seccomp_notify": "true",
"seccomp_proxy_send_notify_fd": "true"
},
"os_name": "Fedora",
"os_version": "34",
"project": "default",
"server": "lxd",
"server_clustered": true,
"server_event_mode": "full-mesh",
"server_name": "f34sv1",
"server_pid": 107570,
"server_version": "5.1",
"storage": "zfs",
"storage_version": "2.1.4-1",
"storage_supported_drivers": [
{
"Name": "lvm",
"Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
"Remote": false
},
{
"Name": "zfs",
"Version": "2.1.4-1",
"Remote": false
},
{
"Name": "ceph",
"Version": "15.2.16",
"Remote": true
},
{
"Name": "btrfs",
"Version": "5.4.1",
"Remote": false
},
{
"Name": "cephfs",
"Version": "15.2.16",
"Remote": true
},
{
"Name": "dir",
"Version": "1",
"Remote": false
}
]
}
}
Creating oel84d10
DEBUG [2022-05-11T07:59:45-05:00] Connecting to a remote simplestreams server URL="https://images.linuxcontainers.org"
DEBUG [2022-05-11T07:59:45-05:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG [2022-05-11T07:59:45-05:00] Sending request to LXD etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG [2022-05-11T07:59:45-05:00]
{
"architecture": "",
"config": {},
"devices": {},
"ephemeral": false,
"profiles": [
"olxc_sx1a"
],
"stateful": false,
"description": "",
"name": "oel84d10",
"source": {
"type": "image",
"certificate": "",
"alias": "oracle/8/amd64",
"server": "https://images.linuxcontainers.org",
"protocol": "simplestreams",
"mode": "pull",
"allow_inconsistent": false
},
"instance_type": "",
"type": "container"
}
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed checking instance exists "local:oel84d10": Instance not found
cannot open 'olxd-693c/containers/oel84d10': dataset does not exist
Also the retry code failed. There must be some additional cleanup steps needed:
lxc-monitor.seed.log
Ok so I’m going to try a run with this setting (and all the previous debug and retry code as well) and see how it goes:
sudo snap install lxd --channel=4.0/stable
I think we need to try without the retry code as it is confusing matters (by starting additional operations after the first one has timed out but still running on the server), making it impossible to see what is happening on the original operation. We aren’t moving forward at the moment.
I notice you’re running a cluster, are the cluster members in the same LAN?
Ok. I’m going to do a complete run of the entire program with the retry code (which should be moot anyway if 5.1 is the culprit):
sudo snap install lxd --channel=4.0/stable
Then a complete run of the entire program without the retry code with:
sudo snap install lxd
This node is the «first» node in the cluster, so it is created as a «single-node cluster». When additional cluster members are added, yes, they are all in the same LAN.
Ok change of plan I decided to try the 5.0 again so that you’d have something hopefully to «crunch away» on while I’m at my daytime job …
So first I made this change:
LaunchCode=1
n=0
while [ $LaunchCode -ne 0 ] && [ $n -le 5 ]
do
eval echo "'/var/lib/snapd/snap/bin/lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix --debug' | sg lxd $CGROUP_SUFFIX"
LaunchCode=`echo $?`
if [ $LaunchCode -ne 0 ]
then
sleep 300
eval echo "'/var/lib/snapd/snap/bin/lxc stop -f oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
eval echo "'/var/lib/snapd/snap/bin/lxc delete oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
sleep 300
fi
n=$((n+1))
done
with the «sleep 300» in there bracketed so that this would not «step on» the logging as you had indicated.
Now here is what I’m getting from debug (and note the operations that went on in the next chunk of output after this that I was doing in another terminal session to «clean up» zfs).
==============================================
Launch Oracle LXD Seed Container...
==============================================
==============================================
(takes a minute or two ... patience) ...
==============================================
DEBUG [2022-05-11T09:56:34-05:00] Connecting to a local LXD over a Unix socket
DEBUG [2022-05-11T09:56:34-05:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
DEBUG [2022-05-11T09:56:34-05:00] Got response struct from LXD
DEBUG [2022-05-11T09:56:34-05:00]
{
"config": {
"cluster.https_address": "10.209.53.1:8443",
"core.https_address": "10.209.53.1:8443",
"core.trust_password": true
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"candid_authentication",
"backup_compression",
"candid_config",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"candid_config_key",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"rbac",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"10.209.53.1:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "-----BEGIN CERTIFICATE-----nMIICBDCCAYmgAwIBAgIQAwl2IyAmGY2+3ouePQ88zjAKBggqhkjOPQQDAzA0MRwwnGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2nMTAeFw0yMjA1MTExNDU1MTVaFw0zMjA1MDgxNDU1MTVaMDQxHDAaBgNVBAoTE2xpnbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZjM0c3YxMHYwEAYHKoZInzj0CAQYFK4EEACIDYgAEts7SvdM2DhS4l/UYWeX/HsK7w07rUnAFwnAJQNDTCWacnIXhK3VFCSQ0CjBjFozjPIaOQOklAsrz7qbfk13cthBsyygvbuKi1lfTPR7KBzJFYngpugSZwdApBkfPChIkOyo2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYInKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZmMzRzdjGHBH8AAAGHnEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxALSKfieSrggNLT5/nYPQrfS/xz1KDfjV5Dt2Fsaqi4RkKPNZ5SnCY61mhC5J46lTvCgIxAIfA+SiOoCdqnzQv8qxTEyv9gonPNdXPrZmYhUwHokYW1NVWJH7bdn3HD5T8jqaqWJA==n-----END CERTIFICATE-----n",
"certificate_fingerprint": "65cbbaa39a3fb9be2da92e754ecf3db4341d1fec2184b9e2b80802bd62147c0b",
"driver": "lxc",
"driver_version": "4.0.12",
"firewall": "nftables",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": {
"idmapped_mounts": "true",
"netnsid_getifaddrs": "true",
"seccomp_listener": "true",
"seccomp_listener_continue": "true",
"shiftfs": "false",
"uevent_injection": "true",
"unpriv_fscaps": "true"
},
"kernel_version": "5.17.6-100.fc34.x86_64",
"lxc_features": {
"cgroup2": "true",
"core_scheduling": "true",
"devpts_fd": "true",
"idmapped_mounts_v2": "true",
"mount_injection_file": "true",
"network_gateway_device_route": "true",
"network_ipvlan": "true",
"network_l2proxy": "true",
"network_phys_macvlan_mtu": "true",
"network_veth_router": "true",
"pidfd": "true",
"seccomp_allow_deny_syntax": "true",
"seccomp_notify": "true",
"seccomp_proxy_send_notify_fd": "true"
},
"os_name": "Fedora",
"os_version": "34",
"project": "default",
"server": "lxd",
"server_clustered": true,
"server_event_mode": "full-mesh",
"server_name": "f34sv1",
"server_pid": 49758,
"server_version": "5.1",
"storage": "zfs",
"storage_version": "2.1.4-1",
"storage_supported_drivers": [
{
"Name": "btrfs",
"Version": "5.4.1",
"Remote": false
},
{
"Name": "cephfs",
"Version": "15.2.14",
"Remote": true
},
{
"Name": "dir",
"Version": "1",
"Remote": false
},
{
"Name": "lvm",
"Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
"Remote": false
},
{
"Name": "zfs",
"Version": "2.1.4-1",
"Remote": false
},
{
"Name": "ceph",
"Version": "15.2.14",
"Remote": true
}
]
}
}
Creating oel84d10
DEBUG [2022-05-11T09:56:34-05:00] Connecting to a remote simplestreams server URL="https://images.linuxcontainers.org"
DEBUG [2022-05-11T09:56:34-05:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG [2022-05-11T09:56:34-05:00] Sending request to LXD etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG [2022-05-11T09:56:34-05:00]
{
"architecture": "",
"config": {},
"devices": {},
"ephemeral": false,
"profiles": [
"olxc_sx1a"
],
"stateful": false,
"description": "",
"name": "oel84d10",
"source": {
"type": "image",
"certificate": "",
"alias": "oracle/8/amd64",
"server": "https://images.linuxcontainers.org",
"protocol": "simplestreams",
"mode": "pull",
"allow_inconsistent": false
},
"instance_type": "",
"type": "container"
}
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: The instance is already stopped
DEBUG [2022-05-11T10:07:05-05:00] Connecting to a local LXD over a Unix socket
DEBUG [2022-05-11T10:07:05-05:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
DEBUG [2022-05-11T10:07:05-05:00] Got response struct from LXD
DEBUG [2022-05-11T10:07:05-05:00]
{
"config": {
"cluster.https_address": "10.209.53.1:8443",
"core.https_address": "10.209.53.1:8443",
"core.trust_password": true
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup",
"devlxd_images",
"container_local_cross_pool_handling",
"proxy_unix",
"proxy_udp",
"clustering_join",
"proxy_tcp_udp_multi_port_handling",
"network_state",
"proxy_unix_dac_properties",
"container_protection_delete",
"unix_priv_drop",
"pprof_http",
"proxy_haproxy_protocol",
"network_hwaddr",
"proxy_nat",
"network_nat_order",
"container_full",
"candid_authentication",
"backup_compression",
"candid_config",
"nvidia_runtime_config",
"storage_api_volume_snapshots",
"storage_unmapped",
"projects",
"candid_config_key",
"network_vxlan_ttl",
"container_incremental_copy",
"usb_optional_vendorid",
"snapshot_scheduling",
"snapshot_schedule_aliases",
"container_copy_project",
"clustering_server_address",
"clustering_image_replication",
"container_protection_shift",
"snapshot_expiry",
"container_backup_override_pool",
"snapshot_expiry_creation",
"network_leases_location",
"resources_cpu_socket",
"resources_gpu",
"resources_numa",
"kernel_features",
"id_map_current",
"event_location",
"storage_api_remote_volume_snapshots",
"network_nat_address",
"container_nic_routes",
"rbac",
"cluster_internal_copy",
"seccomp_notify",
"lxc_features",
"container_nic_ipvlan",
"network_vlan_sriov",
"storage_cephfs",
"container_nic_ipfilter",
"resources_v2",
"container_exec_user_group_cwd",
"container_syscall_intercept",
"container_disk_shift",
"storage_shifted",
"resources_infiniband",
"daemon_storage",
"instances",
"image_types",
"resources_disk_sata",
"clustering_roles",
"images_expiry",
"resources_network_firmware",
"backup_compression_algorithm",
"ceph_data_pool_name",
"container_syscall_intercept_mount",
"compression_squashfs",
"container_raw_mount",
"container_nic_routed",
"container_syscall_intercept_mount_fuse",
"container_disk_ceph",
"virtual-machines",
"image_profiles",
"clustering_architecture",
"resources_disk_id",
"storage_lvm_stripes",
"vm_boot_priority",
"unix_hotplug_devices",
"api_filtering",
"instance_nic_network",
"clustering_sizing",
"firewall_driver",
"projects_limits",
"container_syscall_intercept_hugetlbfs",
"limits_hugepages",
"container_nic_routed_gateway",
"projects_restrictions",
"custom_volume_snapshot_expiry",
"volume_snapshot_scheduling",
"trust_ca_certificates",
"snapshot_disk_usage",
"clustering_edit_roles",
"container_nic_routed_host_address",
"container_nic_ipvlan_gateway",
"resources_usb_pci",
"resources_cpu_threads_numa",
"resources_cpu_core_die",
"api_os",
"container_nic_routed_host_table",
"container_nic_ipvlan_host_table",
"container_nic_ipvlan_mode",
"resources_system",
"images_push_relay",
"network_dns_search",
"container_nic_routed_limits",
"instance_nic_bridged_vlan",
"network_state_bond_bridge",
"usedby_consistency",
"custom_block_volumes",
"clustering_failure_domains",
"resources_gpu_mdev",
"console_vga_type",
"projects_limits_disk",
"network_type_macvlan",
"network_type_sriov",
"container_syscall_intercept_bpf_devices",
"network_type_ovn",
"projects_networks",
"projects_networks_restricted_uplinks",
"custom_volume_backup",
"backup_override_name",
"storage_rsync_compression",
"network_type_physical",
"network_ovn_external_subnets",
"network_ovn_nat",
"network_ovn_external_routes_remove",
"tpm_device_type",
"storage_zfs_clone_copy_rebase",
"gpu_mdev",
"resources_pci_iommu",
"resources_network_usb",
"resources_disk_address",
"network_physical_ovn_ingress_mode",
"network_ovn_dhcp",
"network_physical_routes_anycast",
"projects_limits_instances",
"network_state_vlan",
"instance_nic_bridged_port_isolation",
"instance_bulk_state_change",
"network_gvrp",
"instance_pool_move",
"gpu_sriov",
"pci_device_type",
"storage_volume_state",
"network_acl",
"migration_stateful",
"disk_state_quota",
"storage_ceph_features",
"projects_compression",
"projects_images_remote_cache_expiry",
"certificate_project",
"network_ovn_acl",
"projects_images_auto_update",
"projects_restricted_cluster_target",
"images_default_architecture",
"network_ovn_acl_defaults",
"gpu_mig",
"project_usage",
"network_bridge_acl",
"warnings",
"projects_restricted_backups_and_snapshots",
"clustering_join_token",
"clustering_description",
"server_trusted_proxy",
"clustering_update_cert",
"storage_api_project",
"server_instance_driver_operational",
"server_supported_storage_drivers",
"event_lifecycle_requestor_address",
"resources_gpu_usb",
"clustering_evacuation",
"network_ovn_nat_address",
"network_bgp",
"network_forward",
"custom_volume_refresh",
"network_counters_errors_dropped",
"metrics",
"image_source_project",
"clustering_config",
"network_peer",
"linux_sysctl",
"network_dns",
"ovn_nic_acceleration",
"certificate_self_renewal",
"instance_project_move",
"storage_volume_project_move",
"cloud_init",
"network_dns_nat",
"database_leader",
"instance_all_projects",
"clustering_groups",
"ceph_rbd_du",
"instance_get_full",
"qemu_metrics",
"gpu_mig_uuid",
"event_project",
"clustering_evacuation_live",
"instance_allow_inconsistent_copy",
"network_state_ovn",
"storage_volume_api_filtering",
"image_restrictions",
"storage_zfs_export",
"network_dns_records",
"storage_zfs_reserve_space",
"network_acl_log",
"storage_zfs_blocksize",
"metrics_cpu_seconds",
"instance_snapshot_never",
"certificate_token",
"instance_nic_routed_neighbor_probe",
"event_hub",
"agent_nic_config",
"projects_restricted_intercept",
"metrics_authentication",
"images_target_project",
"cluster_migration_inconsistent_copy",
"cluster_ovn_chassis",
"container_syscall_intercept_sched_setscheduler",
"storage_lvm_thinpool_metadata_size",
"storage_volume_state_total",
"instance_file_head",
"instances_nic_host_name",
"image_copy_profile",
"container_syscall_intercept_sysinfo",
"clustering_evacuation_mode"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"10.209.53.1:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "-----BEGIN CERTIFICATE-----nMIICBDCCAYmgAwIBAgIQAwl2IyAmGY2+3ouePQ88zjAKBggqhkjOPQQDAzA0MRwwnGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2nMTAeFw0yMjA1MTExNDU1MTVaFw0zMjA1MDgxNDU1MTVaMDQxHDAaBgNVBAoTE2xpnbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZjM0c3YxMHYwEAYHKoZInzj0CAQYFK4EEACIDYgAEts7SvdM2DhS4l/UYWeX/HsK7w07rUnAFwnAJQNDTCWacnIXhK3VFCSQ0CjBjFozjPIaOQOklAsrz7qbfk13cthBsyygvbuKi1lfTPR7KBzJFYngpugSZwdApBkfPChIkOyo2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYInKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZmMzRzdjGHBH8AAAGHnEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxALSKfieSrggNLT5/nYPQrfS/xz1KDfjV5Dt2Fsaqi4RkKPNZ5SnCY61mhC5J46lTvCgIxAIfA+SiOoCdqnzQv8qxTEyv9gonPNdXPrZmYhUwHokYW1NVWJH7bdn3HD5T8jqaqWJA==n-----END CERTIFICATE-----n",
"certificate_fingerprint": "65cbbaa39a3fb9be2da92e754ecf3db4341d1fec2184b9e2b80802bd62147c0b",
"driver": "lxc",
"driver_version": "4.0.12",
"firewall": "nftables",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_features": {
"idmapped_mounts": "true",
"netnsid_getifaddrs": "true",
"seccomp_listener": "true",
"seccomp_listener_continue": "true",
"shiftfs": "false",
"uevent_injection": "true",
"unpriv_fscaps": "true"
},
"kernel_version": "5.17.6-100.fc34.x86_64",
"lxc_features": {
"cgroup2": "true",
"core_scheduling": "true",
"devpts_fd": "true",
"idmapped_mounts_v2": "true",
"mount_injection_file": "true",
"network_gateway_device_route": "true",
"network_ipvlan": "true",
"network_l2proxy": "true",
"network_phys_macvlan_mtu": "true",
"network_veth_router": "true",
"pidfd": "true",
"seccomp_allow_deny_syntax": "true",
"seccomp_notify": "true",
"seccomp_proxy_send_notify_fd": "true"
},
"os_name": "Fedora",
"os_version": "34",
"project": "default",
"server": "lxd",
"server_clustered": true,
"server_event_mode": "full-mesh",
"server_name": "f34sv1",
"server_pid": 49758,
"server_version": "5.1",
"storage": "zfs",
"storage_version": "2.1.4-1",
"storage_supported_drivers": [
{
"Name": "btrfs",
"Version": "5.4.1",
"Remote": false
},
{
"Name": "cephfs",
"Version": "15.2.14",
"Remote": true
},
{
"Name": "dir",
"Version": "1",
"Remote": false
},
{
"Name": "lvm",
"Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
"Remote": false
},
{
"Name": "zfs",
"Version": "2.1.4-1",
"Remote": false
},
{
"Name": "ceph",
"Version": "15.2.14",
"Remote": true
}
]
}
}
Creating oel84d10
DEBUG [2022-05-11T10:07:05-05:00] Connecting to a remote simplestreams server URL="https://images.linuxcontainers.org"
DEBUG [2022-05-11T10:07:05-05:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG [2022-05-11T10:07:05-05:00] Sending request to LXD etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG [2022-05-11T10:07:05-05:00]
{
"architecture": "",
"config": {},
"devices": {},
"ephemeral": false,
"profiles": [
"olxc_sx1a"
],
"stateful": false,
"description": "",
"name": "oel84d10",
"source": {
"type": "image",
"certificate": "",
"alias": "oracle/8/amd64",
"server": "https://images.linuxcontainers.org",
"protocol": "simplestreams",
"mode": "pull",
"allow_inconsistent": false
},
"instance_type": "",
"type": "container"
}
DEBUG [2022-05-11T10:07:05-05:00] Got operation from LXD
DEBUG [2022-05-11T10:07:05-05:00]
{
"id": "1f005db2-b0c6-4ffd-80d7-ac85d454ac54",
"class": "task",
"description": "Creating instance",
"created_at": "2022-05-11T10:07:05.167961848-05:00",
"updated_at": "2022-05-11T10:07:05.167961848-05:00",
"status": "Running",
"status_code": 103,
"resources": {
"containers": [
"/1.0/containers/oel84d10"
],
"instances": [
"/1.0/instances/oel84d10"
]
},
"metadata": null,
"may_cancel": false,
"err": "",
"location": "f34sv1"
}
DEBUG [2022-05-11T10:07:05-05:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/operations/1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
DEBUG [2022-05-11T10:07:05-05:00] Got response struct from LXD
DEBUG [2022-05-11T10:07:05-05:00]
{
"id": "1f005db2-b0c6-4ffd-80d7-ac85d454ac54",
"class": "task",
"description": "Creating instance",
"created_at": "2022-05-11T10:07:05.167961848-05:00",
"updated_at": "2022-05-11T10:07:05.167961848-05:00",
"status": "Running",
"status_code": 103,
"resources": {
"containers": [
"/1.0/containers/oel84d10"
],
"instances": [
"/1.0/instances/oel84d10"
]
},
"metadata": null,
"may_cancel": false,
"err": "",
"location": "f34sv1"
}
Error: Failed instance creation: Failed creating instance from image: Error inserting volume "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6" for project "default" in pool "local" of type "images" into database "UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type"
Then over here in another terminal I was doing some additional steps to see if I could get the cleanup zfs steps down.
orabuntu@olxd1:~/Desktop$ ssh 192.168.1.146
orabuntu@192.168.1.146's password:
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Wed May 11 09:53:15 2022 from 192.168.1.99
[orabuntu@f34sv1 ~]$ lxc list
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
[orabuntu@f34sv1 ~]$ lxc cluster list
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| f34sv1 | https://10.209.53.1:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
[orabuntu@f34sv1 ~]$ lxc list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
[sudo] password for orabuntu:
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ ls -lrt
total 8
drwxr-xr-x. 3 orabuntu orabuntu 51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu 486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu 17 May 11 09:55 snap
-rw-r--r--. 1 root root 0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root root 187 May 11 09:56 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 8
drwxr-xr-x. 3 orabuntu orabuntu 51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu 486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu 17 May 11 09:55 snap
-rw-r--r--. 1 root root 0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root root 3042 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu 51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu 486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu 17 May 11 09:55 snap
-rw-r--r--. 1 root root 0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root root 4161 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu 51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu 486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu 17 May 11 09:55 snap
-rw-r--r--. 1 root root 0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root root 4161 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu 51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu 486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu 17 May 11 09:55 snap
-rw-r--r--. 1 root root 0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root root 4533 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu 51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu 486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu 17 May 11 09:55 snap
-rw-r--r--. 1 root root 0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root root 4533 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T09:57:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:57:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:57:56-05:00" level=debug msg="Completed heartbeat round" duration=5.292361ms local="10.209.53.1:8443"
time="2022-05-11T09:58:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:06-05:00" level=debug msg="Completed heartbeat round" duration=18.984031ms local="10.209.53.1:8443"
time="2022-05-11T09:58:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:16-05:00" level=debug msg="Completed heartbeat round" duration=5.961386ms local="10.209.53.1:8443"
time="2022-05-11T09:58:17-05:00" level=debug msg="Success for task operation: 3dbc8217-c1b5-4cfe-91f5-b1b080bbecc3"
time="2022-05-11T09:58:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:26-05:00" level=debug msg="Completed heartbeat round" duration=22.35438ms local="10.209.53.1:8443"
time="2022-05-11T09:58:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:36-05:00" level=debug msg="Completed heartbeat round" duration=17.02173ms local="10.209.53.1:8443"
time="2022-05-11T09:58:36-05:00" level=debug msg="New task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:58:36-05:00" level=debug msg="Started task operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:58:36-05:00" level=debug msg="Connecting to a remote simplestreams server" URL="https://images.linuxcontainers.org"
time="2022-05-11T09:58:36-05:00" level=debug msg="Lock acquired for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T09:58:36-05:00" level=debug msg="Acquiring lock for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T09:58:36-05:00" level=debug msg="Database error" err="Image not found"
time="2022-05-11T09:58:36-05:00" level=info msg="Downloading image" alias=oracle/8/amd64 fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 operation=8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4 server="https://images.linuxcontainers.org" trigger=/1.0/operations/8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4
time="2022-05-11T09:58:36-05:00" level=debug msg="Database error" err="Image not found"
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.86M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T09:58:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:46-05:00" level=debug msg="Completed heartbeat round" duration=24.448558ms local="10.209.53.1:8443"
time="2022-05-11T09:58:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:56-05:00" level=debug msg="Completed heartbeat round" duration=3.37388ms local="10.209.53.1:8443"
time="2022-05-11T09:59:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:06-05:00" level=debug msg="Completed heartbeat round" duration=11.050051ms local="10.209.53.1:8443"
time="2022-05-11T09:59:07-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:16-05:00" level=debug msg="Completed heartbeat round" duration=5.742284ms local="10.209.53.1:8443"
time="2022-05-11T09:59:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:26-05:00" level=debug msg="Completed heartbeat round" duration=5.348088ms local="10.209.53.1:8443"
time="2022-05-11T09:59:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:36-05:00" level=debug msg="Completed heartbeat round" duration=13.097256ms local="10.209.53.1:8443"
time="2022-05-11T09:59:37-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:38-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:38-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:39-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:39-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:40-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:40-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:40-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:41-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:41-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:42-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:42-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:43-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:43-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:44-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:44-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:45-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:45-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:45-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:46-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:46-05:00" level=debug msg="Completed heartbeat round" duration=4.944594ms local="10.209.53.1:8443"
time="2022-05-11T09:59:47-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:47-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:48-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:48-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:48-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:49-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:49-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:50-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:50-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:51-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:51-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:52-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:52-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:52-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:53-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:53-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:54-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:54-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:54-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:55-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:55-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:56-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:56-05:00" level=debug msg="Completed heartbeat round" duration=17.596825ms local="10.209.53.1:8443"
time="2022-05-11T09:59:56-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:56-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:57-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:57-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:58-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:58-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:59-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:59-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:00-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:00-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:01-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:01-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:01-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:02-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:02-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:03-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:03-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:04-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:04-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:05-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:05-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:06-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:06-05:00" level=debug msg="Completed heartbeat round" duration=7.920559ms local="10.209.53.1:8443"
time="2022-05-11T10:00:06-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:06-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:07-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:07-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:08-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:08-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:09-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:09-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:10-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:10-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:10-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:11-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:11-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:12-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:12-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:13-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:13-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:14-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:14-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:14-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:15-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:16-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:16-05:00" level=debug msg="Completed heartbeat round" duration=9.288516ms local="10.209.53.1:8443"
time="2022-05-11T10:00:16-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:16-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:17-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:17-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:18-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:18-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:19-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:19-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:19-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:20-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:20-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:21-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:21-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:21-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:22-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:23-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:23-05:00" level=info msg="Image downloaded" alias=oracle/8/amd64 fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 operation=8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4 server="https://images.linuxcontainers.org" trigger=/1.0/operations/8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4
time="2022-05-11T10:00:23-05:00" level=debug msg="Lock acquired for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T10:00:23-05:00" level=debug msg="Acquiring lock for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T10:00:23-05:00" level=debug msg="Instance operation lock created" action=create instance=oel84d10 project=default reusable=false
time="2022-05-11T10:00:23-05:00" level=info msg="Creating container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:00:23-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:00:23-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:00:23-05:00" level=debug msg="Adding device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T10:00:23-05:00" level=debug msg="Adding device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T10:00:23-05:00" level=info msg="Created container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:00:23-05:00" level=debug msg="CreateInstanceFromImage started" instance=oel84d10 project=default
time="2022-05-11T10:00:23-05:00" level=debug msg="EnsureImage started" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:23-05:00" level=debug msg="Database error" err="Storage pool volume not found"
time="2022-05-11T10:00:23-05:00" level=debug msg="Running filler function" dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:00:23-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:00:23-05:00" level=info msg="Image unpack started" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:23-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:26-05:00" level=debug msg="Completed heartbeat round" duration=11.115758ms local="10.209.53.1:8443"
time="2022-05-11T10:00:26-05:00" level=info msg="Image unpack stopped" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:33-05:00" level=debug msg="EnsureImage finished" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:33-05:00" level=debug msg="Set new volume size" instance=oel84d10 project=default size=
time="2022-05-11T10:00:33-05:00" level=debug msg="Checking volume size" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
time="2022-05-11T10:00:33-05:00" level=debug msg="UpdateInstanceBackupFile started" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="CreateInstanceFromImage finished" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
time="2022-05-11T10:00:33-05:00" level=debug msg="Success for task operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:33-05:00" level=debug msg="UpdateInstanceBackupFile finished" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T10:00:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:36-05:00" level=debug msg="Completed heartbeat round" duration=15.842063ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 577M 28.0G 96K legacy
olxd-5f5e/containers 220K 28.0G 96K legacy
olxd-5f5e/containers/oel84d10 124K 28.0G 575M legacy
olxd-5f5e/custom 96K 28.0G 96K legacy
olxd-5f5e/deleted 480K 28.0G 96K legacy
olxd-5f5e/deleted/containers 96K 28.0G 96K legacy
olxd-5f5e/deleted/custom 96K 28.0G 96K legacy
olxd-5f5e/deleted/images 96K 28.0G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.0G 96K legacy
olxd-5f5e/images 575M 28.0G 96K legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 575M 28.0G 575M legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6@readonly 0B - 575M -
olxd-5f5e/virtual-machines 96K 28.0G 96K legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
time="2022-05-11T10:00:33-05:00" level=debug msg="Success for task operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:33-05:00" level=debug msg="UpdateInstanceBackupFile finished" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T10:00:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:36-05:00" level=debug msg="Completed heartbeat round" duration=15.842063ms local="10.209.53.1:8443"
time="2022-05-11T10:00:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:46-05:00" level=debug msg="Completed heartbeat round" duration=12.707894ms local="10.209.53.1:8443"
time="2022-05-11T10:00:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:56-05:00" level=debug msg="Completed heartbeat round" duration=18.441977ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 577M 28.0G 96K legacy
olxd-5f5e/containers 220K 28.0G 96K legacy
olxd-5f5e/containers/oel84d10 124K 28.0G 575M legacy
olxd-5f5e/custom 96K 28.0G 96K legacy
olxd-5f5e/deleted 480K 28.0G 96K legacy
olxd-5f5e/deleted/containers 96K 28.0G 96K legacy
olxd-5f5e/deleted/custom 96K 28.0G 96K legacy
olxd-5f5e/deleted/images 96K 28.0G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.0G 96K legacy
olxd-5f5e/images 575M 28.0G 96K legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 575M 28.0G 575M legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6@readonly 0B - 575M -
olxd-5f5e/virtual-machines 96K 28.0G 96K legacy
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
cannot destroy 'olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6': filesystem has children
use '-r' to destroy the following datasets:
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6@readonly
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 -r
cannot destroy 'olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6': filesystem has dependent clones
use '-R' to destroy the following datasets:
olxd-5f5e/containers/oel84d10
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 -R
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.98M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.98M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:02:06-05:00" level=debug msg="Completed heartbeat round" duration=9.257074ms local="10.209.53.1:8443"
time="2022-05-11T10:02:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:16-05:00" level=debug msg="Completed heartbeat round" duration=20.868486ms local="10.209.53.1:8443"
time="2022-05-11T10:02:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:26-05:00" level=debug msg="Completed heartbeat round" duration=9.358115ms local="10.209.53.1:8443"
time="2022-05-11T10:02:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:36-05:00" level=debug msg="Completed heartbeat round" duration=5.913498ms local="10.209.53.1:8443"
time="2022-05-11T10:02:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:46-05:00" level=debug msg="Completed heartbeat round" duration=10.019943ms local="10.209.53.1:8443"
time="2022-05-11T10:02:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:56-05:00" level=debug msg="Completed heartbeat round" duration=8.401611ms local="10.209.53.1:8443"
time="2022-05-11T10:03:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:06-05:00" level=debug msg="Completed heartbeat round" duration=18.892468ms local="10.209.53.1:8443"
time="2022-05-11T10:03:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:16-05:00" level=debug msg="Completed heartbeat round" duration=13.389547ms local="10.209.53.1:8443"
time="2022-05-11T10:03:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:26-05:00" level=debug msg="Completed heartbeat round" duration=11.055607ms local="10.209.53.1:8443"
time="2022-05-11T10:03:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:36-05:00" level=debug msg="Completed heartbeat round" duration=16.803647ms local="10.209.53.1:8443"
time="2022-05-11T10:03:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:46-05:00" level=debug msg="Completed heartbeat round" duration=20.290968ms local="10.209.53.1:8443"
time="2022-05-11T10:03:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:56-05:00" level=debug msg="Completed heartbeat round" duration=5.529903ms local="10.209.53.1:8443"
time="2022-05-11T10:04:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:06-05:00" level=debug msg="Completed heartbeat round" duration=22.841212ms local="10.209.53.1:8443"
time="2022-05-11T10:04:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:16-05:00" level=debug msg="Completed heartbeat round" duration=14.473922ms local="10.209.53.1:8443"
time="2022-05-11T10:04:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:26-05:00" level=debug msg="Completed heartbeat round" duration=17.520211ms local="10.209.53.1:8443"
time="2022-05-11T10:04:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:36-05:00" level=debug msg="Completed heartbeat round" duration=19.716728ms local="10.209.53.1:8443"
time="2022-05-11T10:04:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:46-05:00" level=debug msg="Completed heartbeat round" duration=10.958502ms local="10.209.53.1:8443"
time="2022-05-11T10:04:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:56-05:00" level=debug msg="Completed heartbeat round" duration=12.302142ms local="10.209.53.1:8443"
time="2022-05-11T10:05:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:06-05:00" level=debug msg="Completed heartbeat round" duration=6.756267ms local="10.209.53.1:8443"
time="2022-05-11T10:05:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:17-05:00" level=debug msg="Completed heartbeat round" duration=21.964416ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ sudo getenforce
Permissive
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:04:56-05:00" level=debug msg="Completed heartbeat round" duration=12.302142ms local="10.209.53.1:8443"
time="2022-05-11T10:05:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:06-05:00" level=debug msg="Completed heartbeat round" duration=6.756267ms local="10.209.53.1:8443"
time="2022-05-11T10:05:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:17-05:00" level=debug msg="Completed heartbeat round" duration=21.964416ms local="10.209.53.1:8443"
time="2022-05-11T10:05:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:26-05:00" level=debug msg="Completed heartbeat round" duration=17.583889ms local="10.209.53.1:8443"
time="2022-05-11T10:05:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:36-05:00" level=debug msg="Completed heartbeat round" duration=10.055679ms local="10.209.53.1:8443"
time="2022-05-11T10:05:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:46-05:00" level=debug msg="Completed heartbeat round" duration=14.412593ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ lxc list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
[orabuntu@f34sv1 ~]$ lxc cluste rlist
Error: unknown command "cluste" for "lxc"
Did you mean this?
cluster
[orabuntu@f34sv1 ~]$ lxc cluster list
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| f34sv1 | https://10.209.53.1:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:05:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:46-05:00" level=debug msg="Completed heartbeat round" duration=14.412593ms local="10.209.53.1:8443"
time="2022-05-11T10:05:52-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:05:52-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/instances?recursion=2" username=orabuntu
time="2022-05-11T10:05:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:56-05:00" level=debug msg="Completed heartbeat round" duration=20.03678ms local="10.209.53.1:8443"
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/cluster username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/cluster/members?recursion=1" username=orabuntu
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.98M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:05:56-05:00" level=debug msg="Completed heartbeat round" duration=20.03678ms local="10.209.53.1:8443"
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/cluster username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/cluster/members?recursion=1" username=orabuntu
time="2022-05-11T10:06:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:06-05:00" level=debug msg="Completed heartbeat round" duration=18.965641ms local="10.209.53.1:8443"
time="2022-05-11T10:06:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:16-05:00" level=debug msg="Completed heartbeat round" duration=19.170734ms local="10.209.53.1:8443"
time="2022-05-11T10:06:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:26-05:00" level=debug msg="Completed heartbeat round" duration=5.272961ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ cd /var/lib/lxd
-bash: cd: /var/lib/lxd: No such file or directory
[orabuntu@f34sv1 ~]$ cd /var/lib
[orabuntu@f34sv1 lib]$ l s-lrt
-bash: l: command not found
[orabuntu@f34sv1 lib]$ ls -lrt
total 8
drwxr-xr-x. 2 root root 6 Jan 26 2021 games
drwxr-xr-x. 2 root root 6 Jan 26 2021 fprint
drwxr-xr-x. 2 root root 6 Jan 27 2021 os-prober
drwxr-xr-x. 2 root root 6 Jan 28 2021 sheepdog
drwx------. 2 tss tss 6 Jan 28 2021 tpm
drwxr-xr-x. 2 root root 91 Mar 22 2021 rpm
drwxr-xr-x. 10 root root 120 Aug 16 2021 sss
drwxr-xr-x. 2 root root 6 Aug 22 2021 iwd
drwxr-xr-x. 2 root root 6 Aug 22 2021 ead
drwxr-xr-x. 2 root root 6 Sep 10 2021 PackageKit
drwxr-xr-x. 2 root root 6 Sep 28 2021 kdump
drwx------. 2 root root 6 Oct 4 2021 udisks2
drwxr-xr-x. 2 geoclue geoclue 6 Oct 7 2021 geoclue
drwxr-xr-x. 2 root root 6 Oct 28 2021 initramfs
drwxr-xr-x. 2 root root 6 Nov 2 2021 smartmontools
drwx------. 2 root root 6 Dec 11 19:38 portables
drwx------. 2 root root 6 Dec 11 19:38 private
drwxr-x---. 3 root polkitd 28 Dec 11 19:38 polkit-1
drwxr-xr-x. 5 root root 46 Dec 11 19:38 selinux
drwxr-xr-x. 8 root root 90 Dec 11 19:38 iscsi
drwxr-xr-x. 3 root root 20 Dec 11 19:40 tpm2-tss
drwxr-xr-x. 2 root root 27 Dec 11 19:41 plymouth
drwxr-xr-x. 2 unbound unbound 22 Dec 11 19:55 unbound
drwxr-xr-x. 2 root root 42 Dec 11 19:56 abrt
drwxr-xr-x. 8 root root 110 Jan 13 05:12 systemd
drwxr-xr-x. 4 root root 33 Jan 31 10:57 samba
drwxr-xr-x. 2 root dnsmasq 6 Feb 23 20:13 dnsmasq
drwxr-x---. 2 tss root 6 Mar 9 09:36 swtpm-localca
drwxr-xr-x. 3 root root 18 Mar 21 16:52 bluetooth
drwxr-xr-x. 2 root root 6 Apr 14 09:52 dhclient
drwxr-xr-x. 2 root root 6 Apr 15 08:55 flatpak
drwxr-xr-x. 3 root root 20 May 10 07:51 pcp
drwxr-xr-x. 2 root root 30 May 11 08:53 logrotate
drwxr-x---. 2 root slocate 24 May 11 08:53 mlocate
drwxr-xr-x. 4 root root 29 May 11 09:12 dkms
drwxr-xr-x. 5 root root 55 May 11 09:15 scst
drwxr-x---. 2 chrony chrony 19 May 11 09:20 chrony
drwxr-xr-x. 4 root root 55 May 11 09:21 gssproxy
drwx------. 2 setroubleshoot setroubleshoot 71 May 11 09:21 setroubleshoot
drwxr-xr-x. 11 root root 130 May 11 09:29 libvirt
drwx------. 2 root root 6 May 11 09:29 machines
drwxr-xr-x. 2 root root 58 May 11 09:37 dpkg
drwxr-xr-x. 2 root root 35 May 11 09:37 misc
drwxr-xr-x. 2 root root 4096 May 11 09:38 alternatives
drwxr-xr-x. 3 root root 17 May 11 09:38 color
drwxr-xr-x. 3 colord colord 17 May 11 09:38 colord
drwxr-xr-x. 3 root root 209 May 11 09:38 authselect
drwx------. 3 root root 17 May 11 09:39 ipsec
drwxr-xr-x. 3 root root 17 May 11 09:39 openvswitch
drwxr-xr-x. 5 root root 117 May 11 09:40 nfs
drwxr-xr-x. 3 root root 23 May 11 09:47 xfsdump
drwxr-xr-x. 5 root root 48 May 11 09:50 lxc
drwxr-xr-x. 3 root root 20 May 11 09:52 rpm-state
drwxr-xr-x. 3 root root 93 May 11 09:52 dnf
drwxr-xr-x. 18 root root 4096 May 11 09:59 snapd
drwx------. 2 root root 172 May 11 10:06 NetworkManager
drwx------. 2 root root 29 May 11 10:06 rsyslog
[orabuntu@f34sv1 lib]$ cd
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:06:26-05:00" level=debug msg="Completed heartbeat round" duration=5.272961ms local="10.209.53.1:8443"
time="2022-05-11T10:06:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:36-05:00" level=debug msg="Completed heartbeat round" duration=12.108497ms local="10.209.53.1:8443"
time="2022-05-11T10:06:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:47-05:00" level=debug msg="Completed heartbeat round" duration=19.480469ms local="10.209.53.1:8443"
time="2022-05-11T10:06:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:57-05:00" level=debug msg="Completed heartbeat round" duration=12.578933ms local="10.209.53.1:8443"
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Event listener server handler started" listener=64d2a265-8cd8-42a5-aafd-f7c9cf1206cd local=/var/snap/lxd/common/lxd/unix.socket remote=@
time="2022-05-11T10:07:05-05:00" level=debug msg="Responding to instance create"
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=POST protocol=unix url=/1.0/instances username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Connecting to a remote simplestreams server" URL="https://images.linuxcontainers.org"
time="2022-05-11T10:07:05-05:00" level=debug msg="New task Operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
time="2022-05-11T10:07:05-05:00" level=debug msg="Started task operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
time="2022-05-11T10:07:05-05:00" level=debug msg="Connecting to a remote simplestreams server" URL="https://images.linuxcontainers.org"
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/operations/1f005db2-b0c6-4ffd-80d7-ac85d454ac54 username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Lock acquired for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T10:07:05-05:00" level=debug msg="Acquiring lock for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T10:07:05-05:00" level=debug msg="Image already exists in the DB" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:05-05:00" level=debug msg="Lock acquired for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T10:07:05-05:00" level=debug msg="Acquiring lock for image "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6""
time="2022-05-11T10:07:05-05:00" level=debug msg="Instance operation lock created" action=create instance=oel84d10 project=default reusable=false
time="2022-05-11T10:07:05-05:00" level=info msg="Creating container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:07:05-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:07:05-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:07:05-05:00" level=debug msg="Adding device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T10:07:05-05:00" level=debug msg="Adding device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T10:07:05-05:00" level=info msg="Created container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:07:05-05:00" level=debug msg="CreateInstanceFromImage started" instance=oel84d10 project=default
time="2022-05-11T10:07:05-05:00" level=debug msg="EnsureImage started" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:05-05:00" level=info msg="Image unpack started" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:05-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:07:05-05:00" level=debug msg="Running filler function" dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:07:05-05:00" level=debug msg="Updated metadata for task Operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
time="2022-05-11T10:07:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:07-05:00" level=debug msg="Completed heartbeat round" duration=209.667138ms local="10.209.53.1:8443"
time="2022-05-11T10:07:08-05:00" level=info msg="Image unpack stopped" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:15-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:15-05:00" level=debug msg="Database error" err="UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type"
time="2022-05-11T10:07:15-05:00" level=debug msg="EnsureImage finished" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:15-05:00" level=debug msg="CreateInstanceFromImage finished" instance=oel84d10 project=default
time="2022-05-11T10:07:15-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T10:07:15-05:00" level=debug msg="Removing device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T10:07:15-05:00" level=debug msg="Removing device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T10:07:15-05:00" level=debug msg="Failure for task operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54: Failed creating instance from image: Error inserting volume "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6" for project "default" in pool "local" of type "images" into database "UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type""
time="2022-05-11T10:07:15-05:00" level=debug msg="Event listener server handler stopped" listener=64d2a265-8cd8-42a5-aafd-f7c9cf1206cd local=/var/snap/lxd/common/lxd/unix.socket remote=@
time="2022-05-11T10:07:17-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:17-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:17-05:00" level=debug msg="Completed heartbeat round" duration=4.967137ms local="10.209.53.1:8443"
time="2022-05-11T10:07:27-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:27-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:27-05:00" level=debug msg="Completed heartbeat round" duration=3.641348ms local="10.209.53.1:8443"
time="2022-05-11T10:07:37-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:37-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:37-05:00" level=debug msg="Completed heartbeat round" duration=3.260698ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ lxc list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
olxd-5f5e 1.98M 28.6G 96K legacy
olxd-5f5e/containers 96K 28.6G 96K legacy
olxd-5f5e/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted 480K 28.6G 96K legacy
olxd-5f5e/deleted/containers 96K 28.6G 96K legacy
olxd-5f5e/deleted/custom 96K 28.6G 96K legacy
olxd-5f5e/deleted/images 96K 28.6G 96K legacy
olxd-5f5e/deleted/virtual-machines 96K 28.6G 96K legacy
olxd-5f5e/images 96K 28.6G 96K legacy
olxd-5f5e/virtual-machines 96K 28.6G 96K legacy
[orabuntu@f34sv1 ~]$ sudo zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
olxd-5f5e 29.5G 1.98M 29.5G - - 0% 0% 1.00x ONLINE -
[orabuntu@f34sv1 ~]$ cd Downloads/orabuntu-lxc-master/uekulele/
[orabuntu@f34sv1 uekulele]$ vi uekulele-services-2.sh
[orabuntu@f34sv1 uekulele]$
Later about 12 hours or so from now I will try with
sudo snap install lxd --channel=4.0/stable
Finally here’s the logging:
lxc-monitor.log
HTH Gil just enough time to zip off to work …
And there’s still something missing from my «zfs cleanup» steps; —debug keeps giving this:
DEBUG [2022-05-11T10:17:16-05:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/operations/c970c925-4d1d-4958-93d0-d79725e75d7c"
DEBUG [2022-05-11T10:17:16-05:00] Got response struct from LXD
DEBUG [2022-05-11T10:17:16-05:00]
{
"id": "c970c925-4d1d-4958-93d0-d79725e75d7c",
"class": "task",
"description": "Creating instance",
"created_at": "2022-05-11T10:17:16.318963974-05:00",
"updated_at": "2022-05-11T10:17:16.318963974-05:00",
"status": "Running",
"status_code": 103,
"resources": {
"containers": [
"/1.0/containers/oel84d10"
],
"instances": [
"/1.0/instances/oel84d10"
]
},
"metadata": null,
"may_cancel": false,
"err": "",
"location": "f34sv1"
}
Error: Failed instance creation: Failed creating instance from image: Error inserting volume "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6" for project "default" in pool "local" of type "images" into database "UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type"
I’d like to also get the cleanup steps figured out so that I can code that into the project, just to make it more bulletproof, aside from solving this current issue.
will try the 4.0 tonight much later…
Only LXD 5.1 has the client timeouts so the tests need to be run on that version please, anything before that isn’t that helpful.
Although if you’re seeing issues on < LXD 5.1, then resolving them may also resolve the timeout issue you see on LXD 5.1
Everything I just posted was 5.1
I’m just gonna do a 4.0 test tonight to prove to myself no issues with 4.0
«os_name»: «Fedora»,
«os_version»: «34»,
«project»: «default»,
«server»: «lxd»,
«server_clustered»: true,
«server_event_mode»: «full-mesh»,
«server_name»: «f34sv1»,
«server_pid»: 49758,
«server_version»: «5.1»,
«storage»: «zfs»,
«storage_version»: «2.1.4-1»,
Everything I just posted was 5.1
I’m just gonna do a 4.0 test tonight to prove to myself no issues with 4.0
Ah it was «Ok change of plan I decided to try the 5.0 again» that confused me.
Sorry! I just thought it made more sense to try to generate better output for you than to do the 4.0 test. Sorry for the confusing flip flop. Yeah and the 5.0 was a typo sorry all the latest output above is 5.1
Sorry I’m afraid your logs still don’t show the corresponding server side for the timed out instance create request.
So I can see the client --debug log here starting at 2022-05-11T09:56:34:
Creating oel84d10
DEBUG [2022-05-11T09:56:34-05:00] Connecting to a remote simplestreams server URL="https://images.linuxcontainers.org"
DEBUG [2022-05-11T09:56:34-05:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG [2022-05-11T09:56:34-05:00] Sending request to LXD etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG [2022-05-11T09:56:34-05:00]
{
"architecture": "",
"config": {},
"devices": {},
"ephemeral": false,
"profiles": [
"olxc_sx1a"
],
"stateful": false,
"description": "",
"name": "oel84d10",
"source": {
"type": "image",
"certificate": "",
"alias": "oracle/8/amd64",
"server": "https://images.linuxcontainers.org",
"protocol": "simplestreams",
"mode": "pull",
"allow_inconsistent": false
},
"instance_type": "",
"type": "container"
}
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: The instance is already stopped
We can see the instance created partially failed due to it timing out and then the instance was attempted to be stopped, but it failed because it was already stopped.
However there is no corresponding server logs for the time around 2022-05-11T09:56:34, the earliest is for 2022-05-11T09:57:56.
A bit busy with other things, so I don’t have much of an update for you this morning, except to report that I changed the lxd version to:
sudo snap install lxd --channel=4.24/stable
and that is working perfectly no issues. I realize you pointed out that 5.0 also does not have the new timeout settings, but for the moment I’ve gone back to 4.24/stable. I’ll do some tests on 5.0 as soon as time permits, and I will also try to get you better logging for the 5.1 timeout.
Thanks, without seeing the associated server debug logs for the specific request that times out I’m not able to progress this issue.
Labels
Incomplete
Waiting on more information from reporter
Error thrown by GitLab CI since past hour, Not able to proceed. .gitlab-ci.yml is not changed and very simple (as shown in end).
Only Dockerfile is changed — but system is not event launching Dockerfile. What is the remedy?
ERROR LOG
Running with gitlab-ci-multi-runner 1.9.0 (82714ae)
Using Docker executor with image docker:latest ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Post https://67.205.149.124:2376/v1.18/images/create?fromImage=docker%3Adind: net/http: timeout awaiting response headers
Will be retried in 3s ...
Using Docker executor with image docker:latest ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Post https://67.205.141.137:2376/v1.18/images/create?fromImage=docker%3Adind: net/http: timeout awaiting response headers
Will be retried in 3s ...
Using Docker executor with image docker:latest ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Post https://192.241.149.44:2376/v1.18/images/create?fromImage=docker%3Adind: net/http: timeout awaiting response headers
Will be retried in 3s ...
ERROR: Build failed (system failure): Post https://192.241.149.44:2376/v1.18/images/create?fromImage=docker%3Adind: net/http: timeout awaiting response headers
.gitlab-ci.yml
image: docker:latest
services:
- docker:dind
stages:
- build
variables:
NAMEBRAN: registry.gitlab.com/myusername/project:$CI_BUILD_REF_NAME
NAMELAST: registry.gitlab.com/myusername/project:latest
build_image:
stage: build
script:
#LOGIN
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
#BUILD [NEEDS PUSH TO NAME] WITH NAME = $NAMEBRAN
- docker build -t $NAMEBRAN .
- docker push $NAMEBRAN
#TAG $NAMEBRAN as $NAMELAST
- docker pull $NAMEBRAN
- docker tag $NAMEBRAN $NAMELAST
- docker push $NAMELAST
only:
- dockerbranch
Loading
Open
Issue created Aug 06, 2019 by
multipart artifact uploads fails with `timeout awaiting response headers`
Summary
multipart artifact uploads to Minio fail with timeout awaiting response headers.
direct_upload set to false and background_upload set to true results in artifacts being uploaded on gitlab’s server (but I’m not sure if they end up being uploaded to minio at a later date)
Minio is using --compat
Steps to reproduce
Use an on-premise distributed instance of minio as an object store for large artifacts
What is the current bug behavior?
Artifact uploads fail
What is the expected correct behavior?
Artifact upload succeeds
Relevant logs and/or screenshots
{"correlation_id":"GcKAAGVxUr2","error":"handleFileUploads: extract files from multipart: persisting multipart file: upload part 4: PUT request "https://minio.company.com/com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25u0026partNumber=4u0026X-Amz-Expires=15299u0026X-Amz-Date=20190806T223446Zu0026X-Amz-Algorithm=AWS4-HMAC-SHA256u0026X-Amz-Credential=FILTERED%2F20190806%2Feu-east-1%2Fs3%2Faws4_requestu0026X-Amz-SignedHeaders=hostu0026X-Amz-Signature=[FILTERED]": Put https://minio.company.com/com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25u0026partNumber=4u0026X-Amz-Expires=15299u0026X-Amz-Date=20190806T223446Zu0026X-Amz-Algorithm=AWS4-HMAC-SHA256u0026X-Amz-Credential=FILTERED%2F20190806%2Feu-east-1%2Fs3%2Faws4_requestu0026X-Amz-SignedHeaders=hostu0026X-Amz-Signature=[FILTERED]: net/http: timeout awaiting response headers","level":"error","method":"POST","msg":"error","time":"2019-08-06T18:36:55-04:00","uri":"/api/v4/jobs/47733/artifacts?artifact_format=zipu0026artifact_type=archiveu0026expire_in=1+day"}
On minio’s side, it sees a bunch of s3.PutObjectParth, but at some point the following happens:
minio0 [REQUEST s3.AbortMultipartUpload] [22:36:55.085716620]
minio0 DELETE /com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25&X-Amz-Expires=15299&X-Amz-Date=20190806T223446Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=YP1A2T6K17SCL7T49R67%2F20190806%2Feu-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=164f6daff194c4dc59c9bac494fdd1ab1af9e646b6b93880c135629420294e36
minio0 Host: minio.company.com
minio0 Accept-Encoding: gzip
minio0 Content-Length: 0
minio0 User-Agent: Go-http-client/1.1
minio0 X-Forwarded-For: 192.168.4.165
minio0
minio0 [RESPONSE] [22:36:55.123376561]
minio0 204 No Content
minio0 Server: MinIO/RELEASE.2019-08-01T22-18-54Z
minio0 Vary: Origin
minio0 X-Amz-Request-Id: 15B8765DA51C02D2
minio0 X-Xss-Protection: 1; mode=block
minio0 Accept-Ranges: bytes
minio0 Content-Length: 0
minio0 Content-Security-Policy: block-all-mixed-content
minio0 minio0
minio2 [REQUEST s3.PutObjectPart] [22:35:56.800987758]
minio2 PUT /com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25&partNumber=4&X-Amz-Expires=15299&X-Amz-Date=20190806T223446Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]%2F20190806%2Feu-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]
minio2 Host: minio.company.com
minio2 User-Agent: Go-http-client/1.1
minio2 X-Forwarded-For: 192.168.4.165
minio2 X-Request-Id: GcKAAGVxUr2
minio2 Accept-Encoding: gzip
minio2 Content-Length: 104857600
minio2 <BODY>
minio2 [RESPONSE] [22:36:59.524203531]
minio2 404 Not Found
minio2 Content-Security-Policy: block-all-mixed-content
minio2 Content-Type: application/xml
minio2 Server: MinIO/RELEASE.2019-08-01T22-18-54Z
minio2 Vary: Origin
minio2 X-Amz-Request-Id: 15B876501313A0FE
minio2 X-Xss-Protection: 1; mode=block
minio2 Accept-Ranges: bytes
minio2 Content-Length: 575
minio2 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchUpload</Code><Message>The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.</Message><Key>tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4</Key><BucketName>com.company.gitlab.artifacts</BucketName><Resource>/com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4</Resource><RequestId>15B876501313A0FE</RequestId><HostId>b2cbe4bd-4e41-4276-b487-8eab69be92d0</HostId></Error>minio2 For some reason gitlab times out after a few parts uploaded, tells minio to abort the multipart upload (rendering the upload ID invalid) and then gitlab tries to upload some parts again with the same upload ID.
I have tried uploading to minio using s3 cli (even with parallel upload set to 1), minio’s CLI and curl and it all works fine, even on the same host as Gitlab is installed.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.1.4 Revision: 34d086f3e14 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.7 URL: https://gitlab.company.com HTTP Clone URL: https://gitlab.company.com/some-group/some-project.git SSH Clone URL: git@gitlab.company.com:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 9.3.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 91/2 ... yes 93/6 ... yes 95/8 ... yes 97/10 ... yes 103/13 ... yes 105/14 ... yes 106/17 ... yes 105/18 ... yes 103/19 ... yes 106/20 ... yes 103/21 ... yes 108/22 ... yes 78/23 ... yes 109/24 ... yes 111/25 ... yes 112/26 ... yes 114/28 ... yes 114/29 ... yes 115/30 ... yes 116/31 ... yes 117/32 ... yes 117/33 ... yes 118/34 ... yes 116/35 ... yes 111/37 ... yes 120/40 ... yes 120/42 ... yes 120/43 ... yes 108/44 ... yes 90/45 ... yes 91/46 ... yes 129/112 ... yes 129/113 ... yes 129/114 ... yes 129/115 ... yes 129/116 ... yes 129/117 ... yes 129/118 ... yes 129/119 ... yes 129/120 ... yes 129/121 ... yes 129/122 ... yes 129/123 ... yes 129/124 ... yes 129/125 ... yes 129/126 ... yes 129/127 ... yes 129/128 ... yes 129/129 ... yes 129/130 ... yes 129/131 ... yes 129/132 ... yes 129/133 ... yes 129/134 ... yes 129/135 ... yes 129/136 ... yes 129/137 ... yes 129/138 ... yes 129/139 ... yes 129/140 ... yes 129/141 ... yes 129/142 ... yes 129/143 ... yes 129/144 ... yes 129/145 ... yes 129/146 ... yes 129/147 ... yes 129/148 ... yes 129/149 ... yes 129/150 ... yes 129/151 ... yes 129/152 ... yes 129/153 ... yes 129/154 ... yes 129/155 ... yes 129/156 ... yes 129/157 ... yes 129/158 ... yes 129/159 ... yes 129/160 ... yes 129/161 ... yes 129/162 ... yes 129/163 ... yes 106/165 ... yes 130/166 ... yes 94/167 ... yes 114/168 ... yes 130/169 ... yes 102/170 ... yes 108/172 ... yes 130/173 ... yes 105/175 ... yes 111/177 ... yes 133/178 ... yes 133/179 ... yes 124/180 ... yes 129/181 ... yes 91/182 ... yes 117/183 ... yes 141/184 ... yes 141/185 ... yes 145/186 ... yes 129/187 ... yes 146/188 ... yes 116/190 ... yes 147/191 ... yes 146/192 ... yes 2/194 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.21.0 ? ... yes (2.21.0) Git user has default SSH configuration? ... yes Active users: ... 84
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Edited Aug 06, 2019 by Pier-Luc Gagnon