Error: NoValidHost: No valid host was found
You get below error message on the console while performing an overcloud deployment from the undercloud director
2018-08-14 06:42:59Z [overcloud.Controller.0.Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2018-08-14 06:42:59Z [overcloud.Controller.0]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2018-08-14 06:42:59Z [overcloud.Controller.0]: CREATE_FAILED ResourceInError: resources[0].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2018-08-14 06:42:59Z [overcloud.Controller]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources[0].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2018-08-14 06:42:59Z [overcloud.Controller]: CREATE_FAILED ResourceInError: resources.Controller.resources[0].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2018-08-14 06:43:00Z [overcloud]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Controller.resources[0].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
In the nova-conductor.log, you may find below messages
nova-conductor.log:2018-08-12 22:50:24.855 11372 WARNING nova.scheduler.utils [req-1bcec69e-4ad1-4460-bdd0-506e701dd81b a557b3e88e20446d921adbea6667e9ec d7f4df43c5db41f08572f582344b4e6f - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
nova-conductor.log:2018-08-12 22:50:26.995 11373 WARNING nova.scheduler.utils [req-0070ba9f-5eed-441d-a762-16bb9eaf9e8a a557b3e88e20446d921adbea6667e9ec d7f4df43c5db41f08572f582344b4e6f - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
In your nova-scheduler.log, below message may appear
nova-scheduler.log:2018-08-13 16:22:48.035 1475 INFO nova.filters [req-291df51f-aac0-42dc-bc72-84dfdb3e4a0c a557b3e88e20446d921adbea6667e9ec d7f4df43c5db41f085744b4e6f - - -] Filtering removed all hosts for the request with instance ID 'f6ec9c8e-4dd9-4995-92b7-b570bb06790f'. Filter results: ['RetryFilter: (start: 3, en, 'TripleOCapabilitiesFilter: (start: 3, end: 3)', 'ComputeCapabilitiesFilter: (start: 3, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'RamFilter: (s1, end: 1)', 'DiskFilter: (start: 1, end: 0)']
Analysis and Solution
Scenario 1
Make sure that enough nodes are in available state, not in maintenance mode and not already used by an existing instance.
[stack@undercloud-director ~]$ openstack baremetal node list +--------------------------------------+---------------------------+---------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+---------------------------+---------------+-------------+--------------------+-------------+ | 8ef9b862-9a4f-4961-813c-4b81be7c8e48 | overcloud-controller | None | power off | available | False | | 7995f1f2-4af7-4c5d-9099-fc928c4c73b3 | overcloud-compute.example | None | power off | available | False | | 7c84cdf2-c5b2-47fb-a741-30c025b54183 | overcloud-ceph.example | None | power off | available | False | +--------------------------------------+---------------------------+---------------+-------------+--------------------+-------------+
Scenario 2
Make sure the undercloud is able to connect to the overcloud hypervisors using the power credentials (e.g. ipmi_address, ipmi_username and ipmi_password)
[stack@undercloud-director ~]$ ipmitool -I lanplus -H 10.43.138.12 -L ADMINISTRATOR -p 6320 -U admin -R 3 -N 5 -P redhat power status Chassis Power is off [stack@undercloud-director ~]$ ipmitool -I lanplus -H 10.43.138.12 -L ADMINISTRATOR -p 6321 -U admin -R 3 -N 5 -P redhat power status Chassis Power is off [stack@undercloud-director ~]$ ipmitool -I lanplus -H 10.43.138.12 -L ADMINISTRATOR -p 6320 -U admin -R 3 -N 5 -P redhat power status Chassis Power is off
Many times it is observed that ipmitool fails to connect to the hypervisor hence the add node functionality breaks during overcloud deployment
Scenario 3
Make sure that you have Compute services running and enabled:
[stack@undercloud-director ~]$ openstack compute service list --service nova-compute +----+--------------+-----------------------------+------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +----+--------------+-----------------------------+------+---------+-------+----------------------------+ | 4 | nova-compute | undercloud-director.example | nova | enabled | up | 2018-08-14T12:51:23.000000 | +----+--------------+-----------------------------+------+---------+-------+----------------------------+
By default, a Compute service is disabled after 10 consecutive build failures on it. This is to ensure that new build requests are not routed to a broken Compute service. If it is the case, make sure to fix the source of the failures, then re-enable it:
$ openstack compute service set --enable <COMPUTE HOST> nova-compute
Scenario 4: (Most possible)
Make sure the node's properties matches completely with the assigned flavor's properties field. The node's property is populated at the introspection stage
For example my node's property
[stack@undercloud-director ~]$ openstack baremetal node show ece1651a-6adc-4826-9f77-5d47891c6c9b -c properties
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| properties | {u'memory_mb': u'10240', u'cpu_arch': u'x86_64', u'local_gb': u'49', u'cpus': u'4', u'capabilities': |
| | u'profile:control,cpu_aes:true,cpu_hugepages:true,boot_option:local'} |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
This node is assigned to "control" profile as highlighted, so check the properties of "control" flavor
[stack@undercloud-director ~]$ openstack flavor show control -c properties -f value capabilities:boot_option='local', capabilities:cpu_aes='true', capabilities:cpu_hugepages='true', capabilities:profile='control', cpu_arch='x86_64'
Here as you see both the property field match exactly.
How to update "node's" property content?
You can manually update the node's property content although I would not recommend that because these are automatically populated by the introspection. You should modify your flavor's property to match the node's content. But still we can update the node's property using the below command
$ ironic node-update overcloud-controller.example add properties/capabilities='profile:control,cpu_aes:true,cpu_hugepages:true,boot_option:local'
How to update flavor's property field?
To update the property section of a flavor use the below command
$ openstack flavor set --property "capabilities:profile"="control" --property "capabilities:cpu_aes"="true" --property "capabilities:cpu_hugepages"="true" --property "capabilities:boot_option"="local" control
https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html
I hope the article was useful.