Error: Error waiting for instance to become ready: unexpected state 'ERROR', wanted target 'ACTIVE'

Job #39506 failed for 9d71a190:

changed milestone to %Build a CI/CD pipeline for XDMOD

added Sprint 23-26 label

assigned to @atlurie

This is the error we saw in nova service logs:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available.

No valid host was found. There are not enough hosts available. means the scheduler couldn’t place the instance anywhere. I think it may be because of memory being heavily overcommitted on the hosts

Try changing your flavor to m1.xlarge-ruffner. This will deploy to the Ruffner nodes that have almost no utilization.

I changed the TF_VAR_flavor var for rc/terraform-openstack

• dev pipeline from m1.medium to m1.medium-ruffner
• staging pipeline from m1.xlarge to m1.medium-ruffner
• production pipeline from m1.xlarge to m1.xlarge-ruffner

This issue was resolved after making the above change

closed

mentioned in issue ohpc-packer#38 (closed)