Incorrect work of getFreeMemory() in LibvirtStartCommandWrapper.java

[DmB991]

ISSUE TYPE

• Bug Report

COMPONENT NAME

LibvirtStartCommandWrapper.java

CLOUDSTACK VERSION

4.10 +

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

SUMMARY

I found error in getFreeMemory() of https://github.com/apache/cloudstack/blob/4.11/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L170
When I create a new VM we take a list of all VMs from libvirt via getFreeMemory(). After that we iterate by this list. If amount of VMs so large(or libvirt response longer than always) this may take some time. At that time one or more VMs, which has "Running" state can start stopping by another process(job) and not displayed in libvirt anymore. So it bring to exception with "libvirt.libvirtError: Domain not found: no domain with matching id 157124." message and we move on catch with wrong error message(Failed to get free memory).

STEPS TO REPRODUCE

1. Start one proces which will create VMs.
2. When we have 100(create more if not reproduce) VMs on host should start another process which will destroy VMs.
3. Still some time we will see error message "Failed to get free memory" in logs with free memory resources on host.

management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ

EXPECTED RESULTS

Correct creation of VM without exceptions.

ACTUAL RESULTS

INFO message "Failed to get free memory" and ERROR with InsufficientCapacityEception.

[yadvr]

Can you try/test against 4.11.3.0 or master @DmB991 ?

[DmB991]

@rhtyd yes, reproduced on version 4.11.3.0
https://yadi.sk/d/8ISgwyGI2D7e_A
logid:0d7c4e6c

[wido]

@GabrielBrascher Could you look into this? Seems like a valid scenario. We probably need a try/catch around the point where we query the domain from libvirt.

[GabrielBrascher]

@wido @rhtyd @DmB991
Commit 9a02190 added the getFreeMemory(conn,libvirtComputingResource) method. However, it was latter removed by commit 65f31f1 on Aug 8, 2018; therefore, it has already been fixed. Upgrading to 4.11.3.0 or 4.12.0.0 should fix it :)

[DmB991]

@GabrielBrascher After upgrading to 4.11.3.0 the problem(INFO message "Failed to get free memory" and ERROR with InsufficientCapacityEception) still appear when hosts has free resources (see the log 2 comments above)

[GabrielBrascher]

Thanks, I will check the second log, missed that message.

[DmB991]

@GabrielBrascher Do you have any results of checking logs?

[GabrielBrascher]

Sorry for the late reply, @DmB991.

Just updating that I have been trying to reproduce this error on KVM nodes but so far I had no success.

For instance, I deployed more than 150 VMs on a node (via API call s done by a script), reaching its CPU limits, but could not reproduce the issue raised here. Also, I was able to list those VMs and delete all of them at once without any problem.

Have you been able to upgrade from 4.11.3 to 4.13.0.0 or 4.14.0.0?

[DmB991]

@GabrielBrascher Are you deleting(first 100 VMs) and creating new VMs(after 100 VMs exists) at the same time?
I return on 4.11.2, it works perfect for me, except this and #3513 bug. So I recreate VMs(via script) when VM is fallen down.

[GabrielBrascher]

@DmB991 I did not execute both, VMs creation and deletion, at the same time, I will re-check considering your use case.

[yadvr]

cc @Spaceman1984

[yadvr]

cc @Spaceman1984 can you check and see if we need to (quick) fix this?

[GabrielBrascher]

@DmB991 @rhtyd @Spaceman1984 I gave another shot trying to see any loose point but I still failed to reproduce it.

Tests were done via CloudMonkey:

1. destroy +100 VMs, and
2. deploy another +100 VMs at the same time

Additionally, the piece of code that was failing with failed to get free memory is not on codebase anymore. LibvirtStartCommandWrapper class has been refactored at PR #2766 by @wido. I don`t think that the issue would happen again if both management and agent nodes get upgraded to recent CloudStack versions.