[4.11/Future] CLOUDSTACK-9782: Host HA and KVM HA provider #1960

yadvr · 2017-02-22T08:06:41Z

Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA

Signed-off-by: Abhinandan Prateek abhinandan.prateek@shapeblue.com
Signed-off-by: Rohit Yadav rohit.yadav@shapeblue.com

borisstoyanov · 2017-02-22T08:17:06Z

@blueorangutan package

blueorangutan · 2017-02-22T08:17:26Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2017-02-22T09:19:07Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-525

borisstoyanov · 2017-02-22T09:25:13Z

@blueorangutan test

blueorangutan · 2017-02-22T09:25:27Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

borisstoyanov · 2017-02-23T10:15:36Z

@rhtyd tests looks good, except this one:

ERROR: Tests default ha providers list
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/marvin/tests/smoke/test_hostha_simulator.py", line 199, in test_ha_list_providers
    response = self.apiclient.listHostHAProviders(cmd)[0]
TypeError: 'NoneType' object has no attribute '__getitem__'

Looks like when simulator is not deployed test would fail.

DaanHoogland

general remark:description fields do not say more then the name of the command or parameter they refer to. some improvement may be possible. e.g. "lists HA providers" may read "return a list of High Availability providers for a Hypervisor type"

DaanHoogland · 2017-02-27T13:43:46Z

.travis.yml

             smoke/test_dynamicroles
             smoke/test_global_settings
             smoke/test_guest_vlan_range
+             smoke/test_ha_for_host


DaanHoogland · 2017-02-27T14:02:11Z

api/src/org/apache/cloudstack/api/command/admin/ha/DisableHAForClusterCmd.java

+
+import javax.inject.Inject;
+
+@APICommand(name = DisableHAForClusterCmd.APINAME, description = "Disables HA cluster-wide",


naming: looking at the code below this actually is DisableHAForHostsInClusterCmd. I don't think it is a biggy but might lead to conflicts at some point

@DaanHoogland can you illustrate an example where we may hit such a conflict? This API actually disables the feature (framework etc) for cluster/zone etc. While the Host specific APIs work on the host.

If we decide to implement some kind of mirroring service for entire zones or clusters for high availability, these would then be disabled / enabled for a zone or cluster. In this case the enabling is done per host in the zone/cluster. That's why i consider the longer name I propose more appropriate. That said the shorter the name the better as long as it is unambiguous.

DaanHoogland · 2017-02-27T14:10:04Z

api/src/org/apache/cloudstack/api/command/admin/ha/DisableHAForZoneCmd.java

+
+import javax.inject.Inject;
+
+@APICommand(name = DisableHAForZoneCmd.APINAME, description = "Disables HA for a zone",


naming: see disable cluster cmd

DaanHoogland · 2017-02-27T14:10:57Z

api/src/org/apache/cloudstack/api/command/admin/ha/EnableHAForClusterCmd.java

+        responseObject = SuccessResponse.class,
+        requestHasSensitiveInfo = false, responseHasSensitiveInfo = false,
+        since = "4.11", authorized = {RoleType.Admin})
+public final class EnableHAForClusterCmd extends BaseAsyncCmd {


naming: see disable cmd

DaanHoogland · 2017-02-27T14:12:34Z

api/src/org/apache/cloudstack/api/command/admin/ha/EnableHAForZoneCmd.java

+        responseObject = SuccessResponse.class,
+        requestHasSensitiveInfo = false, responseHasSensitiveInfo = false,
+        since = "4.11", authorized = {RoleType.Admin})
+public final class EnableHAForZoneCmd extends BaseAsyncCmd {


naming: see disable cluster

DaanHoogland

please do not squash next time. both history is lost and review is very hard on such a big chunk of code. individual smaller commits make for easier reading.

DaanHoogland · 2017-02-27T14:46:03Z

api/src/org/apache/cloudstack/api/command/admin/host/PrepareForMaintenanceCmd.java

        }
    }
+
+    public void setHostId(final Long hostId) {


naming: consistency would dictate this be called setId()

I think the setters and getters are named per the variable, since the variable is hostId IDEs generate getHostId, setHostId named methods.

the field name is id, so i don't understand your reply

DaanHoogland · 2017-02-27T14:46:50Z

...e/cloudstack/api/command/admin/outofbandmanagement/ChangeOutOfBandManagementPasswordCmd.java

        CallContext.current().putContextParameter(Host.class, host.getUuid());

-        final OutOfBandManagementResponse response = outOfBandManagementService.changeOutOfBandManagementPassword(host, getPassword());
+        final OutOfBandManagementResponse response = outOfBandManagementService.changePassword(host, getPassword());


DaanHoogland · 2017-02-28T08:16:08Z

I went through the code and found no real issues. for any other reviewers I recommend reading the FS first. It is quit a big chunk but very neat.

LGTM

yadvr · 2017-02-28T08:19:52Z

Thanks @DaanHoogland wherever applicable I'll address the comments.

koushik-das · 2017-02-28T09:05:20Z

I have already raised some questions on dev@ on the need for a new HA framework when the existing HA framework can do all the things mentioned. The new framework only supports VM HA. If we see some concrete implementation of network/storage or any other type of resource HA which doesn't currently exist using the new framework then it would be much more easier to see the value add. I think more discussion is needed on this.

abhinandanprateek · 2017-02-28T10:12:12Z

@koushik-das The current framework is specifically implemented for Host HA with KVM HA as the initial implementation. This framework is supposed to replace the framework that is specifically written for VM-HA. @rhtyd

yadvr · 2017-02-28T10:30:22Z

@koushik-das I've shared a list of advantages of this work over existing framework on dev@ that explain why existing VM-HA framework cannot be used for host-ha implementation. If you've more questions or comments, we can discuss them here or on dev@.

koushik-das · 2017-02-28T11:48:43Z

@abhinandanprateek If you refer to the discussion on dev@ https://goo.gl/cU8RuX, @rhtyd proposed it as a generic HA framework for any resources (and not limited to VM). Now if it just a replacement of the existing VM-HA framework then we need to discuss the benefits it provides in more detail compared to the existing one. I have already made some specific comments on the justifications provided in favour of a new framework.

borisstoyanov · 2017-03-23T11:57:17Z

@rhtyd PRs 2003 and 2011 got merged, could you please rebase against master so it'll pick up the fixes. Once we package I'll continue with the verification on the physical hosts.

yadvr · 2017-03-26T08:21:46Z

@borisstoyanov copy that, done

borisstoyanov · 2017-03-27T06:37:53Z

@blueorangutan package

blueorangutan · 2017-03-27T06:38:26Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2017-03-27T07:01:11Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-600

borisstoyanov · 2017-03-27T07:36:53Z

@blueorangutan test

blueorangutan · 2017-03-27T07:37:26Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

borisstoyanov · 2017-03-28T06:31:09Z

@rhtyd there seems to be a conflict for this merge, I'm currently running tests and will keep you posted

borisstoyanov

I've executed the following tests and they all look good
HA Configuration:
Setting OOBM, enabling HA -> HA state Ineligible (provider is not set)
Setting Provider, Enabling HA -> HA state Ineligible (OOBM is not set)
Settign OOBM and HA Provider, enabling HA -> HA state is Available

When Agent is killed:
Passed: Transitioned to Suspect/Checking -> Degraded
Agent started:
Passed Transitioned to Suspect/Checking -> Available

When Host is irresponsible
Passed: Recovering a host
Passed: Fencing a Host
Passed: HA Enabled VMs were migrated when host was Fenced
Passed: HA Disabled VMs were in stopped state after the host is Fenced

I think we need to address 2 things:

SmokeTests are looking good the only failure related to these changes is because within the response there was no simulator provider.
merge conflict

DaanHoogland

LGTM overall. i had some comments on naming earlier only one is really. I re-itterated the odd one.

DaanHoogland · 2017-08-29T11:02:22Z

api/src/org/apache/cloudstack/api/command/admin/host/PrepareForMaintenanceCmd.java

    }
+
+    public void setHostId(final Long hostId) {
+        id = hostId;


this setter is still to be called setId(..) instead of setHostId(..), or should the field be renamed?

Fixed @DaanHoogland please see, turns out we dont' need this method.

yadvr · 2017-08-29T11:45:10Z

@DaanHoogland I've fixed the issue now, it was not used so removed the method

yadvr · 2017-08-29T11:46:11Z

Ignoring intermittent test failures and new test failures around isos, this PR has enough LGTMs and test results now, I'll merge this tomorrow and wait for any new comments and objections.

Fixed the issue now, closing the issue now.

borisstoyanov

LGTM

cloudmonger · 2017-08-30T00:37:21Z

ACS CI BVT Run

Sumarry:
Build Number 1164
Hypervisor xenserver
NetworkType Advanced
Passed=109
Failed=8
Skipped=40

Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0

Failed tests:

test_vm_snapshots.py
test_change_service_offering_for_vm_with_snapshots Failed
test_router_dnsservice.py
test_router_dns_guestipquery Failing since 2 runs
test_non_contigiousvlan.py
test_extendPhysicalNetworkVlan Failed
test_volumes.py
test_06_download_detached_volume Failed
test_routers_network_ops.py
test_01_isolate_network_FW_PF_default_routes_egress_true Failing since 28 runs
test_02_isolate_network_FW_PF_default_routes_egress_false Failing since 155 runs
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failing since 150 runs
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failing since 150 runs

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_disable_oobm_ha_state_ineligible
test_ha_kvm_host_degraded
test_ha_kvm_host_fencing
test_ha_kvm_host_recovering
test_hostha_configure_default_driver
test_hostha_enable_ha_when_host_disabled
test_hostha_enable_ha_when_host_disconected
test_hostha_enable_ha_when_host_in_maintenance
test_remove_ha_provider_not_possible
test_configure_ha_provider_invalid
test_configure_ha_provider_valid
test_ha_disable_feature_invalid
test_ha_enable_feature_invalid
test_ha_enabledisable_across_clusterzones
test_ha_list_providers
test_ha_multiple_mgmt_server_ownership
test_ha_verify_fsm_available
test_ha_verify_fsm_degraded
test_ha_verify_fsm_fenced
test_ha_verify_fsm_recovering
test_hostha_configure_default_driver
test_hostha_configure_invalid_provider
test_hostha_disable_feature_valid
test_hostha_enable_feature_valid
test_hostha_enable_feature_without_setting_provider
test_list_ha_for_host
test_list_ha_for_host_invalid
test_list_ha_for_host_valid
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_outofbandmanagement_nestedplugin.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_disk_offerings.py

yadvr · 2017-08-30T08:54:52Z

Travis is failing intermittently due to large log size, I've amended changes to reduce mvn build log and made changes in simulator integration test to pass for intermittent/edge cases.

Considering LGTMs and test results, I'll merge this as soon as Travis is green.

Host-HA offers investigation, fencing and recovery mechanisms for host that for any reason are malfunctioning. It uses Activity and Health checks to determine current host state based on which it may degrade a host or try to recover it. On failing to recover it, it may try to fence the host. The core feature is implemented in a hypervisor agnostic way, with two separate implementations of the driver/provider for Simulator and KVM hypervisors. The framework also allows for implementation of other hypervisor specific provider implementation in future. The Host-HA provider implementation for KVM hypervisor uses the out-of-band management sub-system to issue IPMI calls to reset (recover) or poweroff (fence) a host. The Host-HA provider implementation for Simulator provides a means of testing and validating the core framework implementation. Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com> Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Nested out-of-band management plugin to work with hosts that are VMs in a CloudStack env. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

- Removed three bg thread tasks, uses FSM event-trigger based scheduling - On successful recovery, kicks VM HA - Improves overall HA scheduling and task submission, lower DB access Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

- All tests should pass on KVM, Simulator - Add test cases covering FSM state transitions and actions Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

izenk · 2018-09-12T13:52:03Z

Is this functionality ready to use in 4.11.1? Is there any docs for it? (how to enable host HA, because clicking button in UI just shows error.., what should be configured and so on)

borisstoyanov · 2018-09-13T06:49:20Z

@izenk please find the FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
there's full description what hardware you'll need and how to set it up. You can mail the dev@ or user@ mailing lists if you need assistance.
And yes, the feature is fully functional in 4.11.1.

izenk · 2018-10-24T09:32:23Z

@borisstoyanov thanks, went through docs, but still unclear for me the exact configuration..
There is no description what should be done to enable and configure HA (Is it enough to switch it on in UI or not?).
What are requirements from point of feature support or software versions?
I found some text about:

A host must meet the following criteria to be deemed eligible for HA operations by the KVM HA host provider:
    The host must be a member of a cluster using the KVM hypervisor
    The host must have a power management status of ON or OFF
    The version of the KVM agent deployed on the host must support performing activity checks
    At least one volume attached to the VM(s) must support the activity check capability
    There should be at least one other host in the cluster.

But its not clear for me..
What does "The version of the KVM agent deployed on the host must support performing activity checks" mean? - There is no such term (activity checks) in KVM docs.
"At least one volume attached to the VM(s) must support the activity check capability" - hot it can be determined? In what terms? I terms of storage or kvm?

For example, my installation is CS 4.11.1 (KVM) + CEPH
I can enable HA for hosts - no errors appear, but in fact all hosts are in DEGRADED state.
And I cant understand what point I should start from to investigate the issue.
Are there any KVMHAProvider docs? (like what are checks? how they are executed and so on)

DennisKonrad · 2020-04-09T09:54:54Z

@rhtyd @borisstoyanov I'm trying to read up on the possibilities for high availability.

Is there any way to differentiate the "old" HA from the one that's implemented here? Im sorry to say that the design document only adds to the confusion.

I wasn't able to find any documentation that goes over the current state of HA in Cloudstack. Maybe you can point me in the right direction.

DaanHoogland · 2020-04-09T10:50:21Z

@DennisKonrad I don't considder me to be an expert, but in my recollection 'old' ha is just for VMs and not for hosts.

DennisKonrad · 2020-04-09T12:03:07Z

@DaanHoogland Ok, I understand the reasoning for the "new" ha. But where can I configure the old feature and where the new one.

Specificly which config keys affect one or the other and what does the "HA Enabled" on a host actually do?

Can I activate/deactivate only one or do I need to activate both? I cannot really find out if the features depend on one another?

yadvr changed the title ~~CLOUDSTACK-9782: Host HA and KVM HA provider~~ [4.11/Future] CLOUDSTACK-9782: Host HA and KVM HA provider Feb 22, 2017

yadvr closed this Feb 27, 2017

yadvr reopened this Feb 27, 2017

DaanHoogland reviewed Feb 27, 2017

View reviewed changes

DaanHoogland reviewed Feb 28, 2017

View reviewed changes

yadvr force-pushed the host-ha-master branch from 24ec0dd to a1e27c4 Compare February 28, 2017 11:32

yadvr force-pushed the host-ha-master branch from a1e27c4 to 8825e19 Compare March 1, 2017 06:33

yadvr force-pushed the host-ha-master branch from 8825e19 to 526bee9 Compare March 26, 2017 08:23

borisstoyanov reviewed Mar 28, 2017

View reviewed changes

yadvr force-pushed the host-ha-master branch from 526bee9 to 5af877b Compare March 31, 2017 15:15

yadvr removed status:needs-work status:waiting-for-reviewer labels Aug 29, 2017

DaanHoogland previously requested changes Aug 29, 2017

View reviewed changes

yadvr force-pushed the host-ha-master branch from a5f325a to 5ab8e24 Compare August 29, 2017 11:43

borisstoyanov approved these changes Aug 29, 2017

View reviewed changes

yadvr force-pushed the host-ha-master branch from 5ab8e24 to c558681 Compare August 30, 2017 08:52

yadvr force-pushed the host-ha-master branch from c558681 to 1f8eff8 Compare August 30, 2017 09:15

yadvr added 3 commits August 30, 2017 12:20

CLOUDSTACK-9782: Nested-oobm CloudStack plugin

23306ea

Nested out-of-band management plugin to work with hosts that are VMs in a CloudStack env. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

CLOUDSTACK-9782: Improve scheduling of jobs

10a6b94

- Removed three bg thread tasks, uses FSM event-trigger based scheduling - On successful recovery, kicks VM HA - Improves overall HA scheduling and task submission, lower DB access Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr force-pushed the host-ha-master branch 5 times, most recently from 5af039c to 2c3aad8 Compare August 30, 2017 14:06

CLOUDSTACK-9782: Improve host HA tests

c8d2bf1

- All tests should pass on KVM, Simulator - Add test cases covering FSM state transitions and actions Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr force-pushed the host-ha-master branch from 2c3aad8 to c8d2bf1 Compare August 30, 2017 15:01

yadvr merged commit f917ab6 into apache:master Aug 30, 2017

DaanHoogland removed the status:ready-for-merge label Dec 4, 2024


		import javax.inject.Inject;

		@APICommand(name = DisableHAForClusterCmd.APINAME, description = "Disables HA cluster-wide",


		import javax.inject.Inject;

		@APICommand(name = DisableHAForZoneCmd.APINAME, description = "Disables HA for a zone",

[4.11/Future] CLOUDSTACK-9782: Host HA and KVM HA provider #1960

[4.11/Future] CLOUDSTACK-9782: Host HA and KVM HA provider #1960

Uh oh!

Conversation

yadvr commented Feb 22, 2017

Uh oh!

borisstoyanov commented Feb 22, 2017

Uh oh!

blueorangutan commented Feb 22, 2017

Uh oh!

blueorangutan commented Feb 22, 2017

Uh oh!

borisstoyanov commented Feb 22, 2017

Uh oh!

blueorangutan commented Feb 22, 2017

Uh oh!

borisstoyanov commented Feb 23, 2017

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Feb 28, 2017

Uh oh!

yadvr commented Feb 28, 2017

Uh oh!

koushik-das commented Feb 28, 2017

Uh oh!

abhinandanprateek commented Feb 28, 2017

Uh oh!

yadvr commented Feb 28, 2017

Uh oh!

koushik-das commented Feb 28, 2017

Uh oh!

borisstoyanov commented Mar 23, 2017

Uh oh!

yadvr commented Mar 26, 2017

Uh oh!

borisstoyanov commented Mar 27, 2017

Uh oh!

blueorangutan commented Mar 27, 2017

Uh oh!

blueorangutan commented Mar 27, 2017

Uh oh!

borisstoyanov commented Mar 27, 2017

Uh oh!

blueorangutan commented Mar 27, 2017

Uh oh!

borisstoyanov commented Mar 28, 2017

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr Aug 29, 2017 •

edited

Loading

izenk commented Sep 12, 2018 •

edited

Loading