[KVM] Direct download agnostic of the storage provider #3828

nvazquez · 2020-01-22T05:41:59Z

Description

This FR proposes the extension of the direct download support to primary storage pools using the most commonly used protocols/storage types:

NFS – already supported
Local storage
SharedMountpoint

Other storage providers support can be extended in future by interface implementation on the KVM storage provider depending on the storage adaptor.

The KVM storage processor is extended to achieve the direct template downloads in a storage agnostic way, including the supported storage providers implementations and allowing future implementations for different storage providers:

A new check of the available free space on the downloading host local storage is added before attempting to download a template. As on registration, a CheckUrlCommand is sent to a random KVM host, this host performs a HEAD command to the template URL. In case the server provides the remote file size of the template, this size is passed as part of the DirectDownloadCommand and used to check against the available free space.
The template file is downloaded to a scratch space in the local storage system within the host. This location can be explicitly set by the administrator via agent.properties file on the property: direct.download.temporary.download.location, and in case it is not set the default location var/lib/libvirt/images is used.
After the template file is downloaded to the scratch space location, it is extracted (or copied) to the primary storage pool selected by the CloudStack planner.
The implementation of the extract/copy method is extended to invoke the different StorageAdaptors implementations for different storage providers. These implementations include local storage, shared mount point and NFS in the scope of this PR.
After the template is extracted/copied to the destination storage pool, the downloaded file is removed from the scratch space location in the host.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

Ubuntu based management server
Test zone with local storage for user VMs enabled
2 x CentOS7 based KVM hosts
1 NFS storage pool
1 Shared mount point across the 2 hosts

nvazquez · 2020-01-22T05:51:32Z

@blueorangutan package

blueorangutan · 2020-01-22T05:51:49Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-22T06:11:42Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-646

DaanHoogland

please do not use a [WIP] tag in the title or [DO NOT MERGE]. you can create the PR as draft PR if you want to guard against merging.

DaanHoogland

code looks generally good, I have some doubts about the log messages. please review those @nvazquez .

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java

nvazquez · 2020-01-22T10:23:40Z

Thanks @DaanHoogland, comments are sorted out using your suggestions
@blueorangutan package

blueorangutan · 2020-01-22T10:23:50Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-22T10:43:06Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-648

nvazquez · 2020-01-22T10:46:36Z

@blueorangutan test

blueorangutan · 2020-01-22T10:46:50Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-01-22T21:46:26Z

Trillian test result (tid-805)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37512 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3828-t805-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 76 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_vm_from_direct_download_template_nfs_storage	`Error`	3.34	test_direct_download.py
ContextSuite context=TestDirectDownloadTemplates>:teardown	`Error`	2.23	test_direct_download.py
test_02_upload_direct_download_certificates	`Failure`	0.04	test_direct_download.py
test_02_deploy_vm_from_direct_download_template	`Error`	38.35	test_templates.py
test_03_deploy_vm_wrong_checksum	`Error`	10.61	test_templates.py

DaanHoogland · 2020-01-23T13:29:34Z

travis choked

nvazquez · 2020-01-29T04:41:43Z

@blueorangutan package

blueorangutan · 2020-01-29T04:42:19Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-29T05:02:25Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-680

nvazquez · 2020-01-29T05:03:27Z

@blueorangutan test

blueorangutan · 2020-01-29T05:04:19Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

nvazquez · 2020-03-04T15:05:04Z

@blueorangutan package

blueorangutan · 2020-03-04T15:06:08Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-03-04T15:33:54Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-999

blueorangutan · 2020-03-04T16:14:12Z

Trillian test result (tid-1183)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30769 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3828-t1183-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Smoke tests completed. 81 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

yadvr · 2020-03-05T06:59:34Z

@nvazquez there is one issue specific to simulator, address implement the mocked/override method:

[ERROR] /home/travis/build/apache/cloudstack/plugins/hypervisors/simulator/src/main/java/com/cloud/resource/SimulatorStorageProcessor.java:[56,7] error: SimulatorStorageProcessor is not abstract and does not override abstract method copyVolumeFromPrimaryToPrimary(CopyCommand) in StorageProcessor
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project cloud-plugin-hypervisor-simulator: Compilation failure
[ERROR] /home/travis/build/apache/cloudstack/plugins/hypervisors/simulator/src/main/java/com/cloud/resource/SimulatorStorageProcessor.java:[56,7] error: SimulatorStorageProcessor is not abstract and does not override abstract method copyVolumeFromPrimaryToPrimary(CopyCommand) in StorageProcessor
[ERROR] -> [Help 1]

@DaanHoogland @andrijapanicsb cc @borisstoyanov @PaulAngus this is ready to merge, please advice if any manual testing is complete?

borisstoyanov

LGTM

nvazquez · 2020-03-05T12:40:19Z

Fix for the simulator build added

nvazquez · 2020-03-05T13:47:54Z

@blueorangutan package

blueorangutan · 2020-03-05T13:48:05Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-03-05T14:14:12Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1006

nvazquez · 2020-03-05T16:26:37Z

@blueorangutan test

blueorangutan · 2020-03-05T16:27:06Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

andrijapanicsb · 2020-03-06T08:45:17Z

@blueorangutan test

blueorangutan · 2020-03-06T08:46:12Z

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-03-06T16:50:46Z

Trillian test result (tid-1206)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27787 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3828-t1206-kvm-centos7.zip
Smoke tests completed. 81 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

andrijapanicsb · 2020-03-06T18:56:39Z

3 x approvals, extensive manual testing, another review of the same and all regression testing passes.

Merging.

nvazquez added component:kvm type:improvement labels Jan 22, 2020

nvazquez added this to the 4.14.0.0 milestone Jan 22, 2020

nvazquez requested a review from DaanHoogland January 22, 2020 05:42

DaanHoogland closed this Jan 22, 2020

DaanHoogland reopened this Jan 22, 2020

DaanHoogland reviewed Jan 22, 2020

View reviewed changes

DaanHoogland approved these changes Jan 22, 2020

View reviewed changes

svenvogel added the type:enhancement label Jan 22, 2020

nvazquez changed the title ~~[WIP DO NOT MERGE] [KVM] Direct download agnostic of the storage provider~~ [KVM] Direct download agnostic of the storage provider Jan 22, 2020

DaanHoogland closed this Jan 23, 2020

DaanHoogland reopened this Jan 23, 2020

nvazquez force-pushed the direct-download-agnostic branch from e7d9b1d to d485183 Compare January 29, 2020 04:30

yadvr closed this Jan 29, 2020

yadvr reopened this Jan 29, 2020

nvazquez added 7 commits March 4, 2020 12:03

Fix ISO failure after retry

09c2636

Fix metalink filename mismatch error

ba202a3

Fix iso direct download

2923518

Fix for direct download ISOs on local storage and shared mount point

4f7a271

Last fix iso

f3d814d

Fix VM migration with ISO

2102864

Refactor volume migration to remove secondary storage intermediate

9034a8a

nvazquez force-pushed the direct-download-agnostic branch from 62d99cf to 9034a8a Compare March 4, 2020 15:04

nvazquez mentioned this pull request Mar 4, 2020

server: fix vm migration failed if iso is attached #3929

Closed

5 tasks

DaanHoogland requested review from andrijapanicsb and borisstoyanov March 5, 2020 08:23

borisstoyanov approved these changes Mar 5, 2020

View reviewed changes

Fix simulator issue

eef4c63

andrijapanicsb merged commit 73122fd into apache:master Mar 6, 2020

nvazquez deleted the direct-download-agnostic branch April 6, 2020 14:33

[KVM] Direct download agnostic of the storage provider #3828

[KVM] Direct download agnostic of the storage provider #3828

Uh oh!

Conversation

nvazquez commented Jan 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez commented Jan 22, 2020

Uh oh!

blueorangutan commented Jan 22, 2020

Uh oh!

blueorangutan commented Jan 22, 2020

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvazquez commented Jan 22, 2020

Uh oh!

blueorangutan commented Jan 22, 2020

Uh oh!

blueorangutan commented Jan 22, 2020

Uh oh!

nvazquez commented Jan 22, 2020

Uh oh!

blueorangutan commented Jan 22, 2020

Uh oh!

blueorangutan commented Jan 22, 2020

Uh oh!

DaanHoogland commented Jan 23, 2020

Uh oh!

nvazquez commented Jan 29, 2020

Uh oh!

blueorangutan commented Jan 29, 2020

Uh oh!

blueorangutan commented Jan 29, 2020

Uh oh!

nvazquez commented Jan 29, 2020

Uh oh!

blueorangutan commented Jan 29, 2020

Uh oh!

nvazquez commented Mar 4, 2020

Uh oh!

blueorangutan commented Mar 4, 2020

Uh oh!

blueorangutan commented Mar 4, 2020

Uh oh!

blueorangutan commented Mar 4, 2020

Uh oh!

yadvr commented Mar 5, 2020

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Mar 5, 2020

Uh oh!

nvazquez commented Mar 5, 2020

Uh oh!

blueorangutan commented Mar 5, 2020

Uh oh!

blueorangutan commented Mar 5, 2020

Uh oh!

nvazquez commented Mar 5, 2020

Uh oh!

blueorangutan commented Mar 5, 2020

Uh oh!

andrijapanicsb commented Mar 6, 2020

Uh oh!

blueorangutan commented Mar 6, 2020

Uh oh!

nvazquez commented Jan 22, 2020 •

edited

Loading