api,server,ui: granular resource limit management #8362

shwstppr · 2023-12-15T11:49:48Z

Description

Feature spec: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Granular+Resource+Limit+Management

Introduces the concept of tagged resource limits for granular resource limit management. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types,

Host limits
- user_vm
- cpu
- memory

Storage limits
- volume
- primary_storage

Following global settings can used to specify tags for which limit needs to be enforced,

Host: `resource.limit.host.tags`
Storage: `resource.limit.storage.tags`

Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI.

Enhances the use of templatetag for VM deployment and template creation

Adds option to list service/compute offerings that can be used with a given template. A new parameter named templateid has been added.

Adds option to list disk offering with suitability flag for a virtual machine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the response.

ToDo: Schema changes to be moved in 4.20 upgrade path

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Introduces the concept of tagged resource limits. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types, Host limits user_vm cpu memory Storage limits volume primary_storage Following global settings can used to specify tags for which limit needs to be enforced, Host: resource.limit.host.tags Storage: resource.limit.storage.tags Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI. Enhances use of templatetag for VM deployment and template creation Adds option to list disk offering with suitability flag for a virtualmachine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the reponse. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

codecov · 2023-12-15T11:57:50Z

Codecov Report

Attention: 570 lines in your changes are missing coverage. Please review.

Comparison is base (a31449b) 30.76% compared to head (711d9cb) 30.97%.

Files	Patch %	Lines
...in/java/com/cloud/server/ManagementServerImpl.java	29.62%	80 Missing and 15 partials ⚠️
.../cloud/resourcelimit/ResourceLimitManagerImpl.java	82.90%	35 Missing and 58 partials ⚠️
ui/src/components/view/ResourceLimitTab.vue	0.00%	41 Missing ⚠️
ui/src/components/view/ResourceCountUsage.vue	0.00%	35 Missing ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	71.42%	23 Missing and 5 partials ⚠️
...rc/main/java/com/cloud/user/DomainManagerImpl.java	0.00%	25 Missing ⚠️
...ain/java/com/cloud/api/query/QueryManagerImpl.java	41.02%	21 Missing and 2 partials ⚠️
ui/src/views/infra/Resources.vue	0.00%	22 Missing ⚠️
...n/java/com/cloud/capacity/dao/CapacityDaoImpl.java	36.36%	17 Missing and 4 partials ⚠️
.../agent/manager/allocator/impl/RandomAllocator.java	33.33%	17 Missing and 1 partial ⚠️
... and 32 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8362      +/-   ##
============================================
+ Coverage     30.76%   30.97%   +0.20%     
- Complexity    33076    33473     +397     
============================================
  Files          5353     5355       +2     
  Lines        374607   375708    +1101     
  Branches      54635    54908     +273     
============================================
+ Hits         115238   116363    +1125     
+ Misses       244099   243904     -195     
- Partials      15270    15441     +171

Flag	Coverage Δ
simulator-marvin-tests	`24.84% <56.71%> (+0.21%)`	⬆️
uitests	`4.36% <3.54%> (-0.02%)`	⬇️
unit-tests	`16.57% <38.12%> (+0.13%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

shwstppr · 2023-12-18T06:35:48Z

@blueorangutan package

blueorangutan · 2023-12-18T06:36:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-12-18T07:43:41Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8075

github-actions · 2023-12-18T08:02:16Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2023-12-19T18:54:54Z

@blueorangutan package

blueorangutan · 2023-12-19T18:56:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-12-19T20:02:23Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8092

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2023-12-27T08:30:09Z

@blueorangutan package

api/src/test/java/org/apache/cloudstack/api/command/user/template/UpdateTemplateCmdTest.java

shwstppr · 2023-12-27T09:26:20Z

@blueorangutan package

blueorangutan · 2023-12-27T09:38:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

vishesh92

clgtm

DaanHoogland · 2024-02-15T09:46:39Z

@shwstppr is this ready/can it go out of draft ?

shwstppr · 2024-02-15T10:30:41Z

@shwstppr is this ready/can it go out of draft ?

Waiting for @vladimirpetrov to finish some manual QA

vladimirpetrov

LGTM based on manual testing but let's wait for the final round of smoke tests before merging it. The following areas have been tested:

Migrating tagged resources
Global settings configuration
List resources by tag
Resource limits
Template tags

blueorangutan · 2024-02-15T18:19:51Z

[SF] Trillian test result (tid-9213)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46491 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8362-t9213-kvm-centos7.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

harikrishna-patnala

Code LGTM

DaanHoogland · 2024-02-16T13:30:50Z

@JoaoJandre @GutoVeronezi , this has been reviewed by several and has gone through our QA procedure (manual and automated testing. Do you guys hace any concerns with merging this?

GutoVeronezi · 2024-02-16T16:46:18Z

@JoaoJandre @GutoVeronezi , this has been reviewed by several and has gone through our QA procedure (manual and automated testing. Do you guys hace any concerns with merging this?

@DaanHoogland, my concerns are regarding the structure of the proposal, as I mentioned here. Regarding the code itself, I did not have time to review 7,000+ lines of changes; therefore, I do not have an opinion on it. I am -0.

shwstppr · 2024-02-17T08:16:36Z

@blueorangutan package

blueorangutan · 2024-02-17T08:18:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-17T09:24:13Z

Packaging result [SF]: ✖️ el7 ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 8700

shwstppr · 2024-02-17T09:30:10Z

@blueorangutan package

blueorangutan · 2024-02-17T09:32:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-17T10:38:07Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8701

shwstppr · 2024-02-17T10:50:05Z

@blueorangutan test

blueorangutan · 2024-02-17T10:52:03Z

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-02-17T23:36:56Z

[SF] Trillian test result (tid-9277)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44396 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8362-t9277-kvm-centos7.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

shwstppr · 2024-02-19T08:47:06Z

Merging based on approvals, manual QA and smoke test results

Feature spec: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Granular+Resource+Limit+Management Introduces the concept of tagged resource limits for granular resource limit management. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types, Host limits - user_vm - cpu - memory Storage limits - volume - primary_storage Following global settings can used to specify tags for which limit needs to be enforced, Host: `resource.limit.host.tags` Storage: `resource.limit.storage.tags` Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI. Enhances the use of templatetag for VM deployment and template creation Adds option to list service/compute offerings that can be used with a given template. A new parameter named templateid has been added. Adds option to list disk offering with suitability flag for a virtual machine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the response.

* feature: tagged resource limits apache/cloudstack#8362 Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * missing Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * update template tag details Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * instance deploy Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> * Update source/adminguide/usage.rst Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com> * Update source/adminguide/usage.rst Co-authored-by: Vishesh <vishesh92@gmail.com> --------- Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com> Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com> Co-authored-by: Vishesh <vishesh92@gmail.com>

boring-cyborg bot added component:api component:compute component:integration component:orchestration labels Dec 15, 2023

shwstppr added this to the 4.20.0.0 milestone Dec 15, 2023

github-actions bot added the status:has-conflicts label Dec 18, 2023

shwstppr added 3 commits December 19, 2023 16:01

Merge remote-tracking branch 'apache/main' into add-resource-limit-tag

bb4b1cf

add unit tests

533be6e

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix eof

a1806bb

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions bot removed the status:has-conflicts label Dec 19, 2023

shwstppr added 3 commits December 19, 2023 16:11

fix import

f423c5b

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

596ac91

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

build fix

fa15616

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added 4 commits December 27, 2023 12:13

ui: fix templatetag required

2f6df91

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

api,server; allow template tag update

300e0be

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

license

06ef971

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

eof

28ce02f

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

DaanHoogland reviewed Dec 27, 2023

View reviewed changes

api/src/test/java/org/apache/cloudstack/api/command/user/template/UpdateTemplateCmdTest.java Show resolved Hide resolved

vishesh92 approved these changes Feb 15, 2024

View reviewed changes

vladimirpetrov approved these changes Feb 15, 2024

View reviewed changes

shwstppr marked this pull request as ready for review February 15, 2024 10:55

harikrishna-patnala approved these changes Feb 16, 2024

View reviewed changes

DaanHoogland approved these changes Feb 16, 2024

View reviewed changes

vishesh92 mentioned this pull request Feb 19, 2024

Fix resource count discrepancies #8302

Merged

13 tasks

shwstppr merged commit 592038a into apache:main Feb 19, 2024

DaanHoogland deleted the add-resource-limit-tag branch February 19, 2024 08:58

shwstppr mentioned this pull request Feb 29, 2024

engine-schema,test: fix domain reservation searchbuilder, test fixes #8725

Merged

13 tasks

weizhouapache mentioned this pull request Nov 4, 2024

api,agent,server,engine-schema: scalability improvements #9840

Merged

14 tasks

bernardodemarco mentioned this pull request Feb 19, 2025

No method to apply quota on storage tags #10382

Closed

api,server,ui: granular resource limit management #8362

api,server,ui: granular resource limit management #8362

Uh oh!

Conversation

shwstppr commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shwstppr commented Dec 18, 2023

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

blueorangutan commented Dec 18, 2023

Uh oh!

github-actions bot commented Dec 18, 2023

Uh oh!

shwstppr commented Dec 19, 2023

Uh oh!

blueorangutan commented Dec 19, 2023

Uh oh!

blueorangutan commented Dec 19, 2023

Uh oh!

shwstppr commented Dec 27, 2023

Uh oh!

Uh oh!

shwstppr commented Dec 27, 2023

Uh oh!

blueorangutan commented Dec 27, 2023

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Feb 15, 2024

Uh oh!

shwstppr commented Feb 15, 2024

Uh oh!

vladimirpetrov left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Feb 15, 2024

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Feb 16, 2024

Uh oh!

GutoVeronezi commented Feb 16, 2024

Uh oh!

shwstppr commented Feb 17, 2024

Uh oh!

blueorangutan commented Feb 17, 2024

Uh oh!

blueorangutan commented Feb 17, 2024

Uh oh!

shwstppr commented Feb 17, 2024

Uh oh!

blueorangutan commented Feb 17, 2024

Uh oh!

blueorangutan commented Feb 17, 2024

Uh oh!

shwstppr commented Feb 17, 2024

Uh oh!

blueorangutan commented Feb 17, 2024

Uh oh!

blueorangutan commented Feb 17, 2024

Uh oh!

shwstppr commented Feb 19, 2024

Uh oh!

Reviewers

shwstppr commented Dec 15, 2023 •

edited

Loading

codecov bot commented Dec 15, 2023 •

edited

Loading