[WIP][v6r20] Added StorageElement.getOccupancy by fstagni · Pull Request #3572 · DIRACGrid/DIRAC

fstagni · 2017-12-08T16:27:51Z

This PR adds a StorageElement.getOccupancy() call, and then use it inside the RSS.
This is not supposed to be a complete work. It just adds the general interface, which should be applied for all type of SEs. Some "FIXME" comment are there and can be only be resolved later.

BEGINRELEASENOTES

*Resources
NEW: Added StorageElement.getOccupancy() method for DIP and GFAL2_SMR2 SE types

*RSS
CHANGE: Using StorageElement.getOccupancy()

ENDRELEASENOTES

coveralls · 2017-12-08T16:39:44Z

Coverage decreased (-0.01%) to 19.862% when pulling a2fdd92 on fstagni:spaceTokenUsage into e78e42e on DIRACGrid:integration.

coveralls · 2017-12-11T10:33:42Z

Coverage decreased (-0.003%) to 19.869% when pulling 5965036 on fstagni:spaceTokenUsage into e78e42e on DIRACGrid:integration.

chaen · 2017-12-11T14:32:14Z

+      spaceTokenEndpoint, spaceToken = params['Value']
+
+    se = StorageElement(self.args['name'])
+    occupancyResult = se.getOccupancy


getOccupancy()?

chaen · 2017-12-11T14:39:50Z

+        :returns: S_OK/S_ERROR dictionary
+    """
+    #FIXME: put an implementation that just gets a file and check its content
+    return S_ERROR("Storage.occupancy: implement me!")


StorageBase.getOccupancy

Well all the other return codes in this module contains Storage.method:so I just left it like the others.

chaen · 2017-12-11T14:42:37Z


    return S_OK( protocolsList )
+
+  def getOccupancy(self):


No no, there is no lcg_utils anymore.
I will provide you with an implementation for that

I am not sure what exactly you are expecting.

Is this ok ?

In [14]: ctx.getxattr('srm://srm-eoslhcb.cern.ch', 'spacetoken.token?LHCb-EOS') Out[14]: '{ "spacetoken": "LHCb-EOS", "owner": "lhcb", "totalsize": -1, "unusedsize": 429499145907438, "usedsize": 0, "guaranteedsize": 5000000000000000, "lifetimeleft": -1, "retention": "REPLICA", "accesslatency": "ONLINE" }'

And what is ctx? Trying on lxplus:

In [47]: import ctx --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-47-56f9a9ccee81> in <module>() ----> 1 import ctx ImportError: No module named ctx

Oh is it just a context?

yes. But as I said if you want I can implement it, I was just asking if these are the info you were looking for.

Alright so let's do that this is merged, and then it's left to be correctly implemented. What's is added here right now (lcg_util) is what is used today to calculate the space left for space tokens. It works properly so I am not sure why you say that "there's no lcg_utils anymore". The call you posted seems showing the same result, but honestly I didn't properly check, and also it should be seen if the results are consistent among several type of SEs.

lcg_util is gfal version 1, it is deprecated and should not be used, that's what I meant.
OK, as soon as it is merged, I will implement the correct version

chaen · 2017-12-11T14:47:20Z

+    """ Gets the DIPStorage occupancy info
+    """
+    rpc = RPCClient(self.getURLBase(), timeout=120)
+    free = rpc.getFreeDiskSpace("/")


What is the parameter ?
Shouldn't the StorageElementHandler answer with the space free on the partition where it stores its data ?

It's the path, basically it will export the result of st = os.statvfs(path)

Then I would say it is incorrect. Take LogSE as an example: it has a special partition

Yes I see, and I agree with you. I just discussed it with Andrei, he will implement this properly.

coveralls · 2017-12-12T09:22:17Z

Coverage increased (+0.004%) to 19.878% when pulling acb03a6 on fstagni:spaceTokenUsage into e8a7701 on DIRACGrid:integration.

…Usage * origin/spaceTokenUsage: (26 commits) Conflicts resolved Resolved conflicts v6r19p9 v6r19p9 notes testing on protocol using base types allow also space-separated arguments allow giving list of requests in a file using URLBase for everything but SRM2 case when an SE is at no sites added some doc added protection just style use resolveSEGroup avoid instantiating all local SEs marked getStorageElementOptions as deprecated style also considering CEs, not only SEs style docs and pylints ...

coveralls · 2017-12-12T09:49:44Z

Coverage decreased (-0.003%) to 19.871% when pulling fbd0783 on fstagni:spaceTokenUsage into e8a7701 on DIRACGrid:integration.

coveralls · 2017-12-13T17:30:06Z

Coverage decreased (-0.008%) to 19.868% when pulling b11fba1 on fstagni:spaceTokenUsage into 2855339 on DIRACGrid:integration.

* upstream/integration: (25 commits) v6r19p10, v6r20-pre10 v6r19p10 v6r20-pre10 wrap lines better fix isinstance Added warning log message in case of exception log the delay only set delay once again logging fix logging delay execution if no active replicas found delay execution by 1 h if no active replicas found print NotBefore use defaultdict Catch exception when closing closed connection few more logging changes less logging in DM transfers limit logging fix case when no site is found for SE don't try and get SE metadata if replica is inactive ...

* upstream/integration: styled bug fix (double return) DMS: fix bad merge of ReplicateAndRegister

coveralls · 2017-12-19T17:09:36Z

Coverage increased (+0.01%) to 19.88% when pulling ce2ed75 on fstagni:spaceTokenUsage into e58ba72 on DIRACGrid:integration.

coveralls · 2017-12-20T15:15:48Z

Coverage increased (+0.008%) to 19.876% when pulling 2c249d7 on fstagni:spaceTokenUsage into e58ba72 on DIRACGrid:integration.

coveralls · 2017-12-20T15:32:06Z

Coverage increased (+0.01%) to 19.88% when pulling f3aa637 on fstagni:spaceTokenUsage into e58ba72 on DIRACGrid:integration.

coveralls · 2018-01-08T15:36:20Z

Coverage increased (+0.006%) to 19.875% when pulling 397c2e6 on fstagni:spaceTokenUsage into e58ba72 on DIRACGrid:integration.

* upstream/integration: v6r19p11 v6r20-pre10 notes v6r19p11 notes using python modules always removing the sandboxes of deleted jobs just style just style added doc about shifterProxy 64 chars instead of 32 new option --Force to reset even non-failed requests + accept job IDs from a file new option --Maximum to limit the number of jobs selected + autopep8 autopep8 reset the NotBefore field for all requests when --Reset is used make pylint happier with data members RMS: add missing JOIN in the web summary query style fixed exception handling

fstagni · 2018-01-15T15:49:13Z

I resolved the conflicts, please review.

coveralls · 2018-01-15T15:59:05Z

Coverage increased (+0.02%) to 19.857% when pulling 5f15112 on fstagni:spaceTokenUsage into 101301c on DIRACGrid:integration.

chaen · 2018-01-19T09:41:55Z

  types_getFreeDiskSpace = [basestring]
-  def export_getFreeDiskSpace( self, path, size = 'TB' ):
+
+  def export_getFreeDiskSpace(self, path):


I will do it because I am working against your branch but...

The docstring needs updating

In all the plugins, you wrote MB or Should be MB. Shall we take the opportunity and make it MB or TB everywhere ?
If up to now srm was returning TB, we shall do TB, because probably many plots rely on that

fstagni · 2018-01-22T09:24:12Z

Replaced by #3601

fstagni added 6 commits December 7, 2017 15:51

added spaceTokenUsage scripts

111b871

moved to functions

ede4893

scaffolding for getOccupancy calls

30c7513

few fixes

23c044d

style

7146c26

re-migrated

a2fdd92

fstagni requested a review from atsareg as a code owner December 8, 2017 16:27

fstagni added 3 commits December 11, 2017 11:11

just style

884b9f9

using StorageElement.getOccupancy()

3acc887

just style

5965036

fstagni changed the title ~~[WIP] [v6r20] Added StorageElement.getOccupancy~~ [v6r20] Added StorageElement.getOccupancy Dec 11, 2017

chaen reviewed Dec 11, 2017

View reviewed changes

fstagni added 4 commits December 11, 2017 16:33

bug fix

bd97a88

Merge branch 'integration' into spaceTokenUsage

0897a84

Merge branch 'integration' into spaceTokenUsage

3528faf

Merge branch 'integration' into spaceTokenUsage

acb03a6

fstagni added 2 commits December 12, 2017 10:33

autopep8 style

fbd0783

fstagni changed the title ~~[v6r20] Added StorageElement.getOccupancy~~ [WIP][v6r20] Added StorageElement.getOccupancy Dec 13, 2017

Merge branch 'integration' into spaceTokenUsage

b11fba1

fstagni added 2 commits December 19, 2017 10:01

FreeDiskSpace replaces SpaceTokenOccupancy

87f55a0

fstagni added 3 commits December 19, 2017 17:33

styling

739710a

Merge remote-tracking branch 'upstream/integration' into spaceTokenUsage

871b345

* upstream/integration: styled bug fix (double return) DMS: fix bad merge of ReplicateAndRegister

fix test

ce2ed75

using unit inside the command

2c249d7

fstagni changed the title ~~[WIP][v6r20] Added StorageElement.getOccupancy~~ [v6r20] Added StorageElement.getOccupancy Dec 20, 2017

bug fix plus style

f3aa637

restored function for extensibility

397c2e6

fstagni changed the title ~~[v6r20] Added StorageElement.getOccupancy~~ [WIP][v6r20] Added StorageElement.getOccupancy Jan 18, 2018

chaen reviewed Jan 19, 2018

View reviewed changes

chaen mentioned this pull request Jan 19, 2018

[WIP][v6r20] Space token usage #3601

Closed

fstagni closed this Jan 22, 2018

Uh oh!

Conversation

fstagni commented Dec 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Dec 8, 2017

Uh oh!

coveralls commented Dec 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Dec 12, 2017

Uh oh!

coveralls commented Dec 12, 2017

Uh oh!

coveralls commented Dec 13, 2017

Uh oh!

coveralls commented Dec 19, 2017

Uh oh!

coveralls commented Dec 20, 2017

Uh oh!

coveralls commented Dec 20, 2017

Uh oh!

coveralls commented Jan 8, 2018

Uh oh!

fstagni commented Jan 15, 2018

Uh oh!

coveralls commented Jan 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fstagni commented Jan 22, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fstagni commented Dec 8, 2017 •

edited

Loading