Documentation: Document Service/Agent/Executor Options via literalinclude of ConfigTemplate

[andresailer]

Configuration options for agents or services should be documented. Default values are at the moment both present in ConfigTemplate.cfg files, and in the source code of the agent or service.

• Completeness and consistency of options between sourcecode and ConfigTemplate for agents is checked with ConfigurationSystem/test/Test_agentOptions.py (for services see Documentation: Add test for Service or Executor option #4079 )

• Options should be described in the docstring of the source code module, like it is done for the WorkflowTaskAgent for example:
  

  DIRAC/TransformationSystem/Agent/WorkflowTaskAgent.py

  Lines 5 to 9 in 1e234f7

  	.. literalinclude:: ../ConfigTemplate.cfg
  	:start-after: ##BEGIN WorkflowTaskAgent
  	:end-before: ##END
  	:dedent: 2
  	:caption: WorkflowTaskAgent options

  
  

  DIRAC/TransformationSystem/ConfigTemplate.cfg

  Lines 67 to 70 in 1e234f7

  	##BEGIN WorkflowTaskAgent
  	WorkflowTaskAgent
  	{
  	# Transformation types to be taken into account by the agent

  
  ...
  

  DIRAC/TransformationSystem/ConfigTemplate.cfg

  Lines 126 to 127 in 1e234f7

  	}
  	##END

• Files can be treated individually to avoid overloading PRs too much.

• At the same time information from docs/source/AdministratorGuide/Configuration/ConfReference/Systems/ should be removed. Information only present in the Administrator guide should be moved to the module docstring

These files selected by looking for files that do no include literalinclude, some of these might not be agents or services, but utility files, in which case they should be ignored.

Agents

• AccountingSystem/Agent/NetworkAgent.py [v7r0] Docs for accounting agent/services #4564
• ConfigurationSystem/Agent/Bdii2CSAgent.py [v7r0] Minor fixes + doc on Bdii #4520
• ConfigurationSystem/Agent/GOCDB2CSAgent.py [v7r0] Minor fixes + doc on Bdii #4520
• ConfigurationSystem/Agent/VOMS2CSAgent.py
• FrameworkSystem/Agent/CAUpdateAgent.py
• FrameworkSystem/Agent/MyProxyRenewalAgent.py
• RequestManagementSystem/Agent/CleanReqDBAgent.py [v7r0] Configuration options docs for RMS #4573
• RequestManagementSystem/Agent/RequestExecutingAgent.py [v7r0] Configuration options docs for RMS #4573
• StorageManagementSystem/Agent/RequestFinalizationAgent.py
• StorageManagementSystem/Agent/RequestPreparationAgent.py
• StorageManagementSystem/Agent/StageMonitorAgent.py
• StorageManagementSystem/Agent/StageRequestAgent.py
• TransformationSystem/Agent/InputDataAgent.py
• TransformationSystem/Agent/MCExtensionAgent.py
• TransformationSystem/Agent/RequestTaskAgent.py [v6r21] RequestTaskAgent options; installTS tutorial options #4095
• TransformationSystem/Agent/TaskManagerAgentBase.py
• TransformationSystem/Agent/TransformationAgent.py
• TransformationSystem/Agent/TransformationPlugin.py
• TransformationSystem/Agent/ValidateOutputDataAgent.py
• WorkloadManagementSystem/Agent/JobAgent.py
• WorkloadManagementSystem/Agent/MultiProcessorSiteDirector.py
• WorkloadManagementSystem/Agent/OptimizerModule.py
• WorkloadManagementSystem/Agent/PilotStatusAgent.py
• WorkloadManagementSystem/Agent/StalledJobAgent.py
• WorkloadManagementSystem/Agent/StatesAccountingAgent.py

Services

• AccountingSystem/Service/DataStoreHandler.py [v7r0] Docs for accounting agent/services #4564
• AccountingSystem/Service/ReportGeneratorHandler.py [v7r0] Docs for accounting agent/services #4564
• ConfigurationSystem/Service/ConfigurationHandler.py [v7r0] Docs.ConfigurationSystem: move explanations from rst to code, add some doc for options #4563
• DataManagementSystem/Service/DataIntegrityHandler.py
• DataManagementSystem/Service/FileCatalogHandler.py
• DataManagementSystem/Service/FileCatalogProxyHandler.py
• DataManagementSystem/Service/FTS3ManagerHandler.py
• DataManagementSystem/Service/IRODSStorageElementHandler.py
• DataManagementSystem/Service/StorageElementProxyHandler.py
• FrameworkSystem/Service/BundleDeliveryHandler.py
• FrameworkSystem/Service/ComponentMonitoringHandler.py
• FrameworkSystem/Service/MonitoringHandler.py
• FrameworkSystem/Service/NotificationHandler.py
• FrameworkSystem/Service/PlottingHandler.py
• FrameworkSystem/Service/ProxyManagerHandler.py
• FrameworkSystem/Service/RabbitMQSyncHandler.py
• FrameworkSystem/Service/SecurityLoggingHandler.py
• FrameworkSystem/Service/SystemAdministratorHandler.py
• FrameworkSystem/Service/SystemLoggingHandler.py
• FrameworkSystem/Service/SystemLoggingReportHandler.py
• FrameworkSystem/Service/UserProfileManagerHandler.py
• MonitoringSystem/Service/MonitoringHandler.py
• ProductionSystem/Service/ProductionManagerHandler.py
• RequestManagementSystem/Service/ReqManagerHandler.py [v7r0] Configuration options docs for RMS #4573
• RequestManagementSystem/Service/ReqProxyHandler.py
• ResourceStatusSystem/Service/PublisherHandler.py
• ResourceStatusSystem/Service/ResourceManagementHandler.py
• ResourceStatusSystem/Service/ResourceStatusHandler.py
• StorageManagementSystem/Service/StorageManagerHandler.py
• TransformationSystem/Service/TransformationManagerHandler.py
• WorkloadManagementSystem/Service/JobManagerHandler.py
• WorkloadManagementSystem/Service/JobMonitoringHandler.py
• WorkloadManagementSystem/Service/JobStateUpdateHandler.py
• WorkloadManagementSystem/Service/MatcherHandler.py
• WorkloadManagementSystem/Service/OptimizationMindHandler.py
• WorkloadManagementSystem/Service/PilotManagerHandler.py
• WorkloadManagementSystem/Service/PilotsLoggingHandler.py
• WorkloadManagementSystem/Service/SandboxStoreHandler.py
• WorkloadManagementSystem/Service/WMSAdministratorHandler.py

[kanishk16]

I just want to know if I am on the right track....So, just let me if something is incorrect...

At the same time information from docs/source/AdministratorGuide/Configuration/ConfReference/Systems/ should be removed. Information only present in the Administrator guide should be moved to the module docstring

According to this, /docs/source/AdministratorGuide/Configuration/ConfReference/Systems/Transformation/Agents/transformationagent.rst is redundant and therefore needs to be removed after all the options are documented for this particular agent (and similar process needs to be followed for other agent or service files).

Although the confusion lies in the part that isn't the docstring for the specific agent to be documented according to the code...

[andresailer]

I am not entirely sure I understand you question.

At the moment this rst file is not redundant as there is information in here

https://dirac.readthedocs.io/en/latest/AdministratorGuide/Configuration/ConfReference/Systems/Transformation/Agents/transformationagent.html
This is part of the AdministratorGuide

That is not present here:

https://dirac.readthedocs.io/en/integration/CodeDocumentation/TransformationSystem/Agent/TransformationAgent.html
This is not part of the AdministratorGuide, but of the CodeDocumentation, maybe not very clearly explained.

Once all the options for the agent are described (via ConfigTemplate.cfg and literalinclude), and the information about related options is added to the TransformationAgent.py module docstring, then the rst file in the ConfReference should be removed, and for the time being a reference to the CodeDocumentation entry can be added in the ConfReference page

In the end all the information for a given agent can be found in one place, and not scattered through too many pages. And we only have to document agent options once in the ConfigTemplate

Although the confusion lies in the part that isn't the docstring for the specific agent to be documented according to the code...

I don't understand this phrase.

[kanishk16]

Now it's clear

[TaykYoku]

FrameworkSystem/Service/ProxyManagerHandler.py done with #4853

[fstagni]

I think most of them are done by now.