diff --git a/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/PilotsLoggingDB.png b/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/PilotsLoggingDB.png deleted file mode 100644 index ee54b10f936..00000000000 Binary files a/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/PilotsLoggingDB.png and /dev/null differ diff --git a/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/PilotsLoggingDiagram.png b/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/PilotsLoggingDiagram.png deleted file mode 100644 index 750f67d8c2a..00000000000 Binary files a/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/PilotsLoggingDiagram.png and /dev/null differ diff --git a/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/index.rst b/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/index.rst deleted file mode 100644 index f2efac53728..00000000000 --- a/docs/source/AdministratorGuide/Systems/WorkloadManagement/PilotsLogging/index.rst +++ /dev/null @@ -1,74 +0,0 @@ -.. contents:: Table of contents - :depth: 3 - -======================================================================================= - -======================================================================================= - -============================== -Pilots Logging system overview -============================== - -Pilots Loggins system is designed to allow logging of pilot state on every stage of lifecycle, including before installing -DIRAC client and starting pilot process. - -Each logging entry includes: - -- current status of the Pilot - has to be one of predefined list of possible states, -- additional information about status, -- timestamp of logging the status - if there is no timestamp of actual event provided, time of adding entry to database will be used, -- source of the logging message to distinguish updates from Pilot itself and other services. - -.. image:: PilotsLoggingDiagram.png - :alt: PilotsLogging system - :align: center - - -Server side -================================ - -Server elements of Pilots Logging system is build using five elements: - -- message queue (RabbitMQ) server, -- message queue consumer, -- DIRAC Client, -- DIRAC Service, -- database. - -Message queue --------------------------------- - -Message works as a interface between Pilot and Pilots Logging service. Pilot puts status related messages into queue then -messages are handled by message queue consumer. - -Message queue consumer --------------------------------- - -Consumer registers itself into message queue. When new messages arrive they are handled by callback function. In consumer -messages are processed and passed to DIRAC Service using DIRAC Client. - -DIRAC Client --------------------------------- - -Client handles RPC communication with Service. This is 'thin-client', all business logic is in Service. - -DIRAC Service --------------------------------- - -Service exports functions to be called by Clients. It handles all operations on databases. All server side logic of -Pilots Logging system is defined here. Two databases are accessed to gather all required information. - -Database --------------------------------- - -Database class handles operation on the database. Object-relational mapping is done using SQLAlchemy. Single table stores -record for every status reported by Pilot: - -.. image:: PilotsLoggingDB.png - :alt: PilotsLogging database schema - :align: center - -Pilot side -================================ - -TBD diff --git a/docs/source/AdministratorGuide/Systems/WorkloadManagement/index.rst b/docs/source/AdministratorGuide/Systems/WorkloadManagement/index.rst index fb06f7cd52e..2b0ab2319a7 100644 --- a/docs/source/AdministratorGuide/Systems/WorkloadManagement/index.rst +++ b/docs/source/AdministratorGuide/Systems/WorkloadManagement/index.rst @@ -83,7 +83,6 @@ The following sections add some detail for the WMS systems. architecture Pilots/index Pilots/Pilots3 - PilotsLogging/index Jobs/index JobsPriorities JobsMatching diff --git a/docs/source/DeveloperGuide/Systems/WorkloadManagement/index.rst b/docs/source/DeveloperGuide/Systems/WorkloadManagement/index.rst index 40e0923f26a..8a961a2ec11 100644 --- a/docs/source/DeveloperGuide/Systems/WorkloadManagement/index.rst +++ b/docs/source/DeveloperGuide/Systems/WorkloadManagement/index.rst @@ -78,7 +78,6 @@ It is based on layered architecture and is based on DIRAC architecture: * JobStateUpdateHandler * MatcherHandler * OptimizationMindHandler - * PilotsLoggingHandler * SandboxStoreHandler * WMSAdministratorHandler * WMSUtilities @@ -97,11 +96,6 @@ It is based on layered architecture and is based on DIRAC architecture: This database keeps track of all the submitted grid pilot jobs. It also registers the mapping of the DIRAC jobs to the pilot agents. - * PilotsLoggingDB: - PilotsLoggingDB class is a front-end to the Pilots Logging Database. - This database keeps track of all the submitted grid pilot jobs. - It also registers the mapping of the DIRAC jobs to the pilot agents. - * SandboxMetadataDB SandboxMetadataDB class is a front-end to the metadata for sandboxes. diff --git a/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py b/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py index c0b447a391a..eedfca8980b 100644 --- a/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py +++ b/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py @@ -984,11 +984,6 @@ def _getPilotOptions(self, queue, **kwargs): else: self.log.info("DIRAC project will be installed by pilots") - # Pilot Logging defined? - pilotLogging = opsHelper.getValue("/Services/JobMonitoring/usePilotsLoggingFlag", False) - if pilotLogging: - pilotOptions.append("-z ") - pilotOptions.append("--pythonVersion=3") # Debug diff --git a/src/DIRAC/WorkloadManagementSystem/Client/PilotsLoggingClient.py b/src/DIRAC/WorkloadManagementSystem/Client/PilotsLoggingClient.py deleted file mode 100644 index 574b8a23abb..00000000000 --- a/src/DIRAC/WorkloadManagementSystem/Client/PilotsLoggingClient.py +++ /dev/null @@ -1,45 +0,0 @@ -""" Class that contains client access to the PilotsLogging handler. """ - -from DIRAC.Core.Base.Client import Client, createClient - - -@createClient("WorkloadManagement/PilotsLogging") -class PilotsLoggingClient(Client): - """Implementation of interface of Pilots Logging service. Client class should be used to communicate - with PilotsLogging Service - """ - - def __init__(self, **kwargs): - super().__init__(**kwargs) - self.setServer("WorkloadManagement/PilotsLogging") - - def addPilotsLogging(self, pilotUUID, timestamp, source, phase, status, messageContent): - """ - Add new Pilots Logging entry - - :param pilotUUID: Pilot reference - :param status: Pilot status - :param minorStatus: Additional status information - :param timestamp: Date and time of status event - :param source: Source of status information - """ - - return self._getRPC().addPilotsLogging(pilotUUID, timestamp, source, phase, status, messageContent) - - def deletePilotsLogging(self, pilotUUID): - """ - Delete all Logging entries for Pilot - - :param pilotUUID: Pilot reference - """ - - return self._getRPC().deletePilotsLogging(pilotUUID) - - def getPilotsLogging(self, pilotUUID): - """ - Get all Logging entries for Pilot - - :param pilotUUID: Pilot reference - """ - - return self._getRPC().getPilotsLogging(pilotUUID) diff --git a/src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg b/src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg index 2d39398123e..d72948d2967 100644 --- a/src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg +++ b/src/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg @@ -102,7 +102,7 @@ Services } } ##END - #Parameters of the Pilots service + #Parameters of the PilotManager service PilotManager { Port = 9171 @@ -111,20 +111,6 @@ Services Default = authenticated } } - #Parameters of the PilotsLogging service - PilotsLogging - { - Port = 9146 - Authorization - { - Default = Operator - getPilotsLogging = authenticated - addPilotsLogging = Operator - deletePilotsLogging = Operator - } - Enable = No - PilotsLoggingQueue = serviceURL::QueueType::QueueName - } ##BEGIN SandboxStore SandboxStore { diff --git a/src/DIRAC/WorkloadManagementSystem/DB/PilotsLoggingDB.py b/src/DIRAC/WorkloadManagementSystem/DB/PilotsLoggingDB.py deleted file mode 100644 index 3635ac127d7..00000000000 --- a/src/DIRAC/WorkloadManagementSystem/DB/PilotsLoggingDB.py +++ /dev/null @@ -1,190 +0,0 @@ -""" PilotsLoggingDB class is a front-end to the Pilots Logging Database. - This database keeps track of all the submitted grid pilot jobs. - It also registers the mapping of the DIRAC jobs to the pilot - agents. - - Available methods are: - - addPilotsLogging() - getPilotsLogging() - deletePilotsLoggin() - -""" - -from sqlalchemy.orm import sessionmaker, scoped_session -from sqlalchemy.engine.reflection import Inspector -from sqlalchemy import create_engine, Column, MetaData, Integer, String -from sqlalchemy.ext.declarative import declarative_base -from sqlalchemy.exc import SQLAlchemyError - -from DIRAC import gLogger, S_OK, S_ERROR -from DIRAC.Core.Utilities import DErrno -from DIRAC.ConfigurationSystem.Client.Utilities import getDBParameters -from DIRAC.Core.Utilities.ObjectLoader import ObjectLoader - -TABLESLIST = ["PilotsLogging"] - -metadata = MetaData() -Base = declarative_base() - - -class PilotsLoggingDB: - def __init__(self, parentLogger=None): - - if not parentLogger: - parentLogger = gLogger - self.log = parentLogger.getSubLogger(self.__class__.__name__) - - result = getDBParameters("WorkloadManagement/PilotsLoggingDB") - if not result["OK"]: - raise RuntimeError("Cannot get database parameters: %s" % result["Message"]) - - dbParameters = result["Value"] - self.dbHost = dbParameters["Host"] - self.dbPort = dbParameters["Port"] - self.dbUser = dbParameters["User"] - self.dbPass = dbParameters["Password"] - self.dbName = dbParameters["DBName"] - - self.objectLoader = ObjectLoader() - - # These are the list of tables that will be created. - # They can be extended in an extension module - result = self.objectLoader.loadObject(__name__, "TABLESLIST") - if not result["OK"]: - raise Exception(result["Message"]) - self.tablesList = result["Value"] - - self.__initializeConnection() - resp = self.__initializeDB() - if not resp["OK"]: - raise Exception("Couldn't create tables: " + resp["Message"]) - - ########################################################################################## - - def __initializeConnection(self): - """ - This should be in a base class eventually - """ - - self.engine = create_engine( - f"mysql://{self.dbUser}:{self.dbPass}@{self.dbHost}:{self.dbPort}/{self.dbName}", - pool_recycle=3600, - echo_pool=True, - echo=self.log.getLevel() == "DEBUG", - ) - self.sqlalchemySession = scoped_session(sessionmaker(bind=self.engine)) - self.inspector = Inspector.from_engine(self.engine) - - ########################################################################################## - def __initializeDB(self): - """ - Create the tables, if they are not there yet - """ - - tablesInDB = self.inspector.get_table_names() - - for table in self.tablesList: - if table not in tablesInDB: - result = self.objectLoader.loadObject(__name__, table) - if not result["OK"]: - return result - result["Value"].__table__.create(self.engine) - else: - gLogger.debug(f"Table {table} already exists") - - return S_OK() - - ########################################################################################## - def addPilotsLogging(self, pilotUUID, timestamp, source, phase, status, messageContent): - """Add new pilot logging entry""" - - session = self.sqlalchemySession() - logging = PilotsLogging(pilotUUID, timestamp, source, phase, status, messageContent) - - try: - session.add(logging) - except SQLAlchemyError as e: - session.rollback() - session.close() - return S_ERROR(DErrno.ESQLA, "Failed to add PilotsLogging: " + str(e)) - - try: - session.commit() - except SQLAlchemyError as e: - session.rollback() - session.close() - return S_ERROR(DErrno.ESQLA, "Failed to commit PilotsLogging: " + str(e)) - - return S_OK() - - ########################################################################################## - def getPilotsLogging(self, pilotUUID): - """Get list of logging entries for pilot""" - - session = self.sqlalchemySession() - - pilotLogging = [] - for pl in ( - session.query(PilotsLogging) - .filter(PilotsLogging.pilotUUID == pilotUUID) - .order_by(PilotsLogging.timestamp) - .all() - ): - entry = {} - entry["pilotUUID"] = pl.pilotUUID - entry["timestamp"] = pl.timestamp - entry["source"] = pl.source - entry["phase"] = pl.phase - entry["status"] = pl.status - entry["messageContent"] = pl.messageContent - pilotLogging.append(entry) - - return S_OK(pilotLogging) - - ########################################################################################## - def deletePilotsLogging(self, pilotUUID): - """Delete all logging entries for pilot""" - - if isinstance(pilotUUID, str): - pilotUUID = [ - pilotUUID, - ] - - session = self.sqlalchemySession() - - session.query(PilotsLogging).filter(PilotsLogging.pilotUUID.in_(pilotUUID)).delete(synchronize_session="fetch") - - try: - session.commit() - except SQLAlchemyError as e: - session.rollback() - session.close() - return S_ERROR(DErrno.ESQLA, "Failed to commit: " + str(e)) - - return S_OK() - - ########################################################################################## - - -class PilotsLogging(Base): - """PilotsLogging table""" - - __tablename__ = "PilotsLogging" - __table_args__ = {"mysql_engine": "InnoDB", "mysql_charset": "utf8"} - - logID = Column("LogID", Integer, primary_key=True, autoincrement=True) - pilotUUID = Column("pilotUUID", String(255), nullable=False) - timestamp = Column("timestamp", String(255), nullable=False) - source = Column("source", String(255), nullable=False) - phase = Column("phase", String(255), nullable=False) - status = Column("status", String(255), nullable=False) - messageContent = Column("messageContent", String(255), nullable=False) - - def __init__(self, pilotUUID, timestamp, source, phase, status, messageContent): - self.pilotUUID = pilotUUID - self.timestamp = timestamp - self.source = source - self.phase = phase - self.status = status - self.messageContent = messageContent diff --git a/src/DIRAC/WorkloadManagementSystem/DB/PilotsLoggingDB.sql b/src/DIRAC/WorkloadManagementSystem/DB/PilotsLoggingDB.sql deleted file mode 100644 index ef116dde1c0..00000000000 --- a/src/DIRAC/WorkloadManagementSystem/DB/PilotsLoggingDB.sql +++ /dev/null @@ -1,2 +0,0 @@ -# Everything is created by the DB object upon instantiatiation if it does not exists. -use PilotsLoggingDB; diff --git a/src/DIRAC/WorkloadManagementSystem/Service/JobManagerHandler.py b/src/DIRAC/WorkloadManagementSystem/Service/JobManagerHandler.py index 8808e9baf93..7661bdd81e7 100755 --- a/src/DIRAC/WorkloadManagementSystem/Service/JobManagerHandler.py +++ b/src/DIRAC/WorkloadManagementSystem/Service/JobManagerHandler.py @@ -63,17 +63,6 @@ def initializeHandler(cls, serviceInfoDict): except RuntimeError as excp: return S_ERROR("Can't connect to DB: %s" % excp) - cls.pilotsLoggingDB = None - enablePilotsLogging = Operations().getValue("/Services/JobMonitoring/usePilotsLoggingFlag", False) - if enablePilotsLogging: - try: - result = ObjectLoader().loadObject("WorkloadManagementSystem.DB.PilotsLoggingDB", "PilotsLoggingDB") - if not result["OK"]: - return result - cls.pilotsLoggingDB = result["Value"](parentLogger=cls.log) - except RuntimeError as excp: - return S_ERROR("Can't connect to DB: %s" % excp) - cls.msgClient = MessageClient("WorkloadManagement/OptimizationMind") result = cls.msgClient.connect(JobManager=True) if not result["OK"]: @@ -431,8 +420,7 @@ def export_removeJob(self, jobIDs): return S_OK(validJobList) def __deleteJob(self, jobID): - """Set the job status to "Deleted" - and remove the pilot that ran and its logging info if the pilot is finished. + """Set the job status to "Deleted" and remove the pilot that ran. :param int jobID: job ID :return: S_OK()/S_ERROR() @@ -445,7 +433,7 @@ def __deleteJob(self, jobID): if not result["OK"]: self.log.warn("Failed to delete job from the TaskQueue") - # if it was the last job for the pilot, clear PilotsLogging about it + # if it was the last job for the pilot result = self.pilotAgentsDB.getPilotsForJobID(jobID) if not result["OK"]: self.log.error("Failed to get Pilots for JobID", result["Message"]) @@ -455,7 +443,7 @@ def __deleteJob(self, jobID): if not res["OK"]: self.log.error("Failed to get jobs for pilot", res["Message"]) return res - if not res["Value"]: # if list of jobs for pilot is empty, delete pilot and pilotslogging + if not res["Value"]: # if list of jobs for pilot is empty, delete pilot result = self.pilotAgentsDB.getPilotInfo(pilotID=pilot) if not result["OK"]: self.log.error("Failed to get pilot info", result["Message"]) @@ -465,11 +453,6 @@ def __deleteJob(self, jobID): if not ret["OK"]: self.log.error("Failed to delete pilot from PilotAgentsDB", ret["Message"]) return ret - if self.pilotsLoggingDB: - ret = self.pilotsLoggingDB.deletePilotsLogging(pilotRef) - if not ret["OK"]: - self.log.error("Failed to delete pilot logging from PilotAgentsDB", ret["Message"]) - return ret return S_OK() diff --git a/src/DIRAC/WorkloadManagementSystem/Service/PilotManagerHandler.py b/src/DIRAC/WorkloadManagementSystem/Service/PilotManagerHandler.py index 46cb769f71e..71c8d345abf 100644 --- a/src/DIRAC/WorkloadManagementSystem/Service/PilotManagerHandler.py +++ b/src/DIRAC/WorkloadManagementSystem/Service/PilotManagerHandler.py @@ -9,7 +9,6 @@ from DIRAC.Core.DISET.RequestHandler import RequestHandler from DIRAC.Core.Utilities.ObjectLoader import ObjectLoader -from DIRAC.ConfigurationSystem.Client.Helpers.Operations import Operations from DIRAC.ConfigurationSystem.Client.Helpers.Registry import getUsernameForDN, getDNForUsername from DIRAC.WorkloadManagementSystem.Client import PilotStatus from DIRAC.WorkloadManagementSystem.Service.WMSUtilities import ( @@ -34,17 +33,6 @@ def initializeHandler(cls, serviceInfoDict): except RuntimeError as excp: return S_ERROR("Can't connect to DB: %s" % excp) - cls.pilotsLoggingDB = None - enablePilotsLogging = Operations().getValue("/Services/JobMonitoring/usePilotsLoggingFlag", False) - if enablePilotsLogging: - try: - result = ObjectLoader().loadObject("WorkloadManagementSystem.DB.PilotsLoggingDB", "PilotsLoggingDB") - if not result["OK"]: - return result - cls.pilotsLoggingDB = result["Value"](parentLogger=cls.log) - except RuntimeError as excp: - return S_ERROR("Can't connect to DB: %s" % excp) - return S_OK() ############################################################################## @@ -489,17 +477,6 @@ def export_deletePilots(cls, pilotIDs): result = cls.pilotAgentsDB.deletePilots(pilotIDs) if not result["OK"]: return result - if cls.pilotsLoggingDB: - pilotIDs = result["Value"] - pilots = cls.pilotAgentsDB.getPilotInfo(pilotID=pilotIDs) - if not pilots["OK"]: - return pilots - pilotRefs = [] - for pilot in pilots: - pilotRefs.append(pilot["PilotJobReference"]) - result = cls.pilotsLoggingDB.deletePilotsLogging(pilotRefs) - if not result["OK"]: - return result return S_OK() @@ -509,19 +486,4 @@ def export_deletePilots(cls, pilotIDs): @classmethod def export_clearPilots(cls, interval=30, aborted_interval=7): - result = cls.pilotAgentsDB.clearPilots(interval, aborted_interval) - if not result["OK"]: - return result - if cls.pilotsLoggingDB: - pilotIDs = result["Value"] - pilots = cls.pilotAgentsDB.getPilotInfo(pilotID=pilotIDs) - if not pilots["OK"]: - return pilots - pilotRefs = [] - for pilot in pilots: - pilotRefs.append(pilot["PilotJobReference"]) - result = cls.pilotsLoggingDB.deletePilotsLogging(pilotRefs) - if not result["OK"]: - return result - - return S_OK() + return cls.pilotAgentsDB.clearPilots(interval, aborted_interval) diff --git a/src/DIRAC/WorkloadManagementSystem/Service/PilotsLoggingHandler.py b/src/DIRAC/WorkloadManagementSystem/Service/PilotsLoggingHandler.py deleted file mode 100644 index fde90519046..00000000000 --- a/src/DIRAC/WorkloadManagementSystem/Service/PilotsLoggingHandler.py +++ /dev/null @@ -1,102 +0,0 @@ -""" PilotsLoggingHandler is the implementation of the PilotsLogging service - - The following methods are available in the Service interface - - addPilotsLogging() - getPilotsLogging - deletePilotsLogging() - -""" -from DIRAC import S_OK, S_ERROR, gConfig -from DIRAC.Core.Utilities.ObjectLoader import ObjectLoader -from DIRAC.Core.DISET.RequestHandler import RequestHandler - -from DIRAC.Resources.MessageQueue.MQCommunication import createConsumer - - -class PilotsLoggingHandler(RequestHandler): - """Server side functions for Pilots Logging service""" - - @classmethod - def initializeHandler(cls, serviceInfoDict): - """Initialization of Pilots Logging service""" - cls.consumersSet = set() - try: - result = ObjectLoader().loadObject("WorkloadManagementSystem.DB.PilotsLoggingDB", "PilotsLoggingDB") - if not result["OK"]: - return result - cls.pilotsLoggingDB = result["Value"](parentLogger=cls.log) - - except RuntimeError as excp: - return S_ERROR("Can't connect to DB: %s" % excp) - - queue = cls.srv_getCSOption("PilotsLoggingQueue") - # This is pretty awful hack. Somehow, for uknown reason, I cannot access CS with srv_getCSOption. - # The only way is using full CS path, so I'm using it as a backup solution. - if not queue: - queue = gConfig.getValue(serviceInfoDict["serviceSectionPath"] + "/PilotsLoggingQueue") - result = createConsumer(queue, callback=cls.consumingCallback) - if result["OK"]: - cls.consumersSet.add(result["Value"]) - else: - return result - return S_OK() - - @classmethod - def consumingCallback(cls, headers, message): - """ - Callback function for the MQ Consumer, called for every new message and inserting it into database. - - :param headers: Headers of MQ message (not used) - :param message: Message represented as a dictionary - """ - # verify received message format - if set(message) == {"pilotUUID", "timestamp", "source", "phase", "status", "messageContent"}: - cls.pilotsLoggingDB.addPilotsLogging( - message["pilotUUID"], - message["timestamp"], - message["source"], - message["phase"], - message["status"], - message["messageContent"], - ) - - types_addPilotsLogging = [str, str, str, str, str, str] - - @classmethod - def export_addPilotsLogging(cls, pilotUUID, timestamp, source, phase, status, messageContent): - """ - Add new Pilots Logging entry - - :param pilotUUID: Pilot reference - :param status: Pilot status - :param minorStatus: Additional status information - :param timeStamp: Date and time of status event - :param source: Source of statu information - """ - - return cls.pilotsLoggingDB.addPilotsLogging(pilotUUID, timestamp, source, phase, status, messageContent) - - types_getPilotsLogging = [str] - - @classmethod - def export_getPilotsLogging(cls, pilotUUID): - """ - Get all Logging entries for Pilot - - :param pilotUUID: Pilot reference - """ - - return cls.pilotsLoggingDB.getPilotsLogging(pilotUUID) - - types_deletePilotsLogging = [[str, list]] - - @classmethod - def export_deletePilotsLogging(cls, pilotUUID): - """ - Delete all Logging entries for Pilot - - :param pilotUUID: Pilot reference - """ - - return cls.pilotsLoggingDB.deletePilotsLogging(pilotUUID) diff --git a/src/DIRAC/WorkloadManagementSystem/scripts/dirac_admin_pilot_logging_info.py b/src/DIRAC/WorkloadManagementSystem/scripts/dirac_admin_pilot_logging_info.py deleted file mode 100755 index 8d5524400e9..00000000000 --- a/src/DIRAC/WorkloadManagementSystem/scripts/dirac_admin_pilot_logging_info.py +++ /dev/null @@ -1,61 +0,0 @@ -#! /usr/bin/env python -""" -Get Pilots Logging for specific Pilot UUID or Job ID. - -WARNING: Only one option (either uuid or jobid) should be used. -""" -import DIRAC -from DIRAC import S_OK, gLogger -from DIRAC.Core.Base.Script import Script - -uuid = None -jobid = None - - -def setUUID(optVal): - """ - Set UUID from arguments - """ - global uuid - uuid = optVal - return S_OK() - - -def setJobID(optVal): - """ - Set JobID from arguments - """ - global jobid - jobid = optVal - return S_OK() - - -@Script() -def main(): - global uuid - global jobid - Script.registerSwitch("u:", "uuid=", "get PilotsLogging for given Pilot UUID", setUUID) - Script.registerSwitch("j:", "jobid=", "get PilotsLogging for given Job ID", setJobID) - Script.parseCommandLine() - - from DIRAC.WorkloadManagementSystem.Client.PilotManagerClient import PilotManagerClient - - if jobid: - result = PilotManagerClient().getPilots(jobid) - if not result["OK"]: - gLogger.error(result["Message"]) - DIRAC.exit(1) - gLogger.debug(result["Value"]) - uuid = list(result["Value"])[0] - - result = PilotManagerClient().getPilotLoggingInfo(uuid) - if not result["OK"]: - gLogger.error(result["Message"]) - DIRAC.exit(1) - gLogger.notice(result["Value"]) - - DIRAC.exit(0) - - -if __name__ == "__main__": - main() diff --git a/tests/Integration/WorkloadManagementSystem/Test_PilotsLoggingClient.py b/tests/Integration/WorkloadManagementSystem/Test_PilotsLoggingClient.py deleted file mode 100644 index 2f93939c334..00000000000 --- a/tests/Integration/WorkloadManagementSystem/Test_PilotsLoggingClient.py +++ /dev/null @@ -1,114 +0,0 @@ -""" This is a test of the chain - PilotsLoggingClient -> PilotsLoggingHandler -> PilotsLoggingDB - - It supposes that the DB is present, and that the service is running -""" -import unittest -import sys - -from DIRAC.WorkloadManagementSystem.Client.PilotsLoggingClient import PilotsLoggingClient - -import DIRAC - -DIRAC.initialize() # Initialize configuration - - -class TestPilotsLogging(unittest.TestCase): - def setUp(self): - self.pilotsLoggingClient = PilotsLoggingClient() - - def tearDown(self): - pass - - -class PilotsLogging(TestPilotsLogging): - def test_PilotsLoggingAddGetDelete(self): - resp = self.pilotsLoggingClient.addPilotsLogging( - "11111111-1111-1111-1111-111111111111", "timestamp1", "test", "phase", "status", "messageContent" - ) - self.assertTrue(resp["OK"], "Failed to add PilotsLogging") - resp = self.pilotsLoggingClient.addPilotsLogging( - "11111111-1111-1111-1111-111111111111", "timestamp2", "test2", "phase2", "status2", "messageContent2" - ) - self.assertTrue(resp["OK"], "Failed to add PilotsLogging") - resp = self.pilotsLoggingClient.getPilotsLogging("11111111-1111-1111-1111-111111111111") - self.assertTrue(resp["OK"], "Failed to get PilotsLogging") - test_sample = { - "pilotUUID": "11111111-1111-1111-1111-111111111111", - "timestamp": "timestamp1", - "source": "test", - "phase": "phase", - "status": "status", - "messageContent": "messageContent", - } - test_sample2 = { - "pilotUUID": "11111111-1111-1111-1111-111111111111", - "timestamp": "timestamp2", - "source": "test2", - "phase": "phase2", - "status": "status2", - "messageContent": "messageContent2", - } - self.assertEqual(resp["Value"], [test_sample, test_sample2], "Wrong data comes out of Service") - resp = self.pilotsLoggingClient.deletePilotsLogging("11111111-1111-1111-1111-111111111111") - self.assertTrue(resp["OK"], "Failed to delete PilotsLogging") - resp = self.pilotsLoggingClient.getPilotsLogging("11111111-1111-1111-1111-111111111111") - self.assertTrue(resp["OK"]) - self.assertEqual(resp["Value"], [], "PilotsLogging was not really deleted") - - def test_PilotsLoggingEmptyGetDelete(self): - - resp = self.pilotsLoggingClient.getPilotsLogging("11111111-1111-1111-1111-111111111111") - self.assertTrue(resp["OK"], "Failed to get PilotsLogging") - resp = self.pilotsLoggingClient.deletePilotsLogging("11111111-1111-1111-1111-111111111111") - self.assertTrue(resp["OK"], "Failed to delete PilotsLogging") - - def test_PilotsLoggingDeleteList(self): - - test_sample1 = { - "pilotUUID": "11111111-1111-1111-1111-111111111111", - "timestamp": "timestamp1", - "source": "test", - "phase": "phase1", - "status": "status1", - "messageContent": "messageContent1", - } - test_sample2 = { - "pilotUUID": "22222222-2222-2222-2222-222222222222", - "timestamp": "timestamp2", - "source": "test", - "phase": "phase2", - "status": "status2", - "messageContent": "messageContent2", - } - - resp = self.pilotsLoggingClient.addPilotsLogging( - "11111111-1111-1111-1111-111111111111", "timestamp1", "test", "phase1", "status1", "messageContent1" - ) - self.assertTrue(resp["OK"], "Failed to add PilotsLogging") - resp = self.pilotsLoggingClient.addPilotsLogging( - "22222222-2222-2222-2222-222222222222", "timestamp2", "test", "phase2", "status2", "messageContent2" - ) - resp = self.pilotsLoggingClient.getPilotsLogging("11111111-1111-1111-1111-111111111111") - self.assertTrue(resp["OK"], "Failed to get PilotsLogging") - self.assertEqual(resp["Value"], [test_sample1], "Wrong data comes out of Service") - resp = self.pilotsLoggingClient.getPilotsLogging("22222222-2222-2222-2222-222222222222") - self.assertTrue(resp["OK"], "Failed to get PilotsLogging") - self.assertEqual(resp["Value"], [test_sample2], "Wrong data comes out of Service") - resp = self.pilotsLoggingClient.deletePilotsLogging( - ["11111111-1111-1111-1111-111111111111", "22222222-2222-2222-2222-222222222222"] - ) - self.assertTrue(resp["OK"], "Failed to delete PilotsLogging") - resp = self.pilotsLoggingClient.getPilotsLogging("11111111-1111-1111-1111-111111111111") - self.assertTrue(resp["OK"], "Failed to get PilotsLogging") - self.assertEqual(resp["Value"], [], "PilotsLogging was not really deleted") - resp = self.pilotsLoggingClient.getPilotsLogging("22222222-2222-2222-2222-222222222222") - self.assertTrue(resp["OK"], "Failed to get PilotsLogging") - self.assertEqual(resp["Value"], [], "PilotsLogging was not really deleted") - - -if __name__ == "__main__": - suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestPilotsLogging) - suite.addTest(unittest.defaultTestLoader.loadTestsFromTestCase(PilotsLogging)) - testResult = unittest.TextTestRunner(verbosity=2).run(suite) - sys.exit(not testResult.wasSuccessful()) diff --git a/tests/Integration/all_integration_client_tests.sh b/tests/Integration/all_integration_client_tests.sh index 75208cbf706..8678d93aec3 100644 --- a/tests/Integration/all_integration_client_tests.sh +++ b/tests/Integration/all_integration_client_tests.sh @@ -52,7 +52,6 @@ pytest --no-check-dirac-environment "${THIS_DIR}/ResourceStatusSystem/Test_Email #-------------------------------------------------------------------------------# echo -e "*** $(date -u) **** WMS TESTS ****\n" -# pytest --no-check-dirac-environment "${CLIENTINSTALLDIR}"/DIRAC/tests/Integration/WorkloadManagementSystem/Test_PilotsLoggingClient.py |& tee -a clientTestOutputs.txt pytest --no-check-dirac-environment "${THIS_DIR}/WorkloadManagementSystem/Test_SandboxStoreClient.py" |& tee -a clientTestOutputs.txt; (( ERR |= "${?}" )) pytest --no-check-dirac-environment "${THIS_DIR}/WorkloadManagementSystem/Test_JobWrapper.py" |& tee -a clientTestOutputs.txt; (( ERR |= "${?}" )) pytest --no-check-dirac-environment "${THIS_DIR}/WorkloadManagementSystem/Test_PilotsClient.py" |& tee -a clientTestOutputs.txt; (( ERR |= "${?}" )) diff --git a/tests/Jenkins/utilities.sh b/tests/Jenkins/utilities.sh index ebbd9916b8a..a3170518457 100644 --- a/tests/Jenkins/utilities.sh +++ b/tests/Jenkins/utilities.sh @@ -662,7 +662,7 @@ diracServices(){ echo '==> [diracServices]' # Ignore tornado services - local services=$(cut -d '.' -f 1 < services | grep -v Tornado | grep -v TokenManager | grep -v PilotsLogging | grep -v StorageElementHandler | grep -v ^ConfigurationSystem | grep -v RAWIntegrity | grep -v RunDBInterface | grep -v ComponentMonitoring | sed 's/System / /g' | sed 's/Handler//g' | sed 's/ /\//g') + local services=$(cut -d '.' -f 1 < services | grep -v Tornado | grep -v TokenManager | grep -v StorageElementHandler | grep -v ^ConfigurationSystem | grep -v RAWIntegrity | grep -v RunDBInterface | grep -v ComponentMonitoring | sed 's/System / /g' | sed 's/Handler//g' | sed 's/ /\//g') # group proxy, will be uploaded explicitly # echo '==> getting/uploading proxy for prod'