[Bug]: Jobs using setAllowParallelRuns(false) never run

[GorudoYami]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

When using cron background jobs that use setAllowParallelRuns(false) never run. Tried to trace it and looks like reserved_at is set in $jobList->getNext([...]) and because of that in start([...]) of Job !$this->getAllowParallelRuns() && $jobList->hasReservedJob(get_class($this)) is always true so a message saying for example:
"Skipping OCA\\Indexer\\BackgroundJob\\IndexFiles job with ID 127 because another job with the same class is already running" is displayed in logs.

Steps to reproduce

1. Create an app with a TimedJob and use setAllowParallelRuns(false) in the constructor
2. Enable the app
3. Wait for cron to try execute the job

Expected behavior

Job runs at least once and at most once at the time

Installation method

Community Docker image

Nextcloud Server version

27

Operating system

Other

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "debug": true,
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.0.0.8",
        "overwrite.cli.url": "http:\/\/server.home:8080",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "server.home:8080",
        "app_install_overwrite": [
            "duplicatefinder",
            "indexer"
        ],
        "maintenance": false,
        "loglevel": 0,
        "theme": ""
    }
}

List of activated Apps

- activity: 2.19.0
  - admin_audit: 1.17.0
  - cloud_federation_api: 1.10.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - federatedfilesharing: 1.17.0
  - files: 1.22.0
  - files_automatedtagging: 1.17.0
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - indexer: 1.0.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - notifications: 2.15.0
  - oauth2: 1.15.0
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - workflow_script: 1.12.0
  - workflowengine: 2.9.0
Disabled:
  - bruteforcesettings: 2.7.0 (installed 2.7.0)
  - camerarawpreviews: 0.8.1 (installed 0.8.1)
  - circles: 27.0.0 (installed 26.0.0)
  - comments: 1.17.0 (installed 1.16.0)
  - contactsinteraction: 1.8.0 (installed 1.7.0)
  - duplicatefinder: 0.0.15 (installed 0.0.15)
  - encryption: 2.15.0 (installed 2.15.0)
  - federation: 1.17.0 (installed 1.16.0)
  - files_external: 1.19.0
  - firstrunwizard: 2.16.0 (installed 2.15.0)
  - nextcloud_announcements: 1.16.0 (installed 1.15.0)
  - recommendations: 1.6.0 (installed 1.5.0)
  - sharebymail: 1.17.0 (installed 1.16.0)
  - support: 1.10.0 (installed 1.9.0)
  - survey_client: 1.15.0 (installed 1.14.0)
  - suspicious_login: 5.0.0 (installed 5.0.0)
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0
  - weather_status: 1.7.0 (installed 1.6.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"4PD8RX5HuWlD19vzd0KK","level":0,"time":"2023-07-18T18:12:00+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"CLI cron call has selected job with ID 127","userAgent":"--","version":"27.0.0.8","data":{"app":"cron"}}
{"reqId":"4PD8RX5HuWlD19vzd0KK","level":0,"time":"2023-07-18T18:12:00+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Query: SELECT * FROM `*PREFIX*jobs` WHERE (`reserved_at` <> :dcValue1) AND (`class` = :dcValue2) LIMIT 1","userAgent":"--","version":"27.0.0.8","data":[]}
{"reqId":"4PD8RX5HuWlD19vzd0KK","level":0,"time":"2023-07-18T18:12:00+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"HasReservedJobs: 1","userAgent":"--","version":"27.0.0.8","data":[]}
{"reqId":"4PD8RX5HuWlD19vzd0KK","level":0,"time":"2023-07-18T18:12:00+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Class: OCA\\Indexer\\BackgroundJob\\IndexFiles","userAgent":"--","version":"27.0.0.8","data":[]}
{"reqId":"4PD8RX5HuWlD19vzd0KK","level":0,"time":"2023-07-18T18:12:00+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Array\n(\n    [id] => 127\n    [class] => OCA\\Indexer\\BackgroundJob\\IndexFiles\n    [argument] => null\n    [last_run] => 1689703920\n    [last_checked] => 1689703920\n    [reserved_at] => 1689703920\n    [execution_duration] => 0\n    [argument_hash] => 37a6259cc0c1dae299a7866489dff0bd\n    [time_sensitive] => 1\n)\n","userAgent":"--","version":"27.0.0.8","data":[]}
{"reqId":"4PD8RX5HuWlD19vzd0KK","level":0,"time":"2023-07-18T18:12:00+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Skipping OCA\\Indexer\\BackgroundJob\\IndexFiles job with ID 127 because another job with the same class is already running","userAgent":"--","version":"27.0.0.8","data":{"app":"cron"}}

Additional info

Version 27.0.0
There are additional logs that I've inserted myself trying to trace the problem

[szaimen]

cc @marcelklehr @nextcloud/server-backend