Skip to content

USHIFT-7240: work around kernel 7.x SELinux execmem denial on EL10 bootc#6874

Closed
eslutsky wants to merge 1 commit into
mainfrom
BUG-USHIFT-7215
Closed

USHIFT-7240: work around kernel 7.x SELinux execmem denial on EL10 bootc#6874
eslutsky wants to merge 1 commit into
mainfrom
BUG-USHIFT-7215

Conversation

@eslutsky

@eslutsky eslutsky commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • CRI-O fails to start on RHEL 10.2 bootc VMs due to a kernel 7.x regression where SELinux mprotect() checks on composefs/overlayfs evaluate the backing file's context instead of the overlay file's context
  • CRI-O runs as kernel_t instead of container_runtime_t, which denies the execmem permission needed for text relocations (CGO libgpgme binding)
  • Adds a temporary SELinux CIL policy module + systemd oneshot service to the EL10 base containerfile (rhel102-test-agent), unblocking all EL10 bootc CI jobs

Root cause

Kernel 7.0 has a regression in LSM/overlayfs where mprotect() access checks evaluate the backing file's security context instead of the overlay file's context. On composefs (used by bootc), this prevents SELinux domain transitions — CRI-O stays as kernel_t instead of transitioning to container_runtime_t.

Upstream fix

3-commit series by Paul Moore, landed in v7.1-rc1 (not yet backported to RHEL 10.2 kernel):

  1. 880bd496ec72 — fs: prepare for adding LSM blob to backing_file
  2. 6af36aeb147a — lsm: add backing_file LSM hooks
  3. 82544d36b172 — selinux: fix overlayfs mmap() and mprotect() access checks

What this PR does

Adds to the EL10 base containerfile (rhel102-test-agent.containerfile):

  1. A CIL SELinux policy file granting execmem to kernel_t
  2. A systemd oneshot service that loads the policy before crio.service

Since all EL10 bootc images inherit from rhel102-test-agent, this covers every affected job.

Remove this workaround once the upstream kernel fix is backported to the RHEL 10.2 kernel.

References

Test plan

  • Verify EL10 bootc periodic CI jobs pass (e2e-aws-tests-bootc-nightly-el10)
  • Verify CRI-O starts successfully on RHEL 10.2 bootc VMs with kernel 7.x
  • Verify no regressions on EL9 bootc jobs (unaffected by this change)

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

  • Bug Fixes
    • Addressed a kernel 7.x compatibility issue to enhance system stability and reliability.

@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ec89ac92-f0ce-41e4-8734-19624c4ac9eb

📥 Commits

Reviewing files that changed from the base of the PR and between b1d80ad and 1b65a77.

📒 Files selected for processing (1)
  • packaging/selinux/microshift.te

Walkthrough

packaging/selinux/microshift.te adds kernel_t to the gen_require block and inserts an allow kernel_t self:process execmem rule with comments marking it as a temporary workaround for a kernel 7.x composefs/overlayfs regression, with guidance on when to remove it.

Changes

SELinux Policy Workaround

Layer / File(s) Summary
kernel_t execmem workaround rule
packaging/selinux/microshift.te
Adds kernel_t to gen_require and inserts allow kernel_t self:process execmem as a scoped workaround for a kernel 7.x composefs/overlayfs regression, with inline removal guidance.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~5 minutes

Suggested labels

ready-for-human-review

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding a SELinux workaround for kernel 7.x execmem denial on EL10 bootc systems, directly matching the PR's core objective.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed This PR modifies SELinux policy files (packaging/selinux/microshift.te), not Ginkgo test files. The check for stable test names is not applicable.
Test Structure And Quality ✅ Passed PR only adds SELinux policy file; no Ginkgo tests present or modified, making the test structure quality check not applicable.
Microshift Test Compatibility ✅ Passed This PR contains no new Ginkgo e2e tests. Changes are limited to SELinux policy (microshift.te) and container image configuration (rhel102-test-agent.containerfile), making this custom check inappl...
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests are added in this PR. The changes consist of SELinux policy configuration, test scenarios/suites (shell scripts and Robot tests), and vendor dependencies. The SNO compatibil...
Topology-Aware Scheduling Compatibility ✅ Passed PR modifies only a SELinux policy file (microshift.te), not deployment manifests or operator code. No Kubernetes scheduling constraints are introduced.
Ote Binary Stdout Contract ✅ Passed PR modifies only SELinux policy file (packaging/selinux/microshift.te) with no stdout-writing code; OTE stdout contract check is not applicable.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR modifies only SELinux policy file (packaging/selinux/microshift.te); no Ginkgo e2e tests added, so check is not applicable.
No-Weak-Crypto ✅ Passed PR modifies only SELinux policy file (microshift.te), adding kernel_t domain and execmem permission. No weak crypto (MD5/SHA1/DES/RC4/Blowfish/ECB), custom crypto implementations, or non-constant-t...
Container-Privileges ✅ Passed PR changes are limited to SELinux policy (microshift.te) and containerfile updates. No container privilege escalation settings (privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPriv...
No-Sensitive-Data-In-Logs ✅ Passed No logging statements found that expose passwords, tokens, API keys, PII, session IDs, internal hostnames, or customer data in microshift.te or the containerfile.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch BUG-USHIFT-7215

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from kasturinarra and pmtk June 15, 2026 16:58
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 15, 2026
@eslutsky

Copy link
Copy Markdown
Contributor Author

/retest

@eslutsky

Copy link
Copy Markdown
Contributor Author

/test

@eslutsky

Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@eslutsky

Copy link
Copy Markdown
Contributor Author

/retest

@ggiguash

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 17, 2026
@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eslutsky, ggiguash

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@eslutsky

Copy link
Copy Markdown
Contributor Author

/retest

@eslutsky eslutsky changed the title CI: work around kernel 7.x SELinux execmem denial on EL10 bootc USHIFT-7240: work around kernel 7.x SELinux execmem denial on EL10 bootc Jun 17, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 17, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 17, 2026

Copy link
Copy Markdown

@eslutsky: This pull request references USHIFT-7240 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

  • CRI-O fails to start on RHEL 10.2 bootc VMs due to a kernel 7.x regression where SELinux mprotect() checks on composefs/overlayfs evaluate the backing file's context instead of the overlay file's context
  • CRI-O runs as kernel_t instead of container_runtime_t, which denies the execmem permission needed for text relocations (CGO libgpgme binding)
  • Adds a temporary SELinux CIL policy module + systemd oneshot service to the EL10 base containerfile (rhel102-test-agent), unblocking all EL10 bootc CI jobs

Root cause

Kernel 7.0 has a regression in LSM/overlayfs where mprotect() access checks evaluate the backing file's security context instead of the overlay file's context. On composefs (used by bootc), this prevents SELinux domain transitions — CRI-O stays as kernel_t instead of transitioning to container_runtime_t.

Upstream fix

3-commit series by Paul Moore, landed in v7.1-rc1 (not yet backported to RHEL 10.2 kernel):

  1. 880bd496ec72 — fs: prepare for adding LSM blob to backing_file
  2. 6af36aeb147a — lsm: add backing_file LSM hooks
  3. 82544d36b172 — selinux: fix overlayfs mmap() and mprotect() access checks

What this PR does

Adds to the EL10 base containerfile (rhel102-test-agent.containerfile):

  1. A CIL SELinux policy file granting execmem to kernel_t
  2. A systemd oneshot service that loads the policy before crio.service

Since all EL10 bootc images inherit from rhel102-test-agent, this covers every affected job.

Remove this workaround once the upstream kernel fix is backported to the RHEL 10.2 kernel.

References

Test plan

  • Verify EL10 bootc periodic CI jobs pass (e2e-aws-tests-bootc-nightly-el10)
  • Verify CRI-O starts successfully on RHEL 10.2 bootc VMs with kernel 7.x
  • Verify no regressions on EL9 bootc jobs (unaffected by this change)

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes

  • Resolved CRI-O permission denials on kernel 7.x systems caused by a composefs/overlayfs regression by applying an SELinux policy workaround automatically at startup.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@eslutsky

Copy link
Copy Markdown
Contributor Author

/retest

Add allow rule to the microshift SELinux policy module so that CRI-O
can perform text relocations when running as kernel_t on composefs.
This is needed until the RHEL 10.2 kernel backports the upstream fix
(880bd496ec72, 6af36aeb147a, 82544d36b172).

USHIFT-7215

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 17, 2026
@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@coderabbitai coderabbitai Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 17, 2026
@eslutsky

Copy link
Copy Markdown
Contributor Author

/test ci/prow/test-unit

@eslutsky

Copy link
Copy Markdown
Contributor Author

/test pull-ci-openshift-microshift-main-test-unit

@eslutsky

Copy link
Copy Markdown
Contributor Author

/test test-unit

@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@eslutsky: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-tests-bootc-el9 1b65a77 link true /test e2e-aws-tests-bootc-el9
ci/prow/e2e-aws-tests 1b65a77 link true /test e2e-aws-tests
ci/prow/e2e-aws-tests-arm 1b65a77 link true /test e2e-aws-tests-arm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@eslutsky

Copy link
Copy Markdown
Contributor Author

/test ci/prow/e2e-aws-tests-bootc-el9

@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@eslutsky: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test e2e-aws-ai-model-serving
/test e2e-aws-footprint-and-performance
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9
/test e2e-aws-tests-bootc-periodic-arm-el10
/test e2e-aws-tests-bootc-periodic-arm-el9
/test e2e-aws-tests-bootc-periodic-el10
/test e2e-aws-tests-bootc-periodic-el9
/test e2e-aws-tests-bootc-release-arm-el10
/test e2e-aws-tests-bootc-release-arm-el9
/test e2e-aws-tests-bootc-release-el10
/test e2e-aws-tests-bootc-release-el9
/test e2e-aws-tests-bootc-upstream
/test e2e-aws-tests-bootc-upstream-arm
/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm
/test e2e-aws-tests-periodic
/test e2e-aws-tests-periodic-arm
/test e2e-aws-tests-release
/test e2e-aws-tests-release-arm
/test ocp-full-conformance-rhel-eus
/test ocp-full-conformance-serial-rhel-eus
/test test-rpm
/test test-unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test images
/test security
/test test-rebase

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-microshift-main-e2e-aws-tests
pull-ci-openshift-microshift-main-e2e-aws-tests-arm
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-arm-el10
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-arm-el9
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-el10
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-el9
pull-ci-openshift-microshift-main-images
pull-ci-openshift-microshift-main-security
pull-ci-openshift-microshift-main-test-rpm
pull-ci-openshift-microshift-main-test-unit
pull-ci-openshift-microshift-main-verify
pull-ci-openshift-microshift-main-verify-deps
Details

In response to this:

/test ci/prow/e2e-aws-tests-bootc-el9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@eslutsky

Copy link
Copy Markdown
Contributor Author

/test ci/prow/e2e-aws-tests-bootc-el9

@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@eslutsky: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test e2e-aws-ai-model-serving
/test e2e-aws-footprint-and-performance
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9
/test e2e-aws-tests-bootc-periodic-arm-el10
/test e2e-aws-tests-bootc-periodic-arm-el9
/test e2e-aws-tests-bootc-periodic-el10
/test e2e-aws-tests-bootc-periodic-el9
/test e2e-aws-tests-bootc-release-arm-el10
/test e2e-aws-tests-bootc-release-arm-el9
/test e2e-aws-tests-bootc-release-el10
/test e2e-aws-tests-bootc-release-el9
/test e2e-aws-tests-bootc-upstream
/test e2e-aws-tests-bootc-upstream-arm
/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm
/test e2e-aws-tests-periodic
/test e2e-aws-tests-periodic-arm
/test e2e-aws-tests-release
/test e2e-aws-tests-release-arm
/test ocp-full-conformance-rhel-eus
/test ocp-full-conformance-serial-rhel-eus
/test test-rpm
/test test-unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test images
/test security
/test test-rebase

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-microshift-main-e2e-aws-tests
pull-ci-openshift-microshift-main-e2e-aws-tests-arm
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-arm-el10
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-arm-el9
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-el10
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-el9
pull-ci-openshift-microshift-main-images
pull-ci-openshift-microshift-main-security
pull-ci-openshift-microshift-main-test-rpm
pull-ci-openshift-microshift-main-test-unit
pull-ci-openshift-microshift-main-verify
pull-ci-openshift-microshift-main-verify-deps
Details

In response to this:

/test ci/prow/e2e-aws-tests-bootc-el9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@eslutsky

Copy link
Copy Markdown
Contributor Author

/close
closing in favor of #6903

@openshift-ci openshift-ci Bot closed this Jun 18, 2026
@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

@eslutsky: Closed this PR.

Details

In response to this:

/close
closing in favor of #6903

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants