Restart MicroShift when IP address changes by oglok · Pull Request #650 · openshift/microshift

oglok · 2022-04-07T16:56:50Z

MicroShift endpoints rely on the host IP address. If that IP changes for whatever reason, APIs will be completely unavailable and MicroShift will stop working. This PR will detect IP host changes, and shutdown MicroShift. Since the production deployment model is based on systemd service, MicroShift service will be automatically restarted and endpoints will be recreated to be back in a functional state.

Signed-off-by: Miguel Angel Ajo majopela@redhat.com
Signed-off-by: Ricardo Noriega rnoriega@redhat.com

Which issue(s) this PR addresses:

Closes #556

openshift-ci · 2022-04-07T16:56:53Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

when IP address changes have been detected. Signed-off-by: Miguel Angel Ajo <majopela@redhat.com> Signed-off-by: Ricardo Noriega <rnoriega@redhat.com>

oglok · 2022-04-08T11:42:20Z

To show some testing. After the IP address changes, it is detected and MicroShift exits:

W0408 13:17:35.906386  595827 ipwatch.go:58] IP address has changed from "192.168.0.140" to "192.168.0.99", restarting MicroShift

Systemd will restart MicroShift service, and endpoints will be recreated:

I0408 13:22:29.958411  599949 apiservice.go:86] deleting outdated endpoints openshift-apiserver
I0408 13:22:29.997946  599949 apiservice.go:86] deleting outdated endpoints openshift-oauth-apiserver

mangelajo · 2022-04-08T13:53:57Z

^checker complains;, the port is missing in the args?

mangelajo · 2022-04-08T13:54:38Z

why logrus? no idea if it was originally in my patch, could be: bad Ajo! X'D

My bad too, the original PR was before the klog replacement.

mangelajo

some comments, thanks for handling it

Signed-off-by: Miguel Angel Ajo <majopela@redhat.com> Co-authored-by: Ricardo Noriega <rnoriega@redhat.com>

oglok · 2022-04-11T09:44:09Z

/retest

copejon · 2022-04-11T12:34:45Z

/test e2e-rpm-install

copejon · 2022-04-11T13:08:30Z

/test e2e-rpm-install

oglok · 2022-04-11T15:30:55Z

/retest

mangelajo · 2022-04-11T22:24:49Z

/retest Sent with a Spark

…

On 11 Apr 2022, 17:42 +0200, openshift-ci[bot] ***@***.***>, wrote: @oglok: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests: Test name Commit Details Required Rerun command ci/prow/e2e-rpm-install 3e5d8cf link false /test e2e-rpm-install Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because your review was requested.Message ID: ***@***.***>

mangelajo · 2022-04-12T11:20:39Z

/retest

mangelajo · 2022-04-12T13:46:16Z

/lgtm

mangelajo · 2022-04-13T14:24:57Z

Oh, it's not CI, something is broken:

[root@fedora ~]# oc get endpoints -A
NAMESPACE           NAME                      ENDPOINTS                                      AGE
default             kubernetes                192.168.1.70:6443                              2m12s
openshift-dns       dns-default               10.42.0.4:5353,10.42.0.4:9154,10.42.0.4:5353   83s
openshift-ingress   router-external-default                                                  83s
openshift-ingress   router-internal-default                                                  83s
[root@fedora ~]# oc get pods -A
NAMESPACE                       NAME                                  READY   STATUS    RESTARTS   AGE
kube-system                     kube-flannel-ds-s7tbr                 1/1     Running   0          5m16s
kubevirt-hostpath-provisioner   kubevirt-hostpath-provisioner-xqsg7   1/1     Running   0          5m6s
openshift-dns                   dns-default-n4kl9                     2/2     Running   0          5m16s
openshift-dns                   node-resolver-srpnj                   1/1     Running   0          5m16s
openshift-ingress               router-default-85bcfdd948-nprmj       0/1     Running   2          5m21s
openshift-service-ca            service-ca-7764c85869-nnv2j           1/1     Running   0          5m21s

mangelajo · 2022-04-20T09:32:59Z

gogogo %)

oglok · 2022-04-20T11:52:48Z

There was more subtleness into the code than I expected xD xD

DeepEqual condition was not working properly, so I decided to change a bit the logic. Check if that endpoints name exist, do a proper deepEqual check and return if it exists, otherwise, create the endpoints.

Tests are ok locally, let's see CI.

Signed-off-by: Ricardo Noriega <rnoriega@redhat.com>

mangelajo

/lgtm

openshift-ci · 2022-04-20T13:09:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mangelajo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mangelajo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ggiguash · 2022-07-04T13:35:19Z

USHIFT-37

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 7, 2022

openshift-ci Bot requested review from cooktheryan and mangelajo April 7, 2022 16:56

oglok removed request for cooktheryan and mangelajo April 7, 2022 16:57

oglok changed the title ~~WIP: Restart MicroShift when IP address changes~~ Restart MicroShift when IP address changes Apr 8, 2022

oglok marked this pull request as ready for review April 8, 2022 11:29

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 8, 2022

openshift-ci Bot requested review from copejon and husky-parul April 8, 2022 11:29

The IPWatchController will restart MicroShift

3aa6236

when IP address changes have been detected. Signed-off-by: Miguel Angel Ajo <majopela@redhat.com> Signed-off-by: Ricardo Noriega <rnoriega@redhat.com>

oglok force-pushed the ipnew branch 2 times, most recently from 625dc1d to d55858d Compare April 8, 2022 11:37

oglok mentioned this pull request Apr 8, 2022

WIP: Add ipwatch controller to handle IP address changes #564

Closed

mangelajo reviewed Apr 8, 2022

View reviewed changes

Recreate Endpoints after IP change

3e5d8cf

Signed-off-by: Miguel Angel Ajo <majopela@redhat.com> Co-authored-by: Ricardo Noriega <rnoriega@redhat.com>

oglok force-pushed the ipnew branch from d55858d to 3e5d8cf Compare April 11, 2022 08:43

mangelajo self-requested a review April 12, 2022 13:45

openshift-ci Bot assigned mangelajo Apr 12, 2022

openshift-ci Bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Apr 12, 2022

mangelajo reviewed Apr 13, 2022

View reviewed changes

Comment thread pkg/controllers/apiservice.go Outdated

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 20, 2022

Fix DeepEqual condition and endpoints creation logic

5c2305e

Signed-off-by: Ricardo Noriega <rnoriega@redhat.com>

oglok force-pushed the ipnew branch from 55ad63b to 5c2305e Compare April 20, 2022 12:35

mangelajo approved these changes Apr 20, 2022

View reviewed changes

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Apr 20, 2022

oglok merged commit 48863ff into main Apr 20, 2022

oglok deleted the ipnew branch April 20, 2022 13:52

oglok mentioned this pull request Apr 20, 2022

[BUG] the server doesn't have a resource type "route" #658

Closed

mangelajo mentioned this pull request Jun 10, 2022

cleanup extra format #724

Merged

Conversation

oglok commented Apr 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Apr 7, 2022

Uh oh!

oglok commented Apr 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mangelajo Apr 8, 2022

Choose a reason for hiding this comment

Uh oh!

oglok Apr 11, 2022

Choose a reason for hiding this comment

Uh oh!

mangelajo Apr 8, 2022

Choose a reason for hiding this comment

Uh oh!

oglok Apr 11, 2022

Choose a reason for hiding this comment

Uh oh!

mangelajo left a comment

Choose a reason for hiding this comment

Uh oh!

oglok commented Apr 11, 2022

Uh oh!

copejon commented Apr 11, 2022

Uh oh!

copejon commented Apr 11, 2022

Uh oh!

oglok commented Apr 11, 2022

Uh oh!

mangelajo commented Apr 11, 2022 via email

Uh oh!

mangelajo commented Apr 12, 2022

Uh oh!

mangelajo commented Apr 12, 2022

Uh oh!

mangelajo commented Apr 13, 2022

Uh oh!

Uh oh!

mangelajo commented Apr 20, 2022

Uh oh!

oglok commented Apr 20, 2022

Uh oh!

mangelajo left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Apr 20, 2022

Uh oh!

ggiguash commented Jul 4, 2022 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

oglok commented Apr 7, 2022 •

edited

Loading

oglok commented Apr 8, 2022 •

edited

Loading

ggiguash commented Jul 4, 2022 •

edited by openshift-ci Bot

Loading