selinux configs and volume for microshift-containerized by husky-parul · Pull Request #457 · openshift/microshift

husky-parul · 2021-11-15T19:56:07Z

Signed-off-by: Parul Singh parsingh@redhat.com

Which issue(s) this PR addresses:
For podman deployment:

systemd unit file for starting and managing microshift-containerized.

husky-parul · 2021-11-17T17:48:12Z

podman systemd file was created out of following command.

sudo podman run --sdnotify=container --label io.containers.autoupdate=registry  --network=host --privileged -d \
            --rm --name microshift \
            -v /var/run/crio/crio.sock:/var/run/crio/crio.sock:rw,rshared \
            -v microshift-data:/var/lib/microshift:rw,rshared \
            -v /var/lib/kubelet:/var/lib/kubelet:rw,rshared \
            -v /var/log:/var/log \
            -e KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig \
            quay.io/microshift/microshift:latest

The volumes mounted are:

-v /var/run/crio/crio.sock:/var/run/crio/crio.sock:rw,rshared \
-v microshift-data:/var/lib/microshift:rw,rshared \
-v /var/lib/kubelet:/var/lib/kubelet:rw,rshared \
-v /var/log:/var/log \

packaging/systemd/microshift-containerized created /var/lib/microshift that gets mounted as microshift-data volume.
packaging/systemd/microshift-containerized-configure-selinux configure selinux policies. Both scripts are run in ExecStartPre

husky-parul · 2021-11-18T19:46:11Z

@rootfs @oglok how are these two secrets created? openshift-ingress/router-certs-default and openshift-dns/dns-default-metrics-tls

microshift-containerized is behaving differently (with same selinux configurations) when mounting host-dir vs host-vol

When a host-vol microshift-data is used, pods openshift-dns/dns-default-b5kwm andopenshift-ingress/router-default-6c96f6bc66-4hnf9are failing because secrets openshift-ingress/router-certs-default and openshift-dns/dns-default-metrics-tls are not present in the respective namespaces.
(however, they are present in the name space when host-dir )

so

sudo podman run --sdnotify=container --label io.containers.autoupdate=registry  --network=host --privileged -d \
            --rm --name microshift \
            -v /var/run/crio/crio.sock:/var/run/crio/crio.sock:rw,rshared \
            -v /var/lib/microshift:/var/lib/microshift:rw,rshared \
            -v /var/lib/kubelet:/var/lib/kubelet:rw,rshared \
            -v /var/log:/var/log \
            -e KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig \
            quay.io/microshift/microshift:latest

But this doesn't

sudo podman run --sdnotify=container --label io.containers.autoupdate=registry  --network=host --privileged -d \
            --rm --name microshift \
            -v /var/run/crio/crio.sock:/var/run/crio/crio.sock:rw,rshared \
            -v /var/lib/containers:/var/lib/containers:rw,rshared \
            -v microshift-data:/var/lib/microshift:rw,rshared \
            -v /var/lib/kubelet:/var/lib/kubelet:rw,rshared \
            -v /var/log:/var/log \
            -e KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig \
            quay.io/microshift/microshift:latest

The only difference is host-dir vs host-vol

cooktheryan · 2021-11-18T19:49:32Z

A little more information. There are some subtle differences in the mounts when doing podman inspect microshift

standard directory mount

"/var/lib/microshift:/var/lib/microshift:rw,rshared,rbind",

podman volume

"microshift-data:/var/lib/microshift:rw,rshared,nosuid,nodev,rbind",

husky-parul · 2021-11-24T17:48:20Z

I have narrowed down to the serving secrets dns-default-metrics-tls
and router-certs-default are not created by services dns-default and router-internal-default respectively in podman named volume deployment. Same behavior with selinux on/off. The filesystem, permissions and labels are identical for both host-dir and host-vol.

husky-parul · 2021-11-24T18:15:36Z

cat /run/crun/*/config.json | jq . > /tmp/config_dir.json  # using podman host_dir mount
cat /run/crun/*/config.json | jq . > /tmp/config_vol.json  # using podman named volume mount

diff --color -Naur /tmp/config_dir.json /tmp/config_vol.json 
--- /tmp/config_dir.json        2021-11-24 13:01:39.314831482 -0500
+++ /tmp/config_vol.json        2021-11-24 13:10:36.436976929 -0500
@@ -207,21 +207,11 @@
     "oomScoreAdj": 0
   },
   "root": {
-    "path": "/var/lib/containers/storage/overlay/a9f020ec061f95a33276a901f419e9e97f0d70db8ac1be5984726f5c55393cb8/merged"
+    "path": "/var/lib/containers/storage/overlay/9e08a7c709dca5c09d54c57682dd9890e7510409e75ece238b94647b11dbdc59/merged"
   },
   "hostname": "fedora.local.com",
   "mounts": [
     {
-      "destination": "/proc",
-      "type": "proc",
-      "source": "proc",
-      "options": [
-        "nosuid",
-        "noexec",
-        "nodev"
-      ]
-    },
-    {
       "destination": "/sys",
       "type": "sysfs",
       "source": "sysfs",
@@ -234,6 +224,16 @@
       ]
     },
     {
+      "destination": "/proc",
+      "type": "proc",
+      "source": "proc",
+      "options": [
+        "nosuid",
+        "noexec",
+        "nodev"
+      ]
+    },
+    {
       "destination": "/dev",
       "type": "tmpfs",
       "source": "tmpfs",
@@ -245,28 +245,18 @@
       ]
     },
     {
-      "destination": "/run/.containerenv",
+      "destination": "/etc/resolv.conf",
       "type": "bind",
-      "source": "/run/containers/storage/overlay-containers/25ec623774d9071640cf64a913b59819f01f91ac0e13b1ce60ab1629e731a043/userdata/.containerenv",
+      "source": "/run/containers/storage/overlay-containers/09428ba251ee0e1677a9ba689265f2b6350992a219b3188b0e27ca7a8c550d08/userdata/resolv.conf",
       "options": [
         "bind",
         "rprivate"
       ]
     },
     {
-      "destination": "/var/log",
-      "type": "bind",
-      "source": "/var/log",
-      "options": [
-        "rw",
-        "rprivate",
-        "rbind"
-      ]
-    },
-    {
       "destination": "/dev/shm",
       "type": "bind",
-      "source": "/var/lib/containers/storage/overlay-containers/25ec623774d9071640cf64a913b59819f01f91ac0e13b1ce60ab1629e731a043/userdata/shm",
+      "source": "/dev/shm",
       "options": [
         "bind",
         "rprivate"
@@ -296,9 +286,19 @@
       ]
     },
     {
+      "destination": "/var/log",
+      "type": "bind",
+      "source": "/var/log",
+      "options": [
+        "rw",
+        "rprivate",
+        "rbind"
+      ]
+    },
+    {
       "destination": "/etc/hostname",
       "type": "bind",
-      "source": "/run/containers/storage/overlay-containers/25ec623774d9071640cf64a913b59819f01f91ac0e13b1ce60ab1629e731a043/userdata/hostname",
+      "source": "/run/containers/storage/overlay-containers/09428ba251ee0e1677a9ba689265f2b6350992a219b3188b0e27ca7a8c550d08/userdata/hostname",
       "options": [
         "bind",
         "rprivate"
@@ -307,25 +307,25 @@
     {
       "destination": "/etc/hosts",
       "type": "bind",
-      "source": "/run/containers/storage/overlay-containers/25ec623774d9071640cf64a913b59819f01f91ac0e13b1ce60ab1629e731a043/userdata/hosts",
+      "source": "/run/containers/storage/overlay-containers/09428ba251ee0e1677a9ba689265f2b6350992a219b3188b0e27ca7a8c550d08/userdata/hosts",
       "options": [
         "bind",
         "rprivate"
       ]
     },
     {
-      "destination": "/run/secrets",
+      "destination": "/run/.containerenv",
       "type": "bind",
-      "source": "/run/containers/storage/overlay-containers/25ec623774d9071640cf64a913b59819f01f91ac0e13b1ce60ab1629e731a043/userdata/run/secrets",
+      "source": "/run/containers/storage/overlay-containers/09428ba251ee0e1677a9ba689265f2b6350992a219b3188b0e27ca7a8c550d08/userdata/.containerenv",
       "options": [
         "bind",
         "rprivate"
       ]
     },
     {
-      "destination": "/etc/resolv.conf",
+      "destination": "/run/secrets",
       "type": "bind",
-      "source": "/run/containers/storage/overlay-containers/25ec623774d9071640cf64a913b59819f01f91ac0e13b1ce60ab1629e731a043/userdata/resolv.conf",
+      "source": "/run/containers/storage/overlay-containers/09428ba251ee0e1677a9ba689265f2b6350992a219b3188b0e27ca7a8c550d08/userdata/run/secrets",
       "options": [
         "bind",
         "rprivate"
@@ -345,19 +345,21 @@
       ]
     },
     {
-      "destination": "/var/lib/kubelet",
+      "destination": "/var/lib/microshift",
       "type": "bind",
-      "source": "/var/lib/kubelet",
+      "source": "/var/lib/containers/storage/volumes/microshift-data/_data",
       "options": [
         "rw",
         "rshared",
+        "nosuid",
+        "nodev",
         "rbind"
       ]
     },
     {
-      "destination": "/var/lib/microshift",
+      "destination": "/var/lib/kubelet",
       "type": "bind",
-      "source": "/var/lib/microshift",
+      "source": "/var/lib/kubelet",
       "options": [
         "rw",
         "rshared",
@@ -379,7 +381,7 @@
   ],
   "annotations": {
     "io.container.manager": "libpod",
-    "io.kubernetes.cri-o.Created": "2021-11-24T12:58:30.125590012-05:00",
+    "io.kubernetes.cri-o.Created": "2021-11-24T13:08:06.096394928-05:00",
     "io.kubernetes.cri-o.TTY": "false",
     "io.podman.annotations.autoremove": "TRUE",
     "io.podman.annotations.init": "FALSE",
@@ -399,15 +401,12 @@
         "limit": 2048
       }
     },
-    "cgroupsPath": "machine.slice:libpod:25ec623774d9071640cf64a913b59819f01f91ac0e13b1ce60ab1629e731a043",
+    "cgroupsPath": "machine.slice:libpod:09428ba251ee0e1677a9ba689265f2b6350992a219b3188b0e27ca7a8c550d08",
     "namespaces": [
       {
         "type": "pid"
       },
       {
-        "type": "ipc"
-      },
-      {
         "type": "uts"
       },
       {
[root@fedora ~]#

mangelajo · 2021-11-25T11:29:04Z

Suggested change

ExecStartPre=/bin/bash /usr/bin/microshift-containerized-configure-selinux # not autogenerated

ExecStartPre=/bin/bash /usr/bin/microshift-containerized-configure-selinux # not autogenerated

We should not do this, it's a lengthy process. ( installing, compiling, etc...)

Let's assume that people will download the rpms for microshift via dnf, which will include both microshift-containerized + the selinux modules compiled.

We can keep the script available just in case.

Suggested change

ExecStartPre=/bin/bash /usr/bin/microshift-containerized-configure-selinux # not autogenerated

https://github.com/redhat-et/microshift/blob/main/packaging/rpm/microshift.spec#L226

mangelajo · 2021-11-25T11:30:10Z

This still has

-v /var/lib/microshift:/var/lib/microshift:rw,rshared
right?

Yes. We are going ahead with /var/lib/microshift:/var/lib/microshift:rw,rshared instead of microshift-data:/var/lib/microshift:rw,rshared

mangelajo

see comments.

husky-parul · 2021-12-01T18:13:07Z

@mangelajo Ready for review

fzdarsky · 2021-12-01T18:13:25Z

Can we fold this file completely into the microshift-containerized.service, such that the UX is like we document in https://microshift.io/docs/getting-started/ ?

fzdarsky · 2021-12-01T18:26:49Z

Is this file still needed / up-to-date?

If we're running containerized, only the crio.sock should be bind-mounted from the host (plus the /var/lib/microshift datadir if we're using hostdir instead of voliumes). So why do we still need to create these other dirs and adjust SELinux policies?

We don't need to adjust the selinux anymore. From @mangelajo comment
"Let's assume that people will download the rpms for microshift via dnf, which will include both microshift-containerized + the selinux modules compiled."
.It was based on the assumption that the users didn't download the rpms and would be only using the microshift-containerized.service. I will remove the files as well.

Wait... I think we're confusing deployment models here:
The containerised deployment is for users who want to deploy MIcroShift at runtime on top of a R4E container host. That is, they cannot dynamically install an .RPM (that would need to be baked into the rpm-ostree).

Yes, they can't dynamically install it, but if you want to bake it, then we need it also to be in the RPM?.

In any case I don't think we should be running make for the selinux files on installation, or downloading seliux-policy-devel etc... in that case we should probably provide the selinux compiled policy, if that's standard across kernel versions, which I don't know.

May be the container itself should be built by installing the RPM we have?, then the user just downloads the .service file? Can we install an selinux policy from inside a privileged container?

According to this, can’t just people install an rpm? We made a subpackage for the microshift container service which will also pull the selinux defs: https://developers.redhat.com/blog/2020/03/12/how-to-customize-fedora-coreos-for-dedicated-workloads-with-ostree

What I meant is we have two deployment models:

The .rpm-based installation, which is for embedding into rpm-ostrees and runs MicroShift as binary on the host / non-containerized. We have corresponding SELinux rules for that.

The containerized deployment, which is for users who don't want to embed into an rpm-ostree but instead use a container host and add/remove MicroShift at runtime - without reboot. This is where it doesn't make sense to install the sytemd unit via an .rpm. Instead, it'll need to be created via configuration management resp. from the application management agent.

copejon · 2021-12-01T22:36:58Z

/ok-to-test

husky-parul · 2021-12-06T18:42:54Z

@fzdarsky consolidated everything in microshift-containerized.service
needs ok to test

cooktheryan · 2022-01-14T14:05:00Z

/retest

husky-parul · 2022-01-14T14:50:29Z

/retest

cooktheryan · 2022-01-17T14:18:43Z

/retest

husky-parul · 2022-01-17T19:06:35Z

/retest

fzdarsky · 2022-01-18T10:01:45Z

@cooktheryan From what I gathered from the community meeting, the systemd file is not supposed to do anything except for starting and managing the lifecycle so I am moving the mkdir -p /var/libe/kubelet to docs.

@husky-parul I think having this in the ExecStartPre instead of the doc is fine as it's part of the pre-reqs for starting.
Anyway, let's get this merged and possibly add in a follow-up?

/lgtm

Signed-off-by: Parul <parsingh@redhat.com>

husky-parul · 2022-01-18T16:23:05Z

/retest

fzdarsky · 2022-01-18T16:30:09Z

/lgtm

fzdarsky · 2022-01-18T16:30:40Z

/ok-to-test

husky-parul · 2022-01-18T18:44:37Z

/retest

husky-parul · 2022-01-18T19:35:28Z

/retest

husky-parul · 2022-01-18T22:15:00Z

/retest

husky-parul · 2022-01-19T02:18:13Z

/retest

cooktheryan · 2022-01-19T04:09:55Z

/retest

husky-parul · 2022-01-19T19:34:38Z

/retest

cooktheryan · 2022-01-20T01:21:05Z

/LGTM

openshift-ci · 2022-01-20T01:21:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cooktheryan, fzdarsky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [cooktheryan,fzdarsky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cooktheryan · 2022-01-20T01:21:13Z

Woot woot!!!!!

openshift-ci Bot requested review from copejon and rootfs November 15, 2021 19:56

husky-parul added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 15, 2021

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 15, 2021

husky-parul marked this pull request as draft November 15, 2021 20:33

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 15, 2021

husky-parul force-pushed the containerized branch from 9a3a23e to 663af0e Compare November 17, 2021 17:40

mangelajo reviewed Nov 25, 2021

View reviewed changes

Comment thread packaging/systemd/microshift-containerized-configure-selinux Outdated

mangelajo suggested changes Nov 25, 2021

View reviewed changes

husky-parul added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Dec 1, 2021

husky-parul marked this pull request as ready for review December 1, 2021 18:12

husky-parul requested a review from mangelajo December 1, 2021 18:12

fzdarsky reviewed Dec 1, 2021

View reviewed changes

openshift-ci Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 1, 2021

husky-parul force-pushed the containerized branch from 3463749 to fec91b1 Compare December 6, 2021 18:41

husky-parul requested a review from fzdarsky December 6, 2021 18:43

husky-parul assigned fzdarsky Dec 6, 2021

husky-parul added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Dec 6, 2021

husky-parul force-pushed the containerized branch from 4079fda to 9484ba1 Compare January 17, 2022 20:49

openshift-ci Bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 18, 2022

microshift containerized with volume mount

395b74e

Signed-off-by: Parul <parsingh@redhat.com>

husky-parul force-pushed the containerized branch from 9484ba1 to 395b74e Compare January 18, 2022 13:37

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 18, 2022

create /var/lib/kubelet

f88d0fb

Signed-off-by: Parul <parsingh@redhat.com>

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jan 18, 2022

openshift-ci Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 18, 2022

openshift-ci Bot assigned cooktheryan Jan 20, 2022

husky-parul merged commit 13c410a into openshift:main Jan 20, 2022

sallyom mentioned this pull request Jan 21, 2022

update kubeconfig cp command from container microshift-io/microshift-documentation#116

Merged

	ExecStartPre=/bin/bash /usr/bin/microshift-containerized-configure-selinux # not autogenerated
	ExecStartPre=/bin/bash /usr/bin/microshift-containerized-configure-selinux # not autogenerated

Uh oh!

Conversation

husky-parul commented Nov 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

husky-parul commented Nov 17, 2021

Uh oh!

husky-parul commented Nov 18, 2021

Uh oh!

cooktheryan commented Nov 18, 2021

Uh oh!

husky-parul commented Nov 24, 2021

Uh oh!

husky-parul commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mangelajo left a comment

Choose a reason for hiding this comment

Uh oh!

husky-parul commented Dec 1, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

husky-parul Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

copejon commented Dec 1, 2021

Uh oh!

husky-parul commented Dec 6, 2021

Uh oh!

cooktheryan commented Jan 14, 2022

Uh oh!

husky-parul commented Jan 14, 2022

Uh oh!

cooktheryan commented Jan 17, 2022

Uh oh!

husky-parul commented Jan 17, 2022

Uh oh!

fzdarsky commented Jan 18, 2022

Uh oh!

husky-parul commented Jan 18, 2022

Uh oh!

fzdarsky commented Jan 18, 2022

Uh oh!

fzdarsky commented Jan 18, 2022

Uh oh!

husky-parul commented Jan 18, 2022

Uh oh!

husky-parul commented Jan 18, 2022

Uh oh!

husky-parul commented Jan 18, 2022

Uh oh!

husky-parul commented Jan 19, 2022

Uh oh!

cooktheryan commented Jan 19, 2022

Uh oh!

husky-parul commented Jan 19, 2022

Uh oh!

cooktheryan commented Jan 20, 2022

Uh oh!

husky-parul commented Nov 15, 2021 •

edited

Loading

husky-parul commented Nov 24, 2021 •

edited

Loading

husky-parul Dec 1, 2021 •

edited

Loading