#101 MLCube and Podman
Closed: scheduled 2 years ago by glb. Opened 2 years ago by fed500.

Possible draft below, would need updating for SELinux. Other suggestions welcome:

MLCubes and Podman

MLCubes are a new open source container based infrastructure introduced to enable reproducibility in Python based machine learning workflows. They can utilize either Podman, Singularity or Docker. Execution of MLCubes on remote platforms is also supported. One of the chairs of the MLCommons Best Practices working group that is developing MLCubes is Diane Feddema from RedHat.

Yazan Monshed has written a very
helpful introduction to Podman on Fedora which gives more details on some of the steps used here.

To use MLCube on Fedora Linux with Podman, first install the necessary dependencies

sudo dnf -y update
sudo dnf -y install podman git virtualenv

Setup a soft link to alias Podman to Docker

cd /usr/bin
sudo ln -s podman docker
cd $HOME

Then following the documentation, setup a virtual environment and get the example code

virtualenv -p python3 ./env_mlcube && source ./env_mlcube/bin/activate
git clone https://github.com/mlcommons/mlcube_examples.git && cd ./mlcube_examples/hello_world
mlcube describe

Next get the container

mlcube configure --mlcube=. --platform=docker

To run the tests, you may need to set SELinux permissions in the directories appropriately or disable SELinux.

sudo setenforce 0

Then run the hello world example

mlcube run --mlcube=. --task=hello --platform=docker
mlcube run --mlcube=. --task=bye --platform=docker

Check the output

cat ./workspace/chats/chat_with_alice.txt

you should find text similar to

Hi, Alice! Nice to meet you.
Bye, Alice! It was great talking to you.

Then restore SElinux permissions if you temporarily disabled them

sudo setenforce 1

There is a further documented example of using an MLCube to classify MNIST digits - this will download and install Tensorflow for you and so require more disc space than the hello world example. In addition you can create your own MLCubes as described here. You can also contribute further examples in the MLCube examples repository.

https://discussion.fedoraproject.org/t/mlcubes-and-podman/38144


Metadata Update from @fed500:
- Issue assigned to fed500

2 years ago

Metadata Update from @fed500:
- Issue tagged with: article, needs-image

2 years ago

Metadata Update from @fed500:
- Custom field preview-link adjusted to https://fedoramagazine.org/?p=36283&preview=true&preview_id=36283

2 years ago

Hi @fed500.

We don't normally like for users to edit files under /usr/bin if avoidable. It looks like there is a package called podman-docker that will take care of setting up the docker → podman redirection (though it does it with a shell script rather than a symlink). Does MLCube work if you add podman-docker to your install list instead of creating the symlink?

As for SELinux, we would really like to see that left enabled if possible as well. I see that someone has reported that this might be as simple as adding :z to some volume mounts.

https://github.com/mlcommons/mlcube/issues/205

Is a patch for SELinux something that could be implemented before this article is published?

Thanks.

@glb Thanks for the feed back. An updated draft is available in Wordpress which:
- modifies the configuration script to use podman command
- gives permissions to the folder directly to enable writing
There is a nice post that discusses volume labels, https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/ , but this presentation introduces Udica which maybe more useful since one may want customized access policies for each container. This is probably a different article.

It may also be helpful to indicate the commit in the repository that is being used as a way of getting community feedback on what they would find useful to have in future releases - a one size fits all approach does not work in production settings, so making adaptation easy and intuitive to have good data hygiene would be good.

Hi Benson.

I just made a quick review of your article and I feel like it is now spending too much time on SELinux for the size of the article. I think it would be better to reduce all of that to just the following two lines and then refer the user to a slightly more authoritative resource for an explanation of SELinux and Containers (i.e. https://opensource.com/article/18/2/selinux-labels-container-runtimes).

$ sudo semanage fcontext -a -t container_file_t "$PWD/workspace(/.*)?"
$ sudo restorecon -Rv $PWD/workspace

Note that in the above example, I explicitly set the desired SELinux type label rather than copying one from another file system location. I think being explicit about what is being changed and how it is being changed is better than having the user just copy whatever happens to be set on some other part of the file system. Also, -F should not be necessary on restorecon in this case since only the type label is being changed.

This is just my opinion. You could certainly write a more thorough explanation of SELinux and how it works with containers. But if you want to do that, I'd rather see that written as a separate article, published before this one, and then referenced in this article rather than intermixing the two subject matters in the one article.

Thanks.

Metadata Update from @glb:
- Custom field editor adjusted to glb

2 years ago

Thanks for the reference and the suggestions. Will update the article
accordingly. Probably a further article on Udica would be helpful at
some point, but this would take some time, and maybe raising the issue
here is helpful to encourage further discussions in these projects. Note
that the correct full label is system_u:object_r:container_file_t:s0
not unconfined_u:object_r:container_file_t:s0 so would need to use

$ sudo semanage fcontext -a -s system_u -t container_file_t 
"$PWD/workspace(/.*)?"
$ sudo restorecon -Rv $PWD/workspace

or

$ sudo semanage fcontext -m -s system_u -t container_file_t 
"$PWD/workspace(/.*)?"
$ sudo restorecon -RFv $PWD/workspace

but will test and update accordingly.

Updated the article in Wordpress. Steps only work on x86_64 because the container at https://hub.docker.com/r/mlcommons/hello_world/tags is built for AMD64. restorecon -RFv requires the F flag to be able to change the user label to system_u

Thanks for your suggestions. Made further updates. It should now work on Arm. It seems system_u tag is not necessary. Let me know if any further suggestions.

Metadata Update from @glb:
- Custom field image-editor adjusted to fed500
- Custom field publish adjusted to 2022-04-18

2 years ago

Hey fed500. Your cover image, in principle, is fine. But the dimensions need to be 1890x800 px and it needs to be in jpg format. Also, the main text of the image should be centered so as not to be clipped when the image is rendered on some third-party feed readers.

Thanks.

Thanks for the feedback. Updated the image.

Hi Benson.

I've made a pass at editing your article for publication. Mostly I just tweaked the formatting to use preformatted blocks or the "keyboard input" typeset instead of "code" blocks so that things render properly with the Fedora Magazine theme. I also reworded the excerpt a little bit to provide a short definition of what MLCube is.

You can still make small revisions if you need. Otherwise, if you are OK with this, it will go out tomorrow at 08:00 UTC.

Thanks!

Thanks. Made a few changes to improve readability.

Metadata Update from @fed500:
- Issue untagged with: needs-image

2 years ago

LGTM. I've scheduled this to go out at 08:00 UTC 2022-04-18.

Thanks!

Issue status updated to: Closed (was: Open)
Issue close_status updated to: scheduled

2 years ago

Login to comment on this ticket.

Metadata