Possible draft below, would need updating for SELinux. Other suggestions welcome:
MLCubes are a new open source container based infrastructure introduced to enable reproducibility in Python based machine learning workflows. They can utilize either Podman, Singularity or Docker. Execution of MLCubes on remote platforms is also supported. One of the chairs of the MLCommons Best Practices working group that is developing MLCubes is Diane Feddema from RedHat.
Yazan Monshed has written a very helpful introduction to Podman on Fedora which gives more details on some of the steps used here.
To use MLCube on Fedora Linux with Podman, first install the necessary dependencies
sudo dnf -y update sudo dnf -y install podman git virtualenv
Setup a soft link to alias Podman to Docker
cd /usr/bin sudo ln -s podman docker cd $HOME
Then following the documentation, setup a virtual environment and get the example code
virtualenv -p python3 ./env_mlcube && source ./env_mlcube/bin/activate git clone https://github.com/mlcommons/mlcube_examples.git && cd ./mlcube_examples/hello_world mlcube describe
Next get the container
mlcube configure --mlcube=. --platform=docker
To run the tests, you may need to set SELinux permissions in the directories appropriately or disable SELinux.
sudo setenforce 0
Then run the hello world example
mlcube run --mlcube=. --task=hello --platform=docker mlcube run --mlcube=. --task=bye --platform=docker
Check the output
cat ./workspace/chats/chat_with_alice.txt
you should find text similar to
Hi, Alice! Nice to meet you. Bye, Alice! It was great talking to you.
Then restore SElinux permissions if you temporarily disabled them
sudo setenforce 1
There is a further documented example of using an MLCube to classify MNIST digits - this will download and install Tensorflow for you and so require more disc space than the hello world example. In addition you can create your own MLCubes as described here. You can also contribute further examples in the MLCube examples repository.
https://discussion.fedoraproject.org/t/mlcubes-and-podman/38144
Metadata Update from @fed500: - Issue assigned to fed500
Metadata Update from @fed500: - Issue tagged with: article, needs-image
Metadata Update from @fed500: - Custom field preview-link adjusted to https://fedoramagazine.org/?p=36283&preview=true&preview_id=36283
Hi @fed500.
We don't normally like for users to edit files under /usr/bin if avoidable. It looks like there is a package called podman-docker that will take care of setting up the docker → podman redirection (though it does it with a shell script rather than a symlink). Does MLCube work if you add podman-docker to your install list instead of creating the symlink?
podman-docker
As for SELinux, we would really like to see that left enabled if possible as well. I see that someone has reported that this might be as simple as adding :z to some volume mounts.
:z
https://github.com/mlcommons/mlcube/issues/205
Is a patch for SELinux something that could be implemented before this article is published?
Thanks.
@glb Thanks for the feed back. An updated draft is available in Wordpress which: - modifies the configuration script to use podman command - gives permissions to the folder directly to enable writing There is a nice post that discusses volume labels, https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/ , but this presentation introduces Udica which maybe more useful since one may want customized access policies for each container. This is probably a different article.
Metadata Update from @fed500: - Custom field preview-link adjusted to https://fedoramagazine.org/?p=36283&preview=1&_ppp=4173639534 (was: https://fedoramagazine.org/?p=36283&preview=true&preview_id=36283)
It may also be helpful to indicate the commit in the repository that is being used as a way of getting community feedback on what they would find useful to have in future releases - a one size fits all approach does not work in production settings, so making adaptation easy and intuitive to have good data hygiene would be good.
Hi Benson.
I just made a quick review of your article and I feel like it is now spending too much time on SELinux for the size of the article. I think it would be better to reduce all of that to just the following two lines and then refer the user to a slightly more authoritative resource for an explanation of SELinux and Containers (i.e. https://opensource.com/article/18/2/selinux-labels-container-runtimes).
$ sudo semanage fcontext -a -t container_file_t "$PWD/workspace(/.*)?" $ sudo restorecon -Rv $PWD/workspace
Note that in the above example, I explicitly set the desired SELinux type label rather than copying one from another file system location. I think being explicit about what is being changed and how it is being changed is better than having the user just copy whatever happens to be set on some other part of the file system. Also, -F should not be necessary on restorecon in this case since only the type label is being changed.
-F
restorecon
This is just my opinion. You could certainly write a more thorough explanation of SELinux and how it works with containers. But if you want to do that, I'd rather see that written as a separate article, published before this one, and then referenced in this article rather than intermixing the two subject matters in the one article.
Metadata Update from @glb: - Custom field editor adjusted to glb
Thanks for the reference and the suggestions. Will update the article accordingly. Probably a further article on Udica would be helpful at some point, but this would take some time, and maybe raising the issue here is helpful to encourage further discussions in these projects. Note that the correct full label is system_u:object_r:container_file_t:s0 not unconfined_u:object_r:container_file_t:s0 so would need to use
system_u:object_r:container_file_t:s0
unconfined_u:object_r:container_file_t:s0
$ sudo semanage fcontext -a -s system_u -t container_file_t "$PWD/workspace(/.*)?" $ sudo restorecon -Rv $PWD/workspace
or
$ sudo semanage fcontext -m -s system_u -t container_file_t "$PWD/workspace(/.*)?" $ sudo restorecon -RFv $PWD/workspace
but will test and update accordingly.
Updated the article in Wordpress. Steps only work on x86_64 because the container at https://hub.docker.com/r/mlcommons/hello_world/tags is built for AMD64. restorecon -RFv requires the F flag to be able to change the user label to system_u
restorecon -RFv
F
system_u
Thanks for your suggestions. Made further updates. It should now work on Arm. It seems system_u tag is not necessary. Let me know if any further suggestions.
Metadata Update from @glb: - Custom field image-editor adjusted to fed500 - Custom field publish adjusted to 2022-04-18
Hey fed500. Your cover image, in principle, is fine. But the dimensions need to be 1890x800 px and it needs to be in jpg format. Also, the main text of the image should be centered so as not to be clipped when the image is rendered on some third-party feed readers.
Thanks for the feedback. Updated the image.
I've made a pass at editing your article for publication. Mostly I just tweaked the formatting to use preformatted blocks or the "keyboard input" typeset instead of "code" blocks so that things render properly with the Fedora Magazine theme. I also reworded the excerpt a little bit to provide a short definition of what MLCube is.
You can still make small revisions if you need. Otherwise, if you are OK with this, it will go out tomorrow at 08:00 UTC.
Thanks!
Metadata Update from @glb: - Custom field preview-link adjusted to https://fedoramagazine.org/?p=36283&preview=true&preview_id=36283&_thumbnail_id=36348 (was: https://fedoramagazine.org/?p=36283&preview=1&_ppp=4173639534)
Metadata Update from @glb: - Custom field preview-link adjusted to https://fedoramagazine.org/?p=36283&preview=true (was: https://fedoramagazine.org/?p=36283&preview=true&preview_id=36283&_thumbnail_id=36348)
Thanks. Made a few changes to improve readability.
Metadata Update from @fed500: - Issue untagged with: needs-image
LGTM. I've scheduled this to go out at 08:00 UTC 2022-04-18.
Issue status updated to: Closed (was: Open) Issue close_status updated to: scheduled
Login to comment on this ticket.