The current experience with developing libtaskotron and testing it on disposable clients if far from great. There are two major issues: a) waiting time For every round trip (running the task and seeing the results) you need to wait at least several minutes before the client is initialized again, and again, and again. The waiting time mostly consists of initializing dnf metadata, and then downloading and installing required packages. Even with high-speed download speeds, it takes so long that you want to bash your computer unconscious. b) no convenient way to transfer modified source code to the disposable client libtaskotron supports runtask --patch. The problem is that for every little change, you need to create the patch manually (provided you figure out how, I failed). Another problem is that this doesn't apply to other projects (let's say you need to patch testcloud or python-fedora, it's not that unusual). You can also create your own RPMs and add a repository for them in the disposable client config (not sure if we support it yet), but that again involves a long series of manual steps (commiting, bumping up specfile, bumping up python files, building RPM, uploading) and it's again even more inconvenient to modify other libraries than libtaskotron. We currently even don't have any option to pause between client set up and task execution, so that you can install your modified library version. In short, people will rather release untested code than go through all this pain to test it.
runtask --patch
We need to do something about it.
Idea 1: Reduce dependency download tree This is covered in #605. However, this will help only partially and will probably still take too long for the common human patience threshold.
Idea 2: Use custom images with dependencies pre-installed This is currently blocked on T568 (or until we implement direct image booting in testcloud without any need for cloud-init). When it starts working, it will most probably drastically reduce the waiting time. dnf still downloads all metadata when checking whether a package is installed, and we might want to optimize that somehow. However, this does not help when you need to run your custom code (not the officially provided libtaskotron version), you still need the option to patch it somehow. This will also not help installing any new dependencies your patch introduced.
Idea 3: Introduce an option to pause between disposable client spawning and task execution, so that people can use their-preferred way of delivering updated code to clients This will be most probably necessary, because no matter what magical solution we come up with (if we do), we'll not cover all use cases and people will need to manually adjust it from time to time. This can be in a form of pausing before execution, or having something like runtask --create-disposable-client, doing what we need to do, and making use of runtask --ssh ip. This still does not solve the ease of use, it just allows you to do it manually.
runtask --create-disposable-client
runtask --ssh ip
Idea 4: Automate code deployment with the taskotron_development profile There has been an idea of creating a new taskotron profile (the current ones are development and production, something like taskotron_development). If this profile was active, we would have some strong assumptions that would make possible to automate certain tasks. For example, we would assume we run from a git checkout. In that case we should be theoretically able to automatically build a new RPM and install it onto the client. Building RPMs is slow, but it's better than the current state. We could also try to copy the current git checkout instead of building RPMs, and install it on the client. That would be fast, and it wouldn't even require you to commit your changes. Since we will have access to specfile and requirements.py, we could even install any recently-added deps, update packages to match development versions, etc. We can basically set up exactly the same environment as the developer has. This approach seems promising, but it isn't well thought out yet and it includes creation of another taskotron profile, adding to confusion of people studying the documentation and the config file.
development
production
taskotron_development
requirements.py
If you have further ideas, please add them here. Or dispute the current ones, both is welcome :) Once we decide what we want to do, let's file tickets for particular tasks. Thanks.
== Waiting Time == I'm open to suggestions on how to solve this locally for devs. I've been working on creating updated, custom-ish images that make things go significantly faster in dev/stg/production. I have "figure out if dnf metadata can be preloaded on the images" on my todo list but haven't gotten to it.
We could look into making those images available publicly, I suppose. Also, D581 is a PoC for breaking libtaskotron up into multiple subpackages that would be quicker to install.
== Client Modification == I never wrote anything down about this but one of the things that I've had in the back of my mind is to support running something like an ansible playbook on clients before they start executing anything.
I strongly suspect that one of the first features requested of us will be "I need to change X in the client in order to do Y, please make it easier to do that" so I'd almost rather.
That being said it's possible to get cloud-init ([[http://cloudinit.readthedocs.org/en/latest/|cloud-init docs]]) to help some here. It's possible to modify the testcloud USER_DATA to add repos, add files, change users, install packages, run commands etc. - one route that's not too bad (for the short term, anyways. not ideal) would be to build the rpms that need changing, put them in a local dir, run createrepo on that dir, host it with python -m SimpleHTTPServer and add the repo definition to cloud-init.
USER_DATA
python -m SimpleHTTPServer
After talking to @kparal earlier today, we agreed that this was done enough for deploying disposable clients and ideas 3 and 4 could be split off into separate enhancement tickets to be debated.
Idea 3: #675 Idea 4: #674
b) no convenient way to transfer modified source code to the disposable client libtaskotron supports runtask --patch. The problem is that for every little change, you need to create the patch manually (provided you figure out how, I failed). >Another problem is that this doesn't apply to other projects (let's say you need to patch testcloud or python-fedora, it's not that unusual). You can also create >your own RPMs and add a repository for them in the disposable client config (not sure if we support it yet), but that again involves a long series of manual steps >(commiting, bumping up specfile, bumping up python files, building RPM, uploading) and it's again even more inconvenient to modify other libraries than >libtaskotron. We currently even don't have any option to pause between client set up and task execution, so that you can install your modified library version. In >short, people will rather release untested code than go through all this pain to test it. I 'm thinking maybe we can use the method I mentioned in #675:let libtasktron create VM on top of the tailored image.We only need to add a test_code option for libtaskotron,and modify the code in minion.py a little :
_run (self): cmdline = python_utils.reverse_argparse(self.arg_data['_orig_args'], ignore=self.arg_data_exclude) cmdline.append('--local') # be paranoid and escape everything cmdline = [pipes.quote(elem) for elem in cmdline] if not self.arg_data['test_code']: task = os.path.basename(self.arg_data['task']) run_cmd = ['cd', pipes.quote(self.taskdir), '&&', 'runtask'] elif self.arg_data['test_code']: # we need cd /root/libtaskotron first,so I prefer to change the task argument instead of making run_cmd too longer task = os.path.join(self.taskdir,os.path.basename(self.arg_data['task'])) run_cmd = ['cd', '/root/libtaskotron', '&&', 'source','env_taskotron/bin/activate','&&','runtask'] cmdline.append(task) task_cmd = run_cmd + cmdline log.info('Executing the task on the minion...') self.exitcode = self.ssh.cmd(' '.join(task_cmd)) log.info('Task execution on the minion is complete.')
And for users,they only need to create a base image with testcloud,and do https://bitbucket.org/fedoraqa/libtaskotron .Whenever they want to use their libtaskotron(or testcloud,or others)on disposbale VM,they can simply scp their code to the base image,and then do the normal runtask with VM created on top of the base image.
Metadata Update from @kparal: - Issue marked as depending on: #283
Login to comment on this ticket.