#1717 buildbot for upstream python code
Closed: Fixed None Opened 10 years ago by dmalcolm.

Caveat: I'm about to become unreachable via email until October 12th (vacation), and I won't be able to actually start the project until then. I'm filing this request now to give you a heads-up, and to get a sense of the feasibility of the project. Hope that's OK.

= Project Sponsor =

Project Team (FAS Names): dmalcolm[[BR]]
Infrastructure Sponsor:

= Project Info =

Project Name: buildbot for upstream python code [[BR]]
Target Audience: Upstream python developers (python.org) [[BR]]
Expiration/Delivery Date (required): initial trial to last until end of January 2010[[BR]]
== Description/Summary: ==
Development of Python on python.org uses a collection of buildbots to perform continuous integration of the python source code: monitor their svn repository for checkins, checkout the code, build and test. (See http://www.python.org/dev/buildbot/ )

Unfortunately, recently many of the hosts involved have been down/unresponsive for one reason or another. (See this thread:
http://mail.python.org/pipermail/python-dev/2009-September/092005.html )

In my $DAYJOB I'm about to be spending a lot of time on Python itself, and may be able to devote some of my $DAYJOB time to keeping an eye on a farm of buildbots.

Would the Fedora infrastructure be able to carve out some virtual machines that I could use to set up build bots?

Ideally I'd want to set up building of various projects (Python 2.6,
Python 2.7 and Python 3.1) across:
- latest released Fedora
- RHEL 5
on all architectures we care about.

However, it's best to start small, trying it out on one vm, and building it out once it's working.

== Benefit to Fedora ==
- ensure that upstream Python always builds on Fedora
- improved visibility for Fedora within the upstream Python community
- assist with upstream release of Python 2.7

== Project plan (Detailed) ==

Initial phase:
- I'd want ssh on a Fedora 11 or RHEL virtual host, with sudo
- I'd figure out a way to deploy the buildbot worker code and config via puppet (buildbot is in Fedora; see:
https://admin.fedoraproject.org/pkgdb/packages/name/buildbot )
- establish contact with admins of the python.org buildbot server
- get it running
- start the ongoing work of watching it, keeping an eye on build failures (answering the question "is it the code, or is it a problem with the box?")

I'd be doing the above jobs, but I'm not able to devote time to this until October 12th. I'm filing this now to get a sense of the
feasibility of the project.

I would be primary contact, responsible for this project.

== Goals ==
- have at least one Fedora/RHEL box as an active and stable buildbot worker within the python.org buildbot farm, serving at least one source branch, by end of January 2010
- assuming the above is running OK, build out to support more source branches, and more hosts


I can't figure out if this belongs in QA or not but I'm going to cc them.

As you stated above, I don't think this would fall to QA to own the outcome of the builds, but the QA team is working on some infrastructure components that would provide the scheduling and results framework that would facilitate running these builds/tests on a repeatable basis. The project is in it's early stages and focused specifically on provided daily automated testing of rawhide (repo + install source).

The target date you've established (Jan 2010) seems feasible with the goals of the AutoQA project. For more information on AutoQA, see https://fedoraproject.org/wiki/AutoQA. If you have questions or would like to discuss plans/next_steps ... please feel free to get in touch with wwoods, myself ... or join an upcoming QA meeting (see https://fedoraproject.org/wiki/QA/Meetings).

Some things to consider ...

  1. The first step when integrating a test into AutoQA is to write the test.
  2. Once a test is available and can be run successfully by hand, the next step is determining how and when your test will run. Keep it simple ... use wget, or poll a mailing list etc...
  3. Once you've determined how/when to run the test, consider writing an AutoQA hook (see https://fedoraproject.org/wiki/Writing_AutoQA_Hooks)

(back from vacation)

My proposal covers a different area than Fedora AutoQA, and I'm coming at this from an upstream development perspective, not from a Fedora QA perspective.

I don't see this as an AutoQA project, here's why:

In AutoQA, you are running a set of tests against new Fedora RPMs. The software under test are candidate versions of Fedora RPMs, and you are reporting results via autoqa. The objective of your testing is to detect various kinds of breakage of Fedora RPMs earlier, in a controlled way.

I have a different objective. I'm requesting a guest in order to build and test upstream python.org SVN checkins. This is currently being done by upstream for various configurations: Windows XP, Ubuntu, etc. I want to add Fedora and RHEL to those configurations. Upstream already has a particular system for doing this kind of testing, I "merely" want to slot in an additional system into their framework.

From this perspective, the software under test is a specific branch of upstream python.org SVN; the results would be reported back to upstream. The objective of the testing is to detect upstream breakage earlier, and thus to improve Fedora's relevance and standing within the upstream community.

I would deploy the Fedora/EPEL guest with a specific configuration/set of RPMs; changes to the configuration/RPMs are a big event, and I would prefer to only deploy security fixes, as needed. There's a chance that we might detect Fedora breakage as a by-product of this process, but if we do, it's a sign that more coverage needs to be added to the Fedora QA process.

To put it another way: we have two changing things here: Fedora RPMs, and the upstream python SVN. AutoQA is aimed at testing the former. I want to test the latter. Having both change is a recipe for madness: one should only vary one variable at a time.

Replying to [comment:3 dmalcolm]:

(back from vacation)

My proposal covers a different area than Fedora AutoQA, and I'm coming at this from an upstream development perspective, not from a Fedora QA perspective.

I don't see this as an AutoQA project, here's why:

In AutoQA, you are running a set of tests against new Fedora RPMs. The software under test are candidate versions of Fedora RPMs, and you are reporting results via autoqa. The objective of your testing is to detect various kinds of breakage of Fedora RPMs earlier, in a controlled way.

I have a different objective. I'm requesting a guest in order to build and test upstream python.org SVN checkins. This is currently being done by upstream for various configurations: Windows XP, Ubuntu, etc. I want to add Fedora and RHEL to those configurations. Upstream already has a particular system for doing this kind of testing, I "merely" want to slot in an additional system into their framework.

AutoQA actually isn't much about testing and comparing packaging just yet. We are working towards that for a future milestone. For now, AutoQA involves provisioning a virtual guest to perform some functional verification of rawhide. In the future, we hope to extend it to post-build package-level verification.

From this perspective, the software under test is a specific branch of upstream python.org SVN; the results would be reported back to upstream. The objective of the testing is to detect upstream breakage earlier, and thus to improve Fedora's relevance and standing within the upstream community.

You are correct, having this performed upstream would be better.

I would deploy the Fedora/EPEL guest with a specific configuration/set of RPMs; changes to the configuration/RPMs are a big event, and I would prefer to only deploy security fixes, as needed. There's a chance that we might detect Fedora breakage as a by-product of this process, but if we do, it's a sign that more coverage needs to be added to the Fedora QA process.

To put it another way: we have two changing things here: Fedora RPMs, and the upstream python SVN. AutoQA is aimed at testing the former. I want to test the latter. Having both change is a recipe for madness: one should only vary one variable at a time.

Whichever you feel is the path of least resistance. Having it upstream and closer to the code is always a good thing. However, if you're not having luck there and need an interim solution. I believe we can satisfy your needs.

Thanks for the information. However, I want to stay close to upstream here, so I think AutoQA is a tangent.

What I'm really looking for here is ssh and sudo on a dedicated guest. How reasonable is the following?
- Fedora 11
- i386 (x86_64 would be better)
- 1GB of RAM (more would be better)
- 1 CPU (more would be better)
- 10 GB of disk space

Thanks.

Is this a reasonable request, or should I seek alternatives?

(6 months went by, I'd want Fedora 12, not 11 now)

(another 6 months); I'd want Fedora 13, not 12 now

We chatted about this on IRC on 2010-09-16. Let me know if/when a guest becomes available for this, or if you need more information from me. Thanks!

Feel free to utilize my Hudson instance for this if you want: http://ci.csh.rit.edu
It's a Xen guest on a Red Hat owned server at RIT, running RHEL5 with Python 2.4, 2.6 and 2.7.

Replying to [comment:10 lmacken]:

Feel free to utilize my Hudson instance for this if you want: http://ci.csh.rit.edu
It's a Xen guest on a Red Hat owned server at RIT, running RHEL5 with Python 2.4, 2.6 and 2.7.

Luke: is this offer still open? I'd be setting up a buildbot worker on it, so it would be a different user/process to the Hudson instance. I'd need to coordinate with the upstream python buildbot folks (e.g. I need to know about outages and upgrades).

(as it turns out the py3k branch seems to be currently broken on RHEL5; see http://bugs.python.org/issue10517 , so having CI for this would be very useful).

Replying to [comment:11 dmalcolm]:

Replying to [comment:10 lmacken]:

Feel free to utilize my Hudson instance for this if you want: http://ci.csh.rit.edu
It's a Xen guest on a Red Hat owned server at RIT, running RHEL5 with Python 2.4, 2.6 and 2.7.

Luke: is this offer still open? I'd be setting up a buildbot worker on it, so it would be a different user/process to the Hudson instance. I'd need to coordinate with the upstream python buildbot folks (e.g. I need to know about outages and upgrades).

(as it turns out the py3k branch seems to be currently broken on RHEL5; see http://bugs.python.org/issue10517 , so having CI for this would be very useful).

Yep, absolutely. Email me your public key and I'll give you ssh access. I can also beef up the guest a little bit if necessary.

Replying to [comment:12 lmacken]:

Replying to [comment:11 dmalcolm]:

Replying to [comment:10 lmacken]:

Feel free to utilize my Hudson instance for this if you want: http://ci.csh.rit.edu
It's a Xen guest on a Red Hat owned server at RIT, running RHEL5 with Python 2.4, 2.6 and 2.7.

Luke: is this offer still open? I'd be setting up a buildbot worker on it, so it would be a different user/process to the Hudson instance. I'd need to coordinate with the upstream python buildbot folks (e.g. I need to know about outages and upgrades).

(as it turns out the py3k branch seems to be currently broken on RHEL5; see http://bugs.python.org/issue10517 , so having CI for this would be very useful).

Yep, absolutely. Email me your public key and I'll give you ssh access. I can also beef up the guest a little bit if necessary.

Emailed.

I'd like to do both debug and optimized builds of the 3 live branches of the code, giving about 6 builds at once (but we can start small and build out)

Some of the test cases will use large amounts of RAM (1GB, 2GB, 4GB) if it's available (boundary testing for 32-bit truncation), but will be skipped if it isn't.

Whats the status here?

lmacken: is that offer still open?

We are looking at deploying some buildbot clients in our private cloud. Note that we haven't moved it to production yet, so no promises on uptime or the like, but it could still be useful.

See us next week and we might have a setup ready to test...

We do have a limited support/non production jenkins now. No buildbots however.

Please re-open this if this is something we should still persue...

Login to comment on this ticket.

Metadata