= phenomenon =
bapp02 is a 32-bit server, as are all our servers. The s3-mirror module uses the s3cmd application to mirror content to each of the Amazon AWS S3 regions. This module runs out of memory (throws a MemoryError, really, an address space problem) when the tree (/pub/fedora) is over about 900k files. The current tree is over 1M files. The only known workaround is to use a 64-bit instance of python.

This is preventing a full sync into the S3 mirrors. They are now stale for any new content (F19 and updates for F17+).

= reason =
s3cmd needs memory to store information about each file in the file tree, both the local tree and the remote tree. It runs out of 32-bit address space when the trees have ~900k files.

= recommendation =
create a 64-bit bapp server, and run the s3-mirror module scripts there. It needs at least 12GB RAM (8GB, given python's doubling of address space usage for any 64-bit process) would leave us exactly where we are with 4GB 32-bit servers.