Getting dangerous with packaged staging

As part of my work on a custom OE-based SDK for our platforms and the requirement to cut the initial build time, I was looking into packaged staging feature of OpenEmbedded. The basic concept is to package first-time built binaries, libraries etc. from the staging area of temp directory into ipkg (or deb) form and reuse them to populate the staging area back when temp directory was blown away, significantly reducing the "build from scratch" time later on.

This feature needs to be enabled by

INHERIT += "packaged-staging"

Which is default in Angstrom. And deploy/pstage directory needs to be preserved from being erased along with temp, which is not done automatically, relying on a user to do the content management on his own. So, by default packaged-staging.bbclass is configured to install ipkgs in the deploy/pstage directory, which is inside the temp directory:

DEPLOY_DIR_PSTAGE ?= "${DEPLOY_DIR}/pstage"

As the variable is conditionally assigned, it can be overwritten in the local.conf file to have the pstage outside the deploy and temp directories, automatically preserving packaged staging:

DEPLOY_DIR_PSTAGE = "somewhere/else/pstage"

The logical next step is to have the complete deploy directory outside of the temp as well. Unfortunately, Angstrom has it hardcoded in the conf/distro/include/angstrom.inc file like this:

DEPLOY_DIR = "${TMPDIR}/deploy/${ANGSTROM_MODE}"

So, in order to "free up" deploy from the tyranny of temp directory, the above file needs to be overlayed.

This takes care of fast consecutive "from scratch" builds by reusing previously built files from pstage. It especially makes sense to save time on rebuilding native, cross and sdk packages, as they usually don't change for regular users, who develops applications for a target platform. But what can be done to speed up the initial build time? Can those native, cross and sdk pstage packages be shared among users?

Apparently, it's not that easy. To provide controlled environment, many of those native and cross apps are built with hardcoded absolute path to their location in the staging area, so they won't interfere with similar system apps. Thus, implying the per-user, per-project naming rule and making sharing pstage contents between users almost impossible. This rule is specifically handled by the packaged-staging.bbclass in the following code:

# These classes encode staging paths into the binary data so can only be
# reused if the path doesn't change/
if bb.data.inherits_class('native', d) or bb.data.inherits_class('cross', d) or bb.data.inherits_class('sdk', d):
    path = bb.data.getVar('PSTAGE_PKGPATH', d, 1)
    path = path + bb.data.getVar('TMPDIR', d, 1).replace('/', '-')
    bb.data.setVar('PSTAGE_PKGPATH', path, d)

One way to solve it, is to set the staging area path to be the same for all users, say in the global /tmp directory. But that may lead to problems if more than one user tries to do a build (multi-user environment) or the same user does more than one build at the same time (parallel build from different project directories)... So, this proves to be doable, but needs to be handled carefully.

Stay tuned...