diff --git a/doc/cross-compilation.xml b/doc/cross-compilation.xml new file mode 100644 index 00000000000..e93d1a98f7f --- /dev/null +++ b/doc/cross-compilation.xml @@ -0,0 +1,153 @@ + + +Cross-compilation + +
+ Introduction + + "Cross-compilation" means compiling a program on one machine for another type of machine. + For example, a typical use of cross compilation is to compile programs for embedded devices. + These devices often don't have the computing power and memory to compile their own programs. + One might think that cross-compilation is a fairly niche concern, but there are advantages to being rigorous about distinguishing build-time vs run-time environments even when one is developing and deploying on the same machine. + Nixpkgs is increasingly adopting this opinion in that packages should be written with cross-compilation in mind, and nixpkgs should evaluate in a similar way (by minimizing cross-compilation-specific special cases) whether or not one is cross-compiling. + + + + This chapter will be organized in three parts. + First, it will describe the basics of how to package software in a way that supports cross-compilation. + Second, it will describe how to use Nixpkgs when cross-compiling. + Third, it will describe the internal infrastructure supporting cross-compilation. + +
+ + + +
+ Packing in a cross-friendly manner + +
+ Platform parameters + + The three GNU Autoconf platforms, build, host, and cross, are historically the result of much confusion. + clears this up somewhat but there is more to be said. + An important advice to get out the way is, unless you are packaging a compiler or other build tool, just worry about the build and host platforms. + Dealing with just two platforms usually better matches people's preconceptions, and in this case is completely correct. + + + In Nixpkgs, these three platforms are defined as attribute sets under the names buildPlatform, hostPlatform, and targetPlatform. + All are guaranteed to contain at least a platform field, which contains detailed information on the platform. + All three are always defined at the top level, so one can get at them just like a dependency in a function that is imported with callPackage: + { stdenv, buildPlatform, hostPlatform, fooDep, barDep, .. }: ... + + + These platforms should all have the same structure in all scenarios, but that is currently not the case. + When not cross-compiling, they will each contain a system field with a short 2-part, hyphen-separated summering string name for the platform. + But, when when cross compiling, hostPlatform and targetPlatform may instead contain config with a fuller 3- or 4-part string in the manner of LLVM. + We should have all 3 platforms always contain both, and maybe give config a better name while we are at it. + + + + buildPlatform + + The "build platform" is the platform on which a package is built. + Once someone has a built package, or pre-built binary package, the build platform should not matter and be safe to ignore. + + + + hostPlatform + + The "host platform" is the platform on which a package is run. + This is the simplest platform to understand, but also the one with the worst name. + + + + targetPlatform + + + The "target platform" is black sheep. + The other two intrinsically apply to all compiled software—or any build process with a notion of "build-time" followed by "run-time". + The target platform only applies to programming tools, and even then only is a good for for some of them. + Briefly, GCC, Binutils, GHC, and certain other tools are written in such a way such that a single build can only compiler code for a single platform. + Thus, when building them, one must think ahead about what platforms they wish to use the tool to produce machine code for, and build binaries for each. + + + There is no fundamental need to think about the target ahead of time like this. + LLVM, for example, was designed from the beginning with cross-compilation in mind, and so a normal LLVM binary will support every architecture that LLVM supports. + If the tool supports modular or pluggable backends, one might imagine specifying a set of target platforms / backends one wishes to support, rather than a single one. + + + The biggest reason for mess, if there is one, is that many compilers have the bad habit a build process that builds the compiler and standard library/runtime together. + Then the specifying target platform is essential, because it determines the host platform of the standard library/runtime. + Nixpkgs tries to avoid this where possible too, but still, because the concept of a target platform is so ingrained now in Autoconf and other tools, it is best to support it as is. + Tools like LLVM that don't need up-front target platforms can safely ignore it like normal packages, and it will do no harm. + + + + + + If you dig around nixpkgs, you may notice there is also stdenv.cross. + This field defined as hostPlatform when the host and build platforms differ, but otherwise not defined at all. + This field is obsolete and will soon disappear—please do not use it. + +
+ +
+ Specifying Dependencies + + As mentioned in the introduction to this chapter, one can think about a build time vs run time distinction whether cross-compiling or not. + In the case of cross-compilation, this corresponds with whether a derivation running on the native or foreign platform is produced. + An interesting thing to think about is how this corresponds with the three Autoconf platforms. + In the run-time case, the depending and depended-on package simply have matching build, host, and target platforms. + But in the build-time case, one can imagine "sliding" the platforms one over. + The depended-on package's host and target platforms (respectively) become the depending package's build and host platforms. + This is the most important guiding principle behind cross-compilation with Nixpkgs, and will be called the sliding window principle. + In this manner, given the 3 platforms for one package, we can determine the three platforms for all its transitive dependencies. + + + The depending package's target platform is unconstrained by the sliding window principle, which makes sense in that one can in principle build cross compilers targeting arbitrary platforms. + + + From the above, one would surmise that if a package is being built with a (build, host, target) platform triple of (foo, bar, bar), then its build-time dependencies would have a triple of (foo, foo, bar), and those packages' build-time dependencies would have triple of (foo, foo, foo). + In other words, it should take two "rounds" of following build-time dependency edges before one reaches a fixed point where, by the sliding window principle, the platform triple no longer changes. + Unfortunately, at the moment, we do not implement this correctly, and after only one round of following build-time dependencies is the fixed point reached, with target incorrectly kept different than the others. + + + How does this work in practice? Nixpkgs is now structured so that build-time dependencies are taken from from buildPackages, whereas run-time dependencies are taken from the top level attribute set. + For example, buildPackages.gcc should be used at build time, while gcc should be used at run time. + Now, for most of Nixpkgs's history, there was no buildPackages, and most packages have not been refactored to use it explicitly. + Instead, one can use the four attributes used for specifying dependencies as documented in . + We "splice" together the run-time and build-time package sets with callPackage, and then mkDerivation for each of four attributes pulls the right derivation out. + This splicing can be skipped when not cross compiling as the package sets are the same, but is a bit slow for cross compiling. + Because of this, a best-of-both-worlds solution is in the works with no splicing or explicit access of buildPackages needed. + For now, feel free to use either method. + +
+ +
+ + + +
+ Cross-building packages + + To be written. + This is basically unchanged so see the old wiki for now. + +
+ + + +
+ Cross-compilation infrastructure + To be written. + + If one explores nixpkgs, they will see derivations with names like gccCross. + Such *Cross derivations is a holdover from before we properly distinguished between the host and target platforms + —the derivation with "Cross" in the name covered the build = host != target case, while the other covered the host = target, with build platform the same or not based on whether one was using its .nativeDrv or .crossDrv. + This ugliness will disappear soon. + +
+ + diff --git a/doc/manual.xml b/doc/manual.xml index 1c0dac6e4df..75bd21557fd 100644 --- a/doc/manual.xml +++ b/doc/manual.xml @@ -13,6 +13,7 @@ + diff --git a/doc/stdenv.xml b/doc/stdenv.xml index 44a0e4601fc..6ec5c9f2814 100644 --- a/doc/stdenv.xml +++ b/doc/stdenv.xml @@ -194,33 +194,52 @@ genericBuild tools. + + + + Variables specifying dependencies + + + nativeBuildInputs + + A list of dependencies used by the new derivation at build-time. + I.e. these dependencies should not make it into the package's runtime-closure, though this is currently not checked. + For each dependency dir, the directory dir/bin, if it exists, is added to the PATH environment variable. + Other environment variables are also set up via a pluggable mechanism. + For instance, if buildInputs contains Perl, then the lib/site_perl subdirectory of each input is added to the PERL5LIB environment variable. + See for details. + + + buildInputs - A list of dependencies used by - stdenv to set up the environment for the build. - For each dependency dir, the directory - dir/bin, if it - exists, is added to the PATH environment variable. - Other environment variables are also set up via a pluggable - mechanism. For instance, if buildInputs - contains Perl, then the lib/site_perl - subdirectory of each input is added to the PERL5LIB - environment variable. See for - details. + + A list of dependencies used by the new derivation at run-time. + Currently, the build-time environment is modified in the exact same way as with nativeBuildInputs. + This is problematic in that when cross-compiling, foreign executables can clobber native ones on the PATH. + Even more confusing is static-linking. + A statically-linked library should be listed here because ultimately that generated machine code will be used at run-time, even though a derivation containing the object files or static archives will only be used at build-time. + A less confusing solution to this would be nice. + - + + + + propagatedNativeBuildInputs + + Like nativeBuildInputs, but these dependencies are propagated: + that is, the dependencies listed here are added to the nativeBuildInputs of any package that uses this package as a dependency. + So if package Y has propagatedBuildInputs = [X], and package Z has buildInputs = [Y], then package X will appear in Z’s build environment automatically. + + + propagatedBuildInputs - Like buildInputs, but these - dependencies are propagated: that is, the - dependencies listed here are added to the - buildInputs of any package that uses - this package as a dependency. So if package - Y has propagatedBuildInputs = [X], and package - Z has buildInputs = [Y], then package X will - appear in Z’s build environment automatically. + + Like buildInputs, but propagated just like propagatedNativeBuildInputs. + This inherits buildInputs's flaws of clobbering native executables when cross-compiling and being confusing for static linking. + - @@ -322,7 +341,7 @@ executed and in what order: $preInstallPhases installPhase fixupPhase $preDistPhases distPhase $postPhases. - + Usually, if you just want to add a few phases, it’s more convenient to set one of the variables below (such as preInstallPhases), as you then don’t specify @@ -706,7 +725,7 @@ makeFlagsArray=(CFLAGS="-O0 -g" LDFLAGS="-lfoo -lbar") - + You can set flags for make through the makeFlags variable. @@ -773,7 +792,7 @@ doCheck = true; - + @@ -840,12 +859,12 @@ install phase. The default fixupPhase does the following: - + It moves the man/, doc/ and info/ subdirectories of $out to share/. - + It strips libraries and executables of debug information. @@ -1091,13 +1110,13 @@ functions. - + substitute infile outfile subs - + Performs string substitution on the contents of infile, writing the result to @@ -1125,7 +1144,7 @@ functions. @...@ in the template as placeholders. - + varName @@ -1134,7 +1153,7 @@ functions. @varName@ by the string s. - + @@ -1162,7 +1181,7 @@ substitute ./foo.in ./foo.out \ - + substituteInPlace @@ -1173,7 +1192,7 @@ substitute ./foo.in ./foo.out \ file. - + substituteAll infile @@ -1233,7 +1252,7 @@ echo @foo@ Strips the directory and hash part of a store path, outputting the name part to stdout. For example: - + # prints coreutils-8.24 stripHash "/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24" @@ -1241,7 +1260,7 @@ stripHash "/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24" If you wish to store the result in another variable, then the following idiom may be useful: - + name="/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24" someVar=$(stripHash $name) @@ -1250,7 +1269,7 @@ someVar=$(stripHash $name) - + @@ -1607,4 +1626,3 @@ Arch Wiki. - diff --git a/nixos/doc/manual/release-notes/rl-1703.xml b/nixos/doc/manual/release-notes/rl-1703.xml index 6be2cd3af7f..aa864b7a757 100644 --- a/nixos/doc/manual/release-notes/rl-1703.xml +++ b/nixos/doc/manual/release-notes/rl-1703.xml @@ -30,6 +30,15 @@ has the following highlights: following incompatible changes: + + + Cross compilation has been rewritten. See the nixpkgs manual for + details. The most obvious breaking change is that derivations absent a + .nativeDrv or .crossDrv are now + cross by default, not native. + + + stdenv.overrides is now expected to take self diff --git a/pkgs/build-support/cc-wrapper/default.nix b/pkgs/build-support/cc-wrapper/default.nix index c8e3d8b4cc8..95e0b360937 100644 --- a/pkgs/build-support/cc-wrapper/default.nix +++ b/pkgs/build-support/cc-wrapper/default.nix @@ -281,9 +281,6 @@ stdenv.mkDerivation { crossAttrs = { shell = shell.crossDrv + shell.crossDrv.shellPath; libc = stdenv.ccCross.libc; - coreutils = coreutils.crossDrv; - binutils = binutils.crossDrv; - cc = cc.crossDrv; # # This is not the best way to do this. I think the reference should be # the style in the gcc-cross-wrapper, but to keep a stable stdenv now I diff --git a/pkgs/development/compilers/ghc/head.nix b/pkgs/development/compilers/ghc/head.nix index 971365eda48..e7f4335d6f6 100644 --- a/pkgs/development/compilers/ghc/head.nix +++ b/pkgs/development/compilers/ghc/head.nix @@ -1,5 +1,6 @@ { stdenv, fetchgit, bootPkgs, perl, gmp, ncurses, libiconv, binutils, coreutils -, autoconf, automake, happy, alex, python3, crossSystem, selfPkgs, cross ? null +, autoconf, automake, happy, alex, python3, buildPlatform, targetPlatform +, selfPkgs, cross ? null }: let @@ -68,9 +69,9 @@ in stdenv.mkDerivation (rec { passthru = { inherit bootPkgs; - } // stdenv.lib.optionalAttrs (crossSystem != null) { + } // stdenv.lib.optionalAttrs (targetPlatform != buildPlatform) { crossCompiler = selfPkgs.ghc.override { - cross = crossSystem; + cross = targetPlatform; bootPkgs = selfPkgs; }; }; diff --git a/pkgs/development/libraries/openssl/default.nix b/pkgs/development/libraries/openssl/default.nix index cd4a696b1d9..7cce57a753f 100644 --- a/pkgs/development/libraries/openssl/default.nix +++ b/pkgs/development/libraries/openssl/default.nix @@ -1,4 +1,4 @@ -{ stdenv, fetchurl, perl +{ stdenv, fetchurl, buildPackages, perl , withCryptodev ? false, cryptodevHeaders , enableSSL2 ? false }: @@ -76,7 +76,7 @@ let postFixup = '' # Check to make sure the main output doesn't depend on perl - if grep -r '${perl}' $out; then + if grep -r '${buildPackages.perl}' $out; then echo "Found an erroneous dependency on perl ^^^" >&2 exit 1 fi diff --git a/pkgs/os-specific/gnu/default.nix b/pkgs/os-specific/gnu/default.nix index 457b670319e..247c73e468d 100644 --- a/pkgs/os-specific/gnu/default.nix +++ b/pkgs/os-specific/gnu/default.nix @@ -3,7 +3,8 @@ args@{ fetchgit, stdenv, autoconf, automake, automake111x, libtool , texinfo, glibcCross, hurdPartedCross, libuuid, samba , gccCrossStageStatic, gccCrossStageFinal -, forceNativeDrv, forceSystem, newScope, platform, config, crossSystem +, forcedNativePackages, forceSystem, newScope, platform, config +, targetPlatform, buildPlatform , overrides ? {} }: with args; @@ -12,25 +13,25 @@ let callPackage = newScope gnu; gnu = { - hurdCross = forceNativeDrv (callPackage ./hurd { + hurdCross = forcedNativePackages.callPackage ./hurd { inherit fetchgit stdenv autoconf libtool texinfo glibcCross hurdPartedCross; inherit (gnu) machHeaders mig; libuuid = libuuid.crossDrv; automake = automake111x; headersOnly = false; - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; gccCross = gccCrossStageFinal; - }); + }; - hurdCrossIntermediate = forceNativeDrv (callPackage ./hurd { + hurdCrossIntermediate = forcedNativePackages.callPackage ./hurd { inherit fetchgit stdenv autoconf libtool texinfo glibcCross; inherit (gnu) machHeaders mig; hurdPartedCross = null; libuuid = null; automake = automake111x; headersOnly = false; - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; # The "final" GCC needs glibc and the Hurd libraries (libpthread in # particular) so we first need an intermediate Hurd built with the @@ -42,7 +43,7 @@ let # libshouldbeinlibc. buildTarget = "libihash libstore libshouldbeinlibc"; installTarget = "libihash-install libstore-install libshouldbeinlibc-install"; - }); + }; hurdHeaders = callPackage ./hurd { automake = automake111x; @@ -58,13 +59,13 @@ let hurd = null; }; - libpthreadCross = forceNativeDrv (callPackage ./libpthread { + libpthreadCross = forcedNativePackages.callPackage ./libpthread { inherit fetchgit stdenv autoconf automake libtool glibcCross; inherit (gnu) machHeaders hurdHeaders; hurd = gnu.hurdCrossIntermediate; gccCross = gccCrossStageStatic; - cross = assert crossSystem != null; crossSystem; - }); + cross = assert targetPlatform != buildPlatform; targetPlatform; + }; # In theory GNU Mach doesn't have to be cross-compiled. However, since it # has to be built for i586 (it doesn't work on x86_64), one needs a cross diff --git a/pkgs/stdenv/adapters.nix b/pkgs/stdenv/adapters.nix index 11f9a43c035..7e0eaeddd2c 100644 --- a/pkgs/stdenv/adapters.nix +++ b/pkgs/stdenv/adapters.nix @@ -56,69 +56,59 @@ rec { # Return a modified stdenv that adds a cross compiler to the # builds. - makeStdenvCross = stdenv: cross: binutilsCross: gccCross: stdenv // - { mkDerivation = {name ? "", buildInputs ? [], nativeBuildInputs ? [], - propagatedBuildInputs ? [], propagatedNativeBuildInputs ? [], - selfNativeBuildInput ? false, ...}@args: let + makeStdenvCross = stdenv: cross: binutilsCross: gccCross: stdenv // { - # *BuildInputs exists temporarily as another name for - # *HostInputs. + mkDerivation = + { name ? "", buildInputs ? [], nativeBuildInputs ? [] + , propagatedBuildInputs ? [], propagatedNativeBuildInputs ? [] + , selfNativeBuildInput ? false, ... + } @ args: - # In nixpkgs, sometimes 'null' gets in as a buildInputs element, - # and we handle that through isAttrs. - getNativeDrv = drv: drv.nativeDrv or drv; - getCrossDrv = drv: drv.crossDrv or drv; - nativeBuildInputsDrvs = map getNativeDrv nativeBuildInputs; - buildInputsDrvs = map getCrossDrv buildInputs; - propagatedBuildInputsDrvs = map getCrossDrv propagatedBuildInputs; - propagatedNativeBuildInputsDrvs = map getNativeDrv propagatedNativeBuildInputs; + let + # *BuildInputs exists temporarily as another name for + # *HostInputs. - # The base stdenv already knows that nativeBuildInputs and - # buildInputs should be built with the usual gcc-wrapper - # And the same for propagatedBuildInputs. - nativeDrv = stdenv.mkDerivation args; + # The base stdenv already knows that nativeBuildInputs and + # buildInputs should be built with the usual gcc-wrapper + # And the same for propagatedBuildInputs. + nativeDrv = stdenv.mkDerivation args; - # Temporary expression until the cross_renaming, to handle the - # case of pkgconfig given as buildInput, but to be used as - # nativeBuildInput. - hostAsNativeDrv = drv: - builtins.unsafeDiscardStringContext drv.nativeDrv.drvPath - == builtins.unsafeDiscardStringContext drv.crossDrv.drvPath; - buildInputsNotNull = stdenv.lib.filter - (drv: builtins.isAttrs drv && drv ? nativeDrv) buildInputs; - nativeInputsFromBuildInputs = stdenv.lib.filter hostAsNativeDrv buildInputsNotNull; + # Temporary expression until the cross_renaming, to handle the + # case of pkgconfig given as buildInput, but to be used as + # nativeBuildInput. + hostAsNativeDrv = drv: + builtins.unsafeDiscardStringContext drv.nativeDrv.drvPath + == builtins.unsafeDiscardStringContext drv.crossDrv.drvPath; + buildInputsNotNull = stdenv.lib.filter + (drv: builtins.isAttrs drv && drv ? nativeDrv) buildInputs; + nativeInputsFromBuildInputs = stdenv.lib.filter hostAsNativeDrv buildInputsNotNull; + in + stdenv.mkDerivation (args // { + name = name + "-" + cross.config; + nativeBuildInputs = nativeBuildInputs + ++ nativeInputsFromBuildInputs + ++ [ gccCross binutilsCross ] + ++ stdenv.lib.optional selfNativeBuildInput nativeDrv + # without proper `file` command, libtool sometimes fails + # to recognize 64-bit DLLs + ++ stdenv.lib.optional (cross.config == "x86_64-w64-mingw32") pkgs.file + ; - # We should overwrite the input attributes in crossDrv, to overwrite - # the defaults for only-native builds in the base stdenv - crossDrv = if cross == null then nativeDrv else - stdenv.mkDerivation (args // { - name = name + "-" + cross.config; - nativeBuildInputs = nativeBuildInputsDrvs - ++ nativeInputsFromBuildInputs - ++ [ gccCross binutilsCross ] - ++ stdenv.lib.optional selfNativeBuildInput nativeDrv - # without proper `file` command, libtool sometimes fails - # to recognize 64-bit DLLs - ++ stdenv.lib.optional (cross.config == "x86_64-w64-mingw32") pkgs.file - ; + # Cross-linking dynamic libraries, every buildInput should + # be propagated because ld needs the -rpath-link to find + # any library needed to link the program dynamically at + # loader time. ld(1) explains it. + buildInputs = []; + propagatedBuildInputs = propagatedBuildInputs ++ buildInputs; + propagatedNativeBuildInputs = propagatedNativeBuildInputs; - # Cross-linking dynamic libraries, every buildInput should - # be propagated because ld needs the -rpath-link to find - # any library needed to link the program dynamically at - # loader time. ld(1) explains it. - buildInputs = []; - propagatedBuildInputs = propagatedBuildInputsDrvs ++ buildInputsDrvs; - propagatedNativeBuildInputs = propagatedNativeBuildInputsDrvs; + crossConfig = cross.config; + } // args.crossAttrs or {}); - crossConfig = cross.config; - } // args.crossAttrs or {}); - in nativeDrv // { - inherit crossDrv nativeDrv; - }; - } // { - inherit cross gccCross binutilsCross; - ccCross = gccCross; - }; + inherit gccCross binutilsCross; + ccCross = gccCross; + + }; /* Modify a stdenv so that the specified attributes are added to diff --git a/pkgs/stdenv/booter.nix b/pkgs/stdenv/booter.nix index 11ca8e1440e..2c82d12da95 100644 --- a/pkgs/stdenv/booter.nix +++ b/pkgs/stdenv/booter.nix @@ -57,12 +57,17 @@ stageFuns: let # debugging purposes. folder = stageFun: finalSoFar: let args = stageFun finalSoFar; - stdenv = args.stdenv // { - # For debugging - __bootPackages = finalSoFar; + args' = args // { + stdenv = args.stdenv // { + # For debugging + __bootPackages = finalSoFar; + }; }; - args' = args // { inherit stdenv; }; in - (if args.__raw or false then lib.id else allPackages) args'; + if args.__raw or false + then args' + else allPackages ((builtins.removeAttrs args' ["selfBuild"]) // { + buildPackages = if args.selfBuild or true then null else finalSoFar; + }); in lib.lists.fold folder {} withAllowCustomOverrides diff --git a/pkgs/stdenv/cross/default.nix b/pkgs/stdenv/cross/default.nix index 16f41671b76..37f403acee9 100644 --- a/pkgs/stdenv/cross/default.nix +++ b/pkgs/stdenv/cross/default.nix @@ -1,10 +1,10 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: let bootStages = import ../. { - inherit lib system platform overlays; + inherit lib localSystem overlays; crossSystem = null; # Ignore custom stdenvs when cross compiling for compatability config = builtins.removeAttrs config [ "replaceStdenv" ]; @@ -12,25 +12,28 @@ let in bootStages ++ [ - # Build Packages. - # - # For now, this is just used to build the native stdenv. Eventually, it - # should be used to build compilers and other such tools targeting the cross - # platform. Then, `forceNativeDrv` can be removed. + # Build Packages (vanillaPackages: { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = crossSystem; + inherit config overlays; + # Should be false, but we're trying to preserve hashes for now + selfBuild = true; # It's OK to change the built-time dependencies allowCustomOverrides = true; stdenv = vanillaPackages.stdenv // { - # Needed elsewhere as a hacky way to pass the target - cross = crossSystem; overrides = _: _: {}; }; }) - # Run packages + # Run Packages (buildPackages: { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = crossSystem; + targetPlatform = crossSystem; + inherit config overlays; + selfBuild = false; stdenv = if crossSystem.useiOSCross or false then let inherit (buildPackages.darwin.ios-cross { diff --git a/pkgs/stdenv/custom/default.nix b/pkgs/stdenv/custom/default.nix index d7e9bf53bed..d5dc977b37a 100644 --- a/pkgs/stdenv/custom/default.nix +++ b/pkgs/stdenv/custom/default.nix @@ -1,12 +1,12 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: assert crossSystem == null; let bootStages = import ../. { - inherit lib system platform crossSystem overlays; + inherit lib localSystem crossSystem overlays; # Remove config.replaceStdenv to ensure termination. config = builtins.removeAttrs config [ "replaceStdenv" ]; }; @@ -15,7 +15,10 @@ in bootStages ++ [ # Additional stage, built using custom stdenv (vanillaPackages: { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = config.replaceStdenv { pkgs = vanillaPackages; }; }) diff --git a/pkgs/stdenv/darwin/default.nix b/pkgs/stdenv/darwin/default.nix index e3a87ea078f..e647c81890e 100644 --- a/pkgs/stdenv/darwin/default.nix +++ b/pkgs/stdenv/darwin/default.nix @@ -1,11 +1,12 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays # Allow passing in bootstrap files directly so we can test the stdenv bootstrap process when changing the bootstrap tools , bootstrapFiles ? let fetch = { file, sha256, executable ? true }: import { url = "http://tarballs.nixos.org/stdenv-darwin/x86_64/33f59c9d11b8d5014dfd18cc11a425f6393c884a/${file}"; - inherit sha256 system executable; + inherit (localSystem) system; + inherit sha256 executable; }; in { sh = fetch { file = "sh"; sha256 = "1rx4kg6358xdj05z0m139a0zn4f4zfmq4n4vimlmnwyfiyn4x7wk"; }; bzip2 = fetch { file = "bzip2"; sha256 = "104qnhzk79vkbp2yi0kci6lszgfppvrwk3rgxhry842ly1xz2r7l"; }; @@ -18,6 +19,8 @@ assert crossSystem == null; let + inherit (localSystem) system platform; + libSystemProfile = '' (import "${./standard-sandbox.sb}") ''; @@ -98,7 +101,10 @@ in rec { }; in { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = thisStdenv; }; @@ -316,7 +322,10 @@ in rec { stage3 stage4 (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = stdenvDarwin prevStage; }) ]; diff --git a/pkgs/stdenv/default.nix b/pkgs/stdenv/default.nix index f60ffec4b56..78dbde13b89 100644 --- a/pkgs/stdenv/default.nix +++ b/pkgs/stdenv/default.nix @@ -7,7 +7,7 @@ { # Args just for stdenvs' usage lib # Args to pass on to the pkgset builder, too -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays } @ args: let @@ -51,4 +51,4 @@ in "i686-cygwin" = stagesNative; "x86_64-cygwin" = stagesNative; "x86_64-freebsd" = stagesFreeBSD; - }.${system} or stagesNative + }.${localSystem.system} or stagesNative diff --git a/pkgs/stdenv/freebsd/default.nix b/pkgs/stdenv/freebsd/default.nix index 2cb059deb34..b926c6bdd90 100644 --- a/pkgs/stdenv/freebsd/default.nix +++ b/pkgs/stdenv/freebsd/default.nix @@ -1,8 +1,9 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: assert crossSystem == null; +let inherit (localSystem) system; in [ @@ -58,7 +59,10 @@ assert crossSystem == null; }) (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = import ../generic { name = "stdenv-freebsd-boot-3"; inherit system config; diff --git a/pkgs/stdenv/generic/default.nix b/pkgs/stdenv/generic/default.nix index 32e0d894818..269d7ef893a 100644 --- a/pkgs/stdenv/generic/default.nix +++ b/pkgs/stdenv/generic/default.nix @@ -115,7 +115,19 @@ let , sandboxProfile ? "" , propagatedSandboxProfile ? "" , ... } @ attrs: - let + let # Rename argumemnts to avoid cycles + buildInputs__ = buildInputs; + nativeBuildInputs__ = nativeBuildInputs; + propagatedBuildInputs__ = propagatedBuildInputs; + propagatedNativeBuildInputs__ = propagatedNativeBuildInputs; + in let + getNativeDrv = drv: drv.nativeDrv or drv; + getCrossDrv = drv: drv.crossDrv or drv; + nativeBuildInputs = map getNativeDrv nativeBuildInputs__; + buildInputs = map getCrossDrv buildInputs__; + propagatedBuildInputs = map getCrossDrv propagatedBuildInputs__; + propagatedNativeBuildInputs = map getNativeDrv propagatedNativeBuildInputs__; + in let pos' = if pos != null then pos diff --git a/pkgs/stdenv/linux/default.nix b/pkgs/stdenv/linux/default.nix index 12da007f2a7..611628b35ab 100644 --- a/pkgs/stdenv/linux/default.nix +++ b/pkgs/stdenv/linux/default.nix @@ -4,21 +4,23 @@ # compiler and linker that do not search in default locations, # ensuring purity of components produced by it. { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays -, bootstrapFiles ? - if system == "i686-linux" then import ./bootstrap-files/i686.nix - else if system == "x86_64-linux" then import ./bootstrap-files/x86_64.nix - else if system == "armv5tel-linux" then import ./bootstrap-files/armv5tel.nix - else if system == "armv6l-linux" then import ./bootstrap-files/armv6l.nix - else if system == "armv7l-linux" then import ./bootstrap-files/armv7l.nix - else if system == "mips64el-linux" then import ./bootstrap-files/loongson2f.nix - else abort "unsupported platform for the pure Linux stdenv" +, bootstrapFiles ? { # switch + "i686-linux" = import ./bootstrap-files/i686.nix; + "x86_64-linux" = import ./bootstrap-files/x86_64.nix; + "armv5tel-linux" = import ./bootstrap-files/armv5tel.nix; + "armv6l-linux" = import ./bootstrap-files/armv6l.nix; + "armv7l-linux" = import ./bootstrap-files/armv7l.nix; + "mips64el-linux" = import ./bootstrap-files/loongson2f.nix; + }.${localSystem.system} + or (abort "unsupported platform for the pure Linux stdenv") }: assert crossSystem == null; let + inherit (localSystem) system platform; commonPreHook = '' @@ -91,7 +93,10 @@ let }; in { - inherit system platform crossSystem config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = thisStdenv; }; @@ -246,7 +251,10 @@ in # dependency (`nix-store -qR') on bootstrapTools or the first # binutils built. (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = import ../generic rec { inherit system config; diff --git a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix index 9f4a4517627..38b3e611bc2 100644 --- a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix +++ b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix @@ -55,11 +55,12 @@ let if toolsArch == "armv6l" then raspberrypiCrossSystem else if toolsArch == "armv7l" then armv7l-hf-multiplatform-crossSystem else null; - pkgs = pkgsFun ({inherit system;} // selectedCrossSystem); + pkgsUnspliced = pkgsFun ({inherit system;} // selectedCrossSystem); + pkgs = pkgsUnspliced.splicedPackages; - inherit (pkgs) stdenv nukeReferences cpio binutilsCross; + inherit (pkgsUnspliced.buildPackages) stdenv nukeReferences cpio binutilsCross; - glibc = pkgs.libcCross; + glibc = pkgs.libcCross.nativeDrv; bash = pkgs.bash.crossDrv; findutils = pkgs.findutils.crossDrv; diffutils = pkgs.diffutils.crossDrv; @@ -71,7 +72,7 @@ let gnumake = pkgs.gnumake.crossDrv; patch = pkgs.patch.crossDrv; patchelf = pkgs.patchelf.crossDrv; - gcc = pkgs.gcc.cc.crossDrv; + gcc = pkgs.gcc.crossDrv.cc; gmpxx = pkgs.gmpxx.crossDrv; mpfr = pkgs.mpfr.crossDrv; zlib = pkgs.zlib.crossDrv; @@ -86,17 +87,17 @@ in rec { - coreutilsMinimal = (pkgs.coreutils.override (args: { + coreutilsMinimal = pkgs.coreutils.override (args: { # We want coreutils without ACL/attr support. aclSupport = false; attrSupport = false; # Our tooling currently can't handle scripts in bin/, only ELFs and symlinks. singleBinary = "symlinks"; - })).crossDrv; + }); - tarMinimal = (pkgs.gnutar.override { acl = null; }).crossDrv; + tarMinimal = pkgs.gnutar.override { acl = null; }; - busyboxMinimal = (pkgs.busybox.override { + busyboxMinimal = pkgs.busybox.override { useMusl = true; enableStatic = true; enableMinimal = true; @@ -109,13 +110,13 @@ rec { CONFIG_TAR y CONFIG_UNXZ y ''; - }).crossDrv; + }; build = stdenv.mkDerivation { name = "stdenv-bootstrap-tools-cross"; - crossConfig = stdenv.cross.config; + crossConfig = pkgsUnspliced.hostPlatform.config; buildInputs = [nukeReferences cpio binutilsCross]; diff --git a/pkgs/stdenv/native/default.nix b/pkgs/stdenv/native/default.nix index 4028638009e..f5c0976bf93 100644 --- a/pkgs/stdenv/native/default.nix +++ b/pkgs/stdenv/native/default.nix @@ -1,10 +1,11 @@ { lib -, system, platform, crossSystem, config, overlays +, localSystem, crossSystem, config, overlays }: assert crossSystem == null; let + inherit (localSystem) system platform; shell = if system == "i686-freebsd" || system == "x86_64-freebsd" then "/usr/local/bin/bash" @@ -134,7 +135,10 @@ in # First build a stdenv based only on tools outside the store. (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = makeStdenv { inherit (prevStage) cc fetchurl; } // { inherit (prevStage) fetchurl; }; @@ -143,7 +147,10 @@ in # Using that, build a stdenv that adds the ‘xz’ command (which most systems # don't have, so we mustn't rely on the native environment providing it). (prevStage: { - inherit system crossSystem platform config overlays; + buildPlatform = localSystem; + hostPlatform = localSystem; + targetPlatform = localSystem; + inherit config overlays; stdenv = makeStdenv { inherit (prevStage.stdenv) cc fetchurl; extraPath = [ prevStage.xz ]; diff --git a/pkgs/stdenv/nix/default.nix b/pkgs/stdenv/nix/default.nix index a5f0a18464c..9aece3ce829 100644 --- a/pkgs/stdenv/nix/default.nix +++ b/pkgs/stdenv/nix/default.nix @@ -9,9 +9,9 @@ assert crossSystem == null; bootStages ++ [ (prevStage: let inherit (prevStage) stdenv; - inherit (stdenv) system platform; in { - inherit system platform crossSystem config; + inherit (prevStage) buildPlatform hostPlatform targetPlatform; + inherit config overlays; stdenv = import ../generic rec { inherit config; diff --git a/pkgs/tools/misc/coreutils/default.nix b/pkgs/tools/misc/coreutils/default.nix index 9e66c6ba918..2c435881f8c 100644 --- a/pkgs/tools/misc/coreutils/default.nix +++ b/pkgs/tools/misc/coreutils/default.nix @@ -1,4 +1,4 @@ -{ lib, stdenv, fetchurl, perl, xz, gmp ? null +{ lib, stdenv, buildPackages, fetchurl, perl, xz, gmp ? null , aclSupport ? false, acl ? null , attrSupport ? false, attr ? null , selinuxSupport? false, libselinux ? null, libsepol ? null @@ -12,104 +12,101 @@ assert selinuxSupport -> libselinux != null && libsepol != null; with lib; -let - self = stdenv.mkDerivation rec { - name = "coreutils-8.26"; +stdenv.mkDerivation rec { + name = "coreutils-8.26"; - src = fetchurl { - url = "mirror://gnu/coreutils/${name}.tar.xz"; - sha256 = "13lspazc7xkviy93qz7ks9jv4sldvgmwpq36ghrbrqpq93br8phm"; - }; + src = fetchurl { + url = "mirror://gnu/coreutils/${name}.tar.xz"; + sha256 = "13lspazc7xkviy93qz7ks9jv4sldvgmwpq36ghrbrqpq93br8phm"; + }; - # FIXME needs gcc 4.9 in bootstrap tools - hardeningDisable = [ "stackprotector" ]; + # FIXME needs gcc 4.9 in bootstrap tools + hardeningDisable = [ "stackprotector" ]; - patches = optional stdenv.isCygwin ./coreutils-8.23-4.cygwin.patch; + patches = optional stdenv.isCygwin ./coreutils-8.23-4.cygwin.patch; - # The test tends to fail on btrfs and maybe other unusual filesystems. - postPatch = optionalString (!stdenv.isDarwin) '' - sed '2i echo Skipping dd sparse test && exit 0' -i ./tests/dd/sparse.sh - sed '2i echo Skipping cp sparse test && exit 0' -i ./tests/cp/sparse.sh - sed '2i echo Skipping rm deep-2 test && exit 0' -i ./tests/rm/deep-2.sh - sed '2i echo Skipping du long-from-unreadable test && exit 0' -i ./tests/du/long-from-unreadable.sh + # The test tends to fail on btrfs and maybe other unusual filesystems. + postPatch = optionalString (!stdenv.isDarwin) '' + sed '2i echo Skipping dd sparse test && exit 0' -i ./tests/dd/sparse.sh + sed '2i echo Skipping cp sparse test && exit 0' -i ./tests/cp/sparse.sh + sed '2i echo Skipping rm deep-2 test && exit 0' -i ./tests/rm/deep-2.sh + sed '2i echo Skipping du long-from-unreadable test && exit 0' -i ./tests/du/long-from-unreadable.sh + ''; + + outputs = [ "out" "info" ]; + + nativeBuildInputs = [ perl xz.bin ]; + configureFlags = + optional (singleBinary != false) + ("--enable-single-binary" + optionalString (isString singleBinary) "=${singleBinary}") + ++ optional stdenv.isSunOS "ac_cv_func_inotify_init=no" + ++ optional withPrefix "--program-prefix=g"; + + buildInputs = [ gmp ] + ++ optional aclSupport acl + ++ optional attrSupport attr + ++ optionals stdenv.isCygwin [ autoconf automake114x texinfo ] # due to patch + ++ optionals selinuxSupport [ libselinux libsepol ]; + + crossAttrs = { + buildInputs = [ gmp.crossDrv ] + ++ optional aclSupport acl.crossDrv + ++ optional attrSupport attr.crossDrv + ++ optionals selinuxSupport [ libselinux.crossDrv libsepol.crossDrv ] + ++ optional (stdenv.ccCross.libc ? libiconv) + stdenv.ccCross.libc.libiconv.crossDrv; + + # Prevents attempts of running 'help2man' on cross-built binaries. + PERL = "missing"; + + # Works around a bug with 8.26: + # Makefile:3440: *** Recursive variable 'INSTALL' references itself (eventually). Stop. + preInstall = '' + sed -i Makefile -e 's|^INSTALL =.*|INSTALL = ${buildPackages.coreutils}/bin/install -c|' ''; - outputs = [ "out" "info" ]; + postInstall = '' + rm $out/share/man/man1/* + cp ${buildPackages.coreutils}/share/man/man1/* $out/share/man/man1 + ''; - nativeBuildInputs = [ perl xz.bin ]; - configureFlags = - optional (singleBinary != false) - ("--enable-single-binary" + optionalString (isString singleBinary) "=${singleBinary}") - ++ optional stdenv.isSunOS "ac_cv_func_inotify_init=no" - ++ optional withPrefix "--program-prefix=g"; - - buildInputs = [ gmp ] - ++ optional aclSupport acl - ++ optional attrSupport attr - ++ optionals stdenv.isCygwin [ autoconf automake114x texinfo ] # due to patch - ++ optionals selinuxSupport [ libselinux libsepol ]; - - crossAttrs = { - buildInputs = [ gmp.crossDrv ] - ++ optional aclSupport acl.crossDrv - ++ optional attrSupport attr.crossDrv - ++ optionals selinuxSupport [ libselinux.crossDrv libsepol.crossDrv ] - ++ optional (stdenv.ccCross.libc ? libiconv) - stdenv.ccCross.libc.libiconv.crossDrv; - - # Prevents attempts of running 'help2man' on cross-built binaries. - PERL = "missing"; - - # Works around a bug with 8.26: - # Makefile:3440: *** Recursive variable 'INSTALL' references itself (eventually). Stop. - preInstall = '' - sed -i Makefile -e 's|^INSTALL =.*|INSTALL = ${self}/bin/install -c|' - ''; - - postInstall = '' - rm $out/share/man/man1/* - cp ${self}/share/man/man1/* $out/share/man/man1 - ''; - - # Needed for fstatfs() - # I don't know why it is not properly detected cross building with glibc. - configureFlags = [ "fu_cv_sys_stat_statfs2_bsize=yes" ]; - doCheck = false; - }; - - # The tests are known broken on Cygwin - # (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19025), - # Darwin (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19351), - # and {Open,Free}BSD. - # With non-standard storeDir: https://github.com/NixOS/nix/issues/512 - doCheck = stdenv ? glibc && builtins.storeDir == "/nix/store"; - - # Saw random failures like ‘help2man: can't get '--help' info from - # man/sha512sum.td/sha512sum’. - enableParallelBuilding = false; - - NIX_LDFLAGS = optionalString selinuxSupport "-lsepol"; - FORCE_UNSAFE_CONFIGURE = optionalString stdenv.isSunOS "1"; - - makeFlags = optionalString stdenv.isDarwin "CFLAGS=-D_FORTIFY_SOURCE=0"; - - meta = { - homepage = http://www.gnu.org/software/coreutils/; - description = "The basic file, shell and text manipulation utilities of the GNU operating system"; - - longDescription = '' - The GNU Core Utilities are the basic file, shell and text - manipulation utilities of the GNU operating system. These are - the core utilities which are expected to exist on every - operating system. - ''; - - license = licenses.gpl3Plus; - - platforms = platforms.all; - - maintainers = [ maintainers.eelco ]; - }; + # Needed for fstatfs() + # I don't know why it is not properly detected cross building with glibc. + configureFlags = [ "fu_cv_sys_stat_statfs2_bsize=yes" ]; + doCheck = false; }; -in - self + + # The tests are known broken on Cygwin + # (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19025), + # Darwin (http://thread.gmane.org/gmane.comp.gnu.core-utils.bugs/19351), + # and {Open,Free}BSD. + # With non-standard storeDir: https://github.com/NixOS/nix/issues/512 + doCheck = stdenv ? glibc && builtins.storeDir == "/nix/store"; + + # Saw random failures like ‘help2man: can't get '--help' info from + # man/sha512sum.td/sha512sum’. + enableParallelBuilding = false; + + NIX_LDFLAGS = optionalString selinuxSupport "-lsepol"; + FORCE_UNSAFE_CONFIGURE = optionalString stdenv.isSunOS "1"; + + makeFlags = optionalString stdenv.isDarwin "CFLAGS=-D_FORTIFY_SOURCE=0"; + + meta = { + homepage = http://www.gnu.org/software/coreutils/; + description = "The basic file, shell and text manipulation utilities of the GNU operating system"; + + longDescription = '' + The GNU Core Utilities are the basic file, shell and text + manipulation utilities of the GNU operating system. These are + the core utilities which are expected to exist on every + operating system. + ''; + + license = licenses.gpl3Plus; + + platforms = platforms.all; + + maintainers = [ maintainers.eelco ]; + }; +} diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix index 8e7bb79858f..1d924df6aea 100644 --- a/pkgs/top-level/all-packages.nix +++ b/pkgs/top-level/all-packages.nix @@ -10,26 +10,12 @@ self: pkgs: with pkgs; -let - defaultScope = pkgs // pkgs.xorg; -in - { # Allow callPackage to fill in the pkgs argument inherit pkgs; - # We use `callPackage' to be able to omit function arguments that - # can be obtained from `pkgs' or `pkgs.xorg' (i.e. `defaultScope'). - # Use `newScope' for sets of packages in `pkgs' (see e.g. `gnome' - # below). - callPackage = newScope {}; - - callPackages = lib.callPackagesWith defaultScope; - - newScope = extra: lib.callPackageWith (defaultScope // extra); - # Override system. This is useful to build i686 packages on x86_64-linux. forceSystem = system: kernel: nixpkgsFun { inherit system; @@ -39,15 +25,9 @@ in # Used by wine, firefox with debugging version of Flash, ... pkgsi686Linux = forceSystem "i686-linux" "i386"; - callPackage_i686 = lib.callPackageWith (pkgsi686Linux // pkgsi686Linux.xorg); + callPackage_i686 = pkgsi686Linux.callPackage; - forceNativeDrv = drv: - # Even when cross compiling, some packages come from the stdenv's - # bootstrapping package set. Those packages are only built for the native - # platform. - if crossSystem != null && drv ? crossDrv - then drv // { crossDrv = drv.nativeDrv; } - else drv; + forcedNativePackages = if hostPlatform == buildPlatform then pkgs else buildPackages; # A stdenv capable of building 32-bit binaries. On x86_64-linux, # it uses GCC compiled with multilib support; on i686-linux, it's @@ -3274,7 +3254,7 @@ in pngout = callPackage ../tools/graphics/pngout { }; hurdPartedCross = - if crossSystem != null && crossSystem.config == "i586-pc-gnu" + if targetPlatform != buildPlatform && targetPlatform.config == "i586-pc-gnu" then (makeOverridable ({ hurd }: (parted.override { @@ -4779,44 +4759,48 @@ in gccApple = throw "gccApple is no longer supported"; - gccCrossStageStatic = let + gccCrossStageStatic = assert targetPlatform != buildPlatform; let libcCross1 = if stdenv.cross.libc == "msvcrt" then windows.mingw_w64_headers else if stdenv.cross.libc == "libSystem" then darwin.xcode else null; in wrapGCCCross { - gcc = forceNativeDrv (gcc.cc.override { - cross = crossSystem; + gcc = forcedNativePackages.gcc.cc.override { + cross = targetPlatform; crossStageStatic = true; langCC = false; libcCross = libcCross1; enableShared = false; - }); + # Why is this needed? + inherit (forcedNativePackages) binutilsCross; + }; libc = libcCross1; binutils = binutilsCross; - cross = crossSystem; + cross = targetPlatform; }; # Only needed for mingw builds - gccCrossMingw2 = wrapGCCCross { + gccCrossMingw2 = assert targetPlatform != buildPlatform; wrapGCCCross { gcc = gccCrossStageStatic.gcc; libc = windows.mingw_headers2; binutils = binutilsCross; - cross = assert crossSystem != null; crossSystem; + cross = targetPlatform; }; - gccCrossStageFinal = wrapGCCCross { - gcc = forceNativeDrv (gcc.cc.override { - cross = crossSystem; + gccCrossStageFinal = assert targetPlatform != buildPlatform; wrapGCCCross { + gcc = forcedNativePackages.gcc.cc.override { + cross = targetPlatform; crossStageStatic = false; # XXX: We have troubles cross-compiling libstdc++ on MinGW (see # ), so don't even try. - langCC = crossSystem.config != "i686-pc-mingw32"; - }); + langCC = targetPlatform.config != "i686-pc-mingw32"; + # Why is this needed? + inherit (forcedNativePackages) binutilsCross; + }; libc = libcCross; binutils = binutilsCross; - cross = crossSystem; + cross = targetPlatform; }; gcc45 = lowPrio (wrapCC (callPackage ../development/compilers/gcc/4.5 { @@ -4834,7 +4818,7 @@ in # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; })); gcc48 = lowPrio (wrapCC (callPackage ../development/compilers/gcc/4.8 { @@ -4847,7 +4831,7 @@ in # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_14 else null; cloog = if !stdenv.isDarwin then cloog else null; @@ -4864,7 +4848,7 @@ in # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_11 else null; @@ -4881,7 +4865,7 @@ in # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_14 else null; })); @@ -4896,7 +4880,7 @@ in # and host != build), `cross' must be null but the cross-libc must still # be passed. cross = null; - libcCross = if crossSystem != null then libcCross else null; + libcCross = if targetPlatform != buildPlatform then libcCross else null; isl = if !stdenv.isDarwin then isl_0_14 else null; })); @@ -5029,7 +5013,7 @@ in # Haskell and GHC - haskell = callPackage ./haskell-packages.nix { inherit crossSystem; }; + haskell = callPackage ./haskell-packages.nix { }; haskellPackages = haskell.packages.ghc801.override { overrides = config.haskellPackageOverrides or (self: super: {}); @@ -5507,12 +5491,12 @@ in wrapGCCCross = {gcc, libc, binutils, cross, shell ? "", name ? "gcc-cross-wrapper"}: - forceNativeDrv (callPackage ../build-support/gcc-cross-wrapper { + forcedNativePackages.callPackage ../build-support/gcc-cross-wrapper { nativeTools = false; nativeLibc = false; noLibc = (libc == null); inherit gcc binutils libc shell name cross; - }); + }; # prolog yap = callPackage ../development/compilers/yap { }; @@ -6092,12 +6076,12 @@ in gold = false; }); - binutilsCross = assert crossSystem != null; lowPrio (forceNativeDrv ( - if crossSystem.libc == "libSystem" then darwin.cctools_cross - else binutils.override { + binutilsCross = assert targetPlatform != buildPlatform; lowPrio ( + if targetPlatform.libc == "libSystem" then darwin.cctools_cross + else forcedNativePackages.binutils.override { noSysDirs = true; - cross = crossSystem; - })); + cross = targetPlatform; + }); bison2 = callPackage ../development/tools/parsing/bison/2.x.nix { }; bison3 = callPackage ../development/tools/parsing/bison/3.x.nix { }; @@ -6566,9 +6550,9 @@ in cross_renaming: we should make all programs use pkgconfig as nativeBuildInput after the renaming. */ - pkgconfig = forceNativeDrv (callPackage ../development/tools/misc/pkgconfig { + pkgconfig = forcedNativePackages.callPackage ../development/tools/misc/pkgconfig { fetchurl = fetchurlBoot; - }); + }; pkgconfigUpstream = lowPrio (pkgconfig.override { vanilla = true; }); postiats-utilities = callPackage ../development/tools/postiats-utilities {}; @@ -6742,7 +6726,7 @@ in gdbGuile = lowPrio (gdb.override { inherit guile; }); gdbCross = lowPrio (callPackage ../development/tools/misc/gdb { - target = crossSystem; + target = if targetPlatform != buildPlatform then targetPlatform else null; }); gdb-multitarget = lowPrio (gdb.override { multitarget = true; }); @@ -7344,10 +7328,10 @@ in withGd = true; }; - glibcCross = forceNativeDrv (glibc.override { + glibcCross = forcedNativePackages.glibc.override { gccCross = gccCrossStageStatic; linuxHeaders = linuxHeadersCross; - }); + }; # We can choose: libcCrossChooser = name: if name == "glibc" then glibcCross @@ -7356,7 +7340,7 @@ in else if name == "libSystem" then darwin.xcode else throw "Unknown libc"; - libcCross = assert crossSystem != null; libcCrossChooser crossSystem.libc; + libcCross = assert targetPlatform != buildPlatform; libcCrossChooser targetPlatform.libc; # Only supported on Linux glibcLocales = if stdenv.isLinux then callPackage ../development/libraries/glibc/locales.nix { } else null; @@ -8265,9 +8249,9 @@ in # glibc provides libiconv so systems with glibc don't need to build libiconv # separately, but we also provide libiconvReal, which will always be a # standalone libiconv, just in case you want it - libiconv = if crossSystem != null then - (if crossSystem.libc == "glibc" then libcCross - else if crossSystem.libc == "libSystem" then darwin.libiconv + libiconv = if stdenv ? cross then + (if stdenv.cross.libc == "glibc" then libcCross + else if stdenv.cross.libc == "libSystem" then darwin.libiconv else libiconvReal) else if stdenv.isGlibc then glibcIconv stdenv.cc.libc else if stdenv.isDarwin then darwin.libiconv @@ -10918,8 +10902,8 @@ in cmdline = callPackage ../os-specific/darwin/command-line-tools {}; apple-source-releases = callPackage ../os-specific/darwin/apple-source-releases { }; in apple-source-releases // rec { - cctools_cross = callPackage (forceNativeDrv (callPackage ../os-specific/darwin/cctools/port.nix {}).cross) { - cross = assert crossSystem != null; crossSystem; + cctools_cross = callPackage (forcedNativePackages.callPackage ../os-specific/darwin/cctools/port.nix {}).cross { + cross = assert targetPlatform != buildPlatform; targetPlatform; inherit maloader; xctoolchain = xcode.toolchain; }; @@ -10992,7 +10976,7 @@ in libossp_uuid = callPackage ../development/libraries/libossp-uuid { }; libuuid = - if crossSystem != null && crossSystem.config == "i586-pc-gnu" + if targetPlatform != buildPlatform && targetPlatform.config == "i586-pc-gnu" then (utillinuxMinimal // { crossDrv = lib.overrideDerivation utillinuxMinimal.crossDrv (args: { # `libblkid' fails to build on GNU/Hurd. @@ -11077,7 +11061,7 @@ in # GNU/Hurd core packages. gnu = recurseIntoAttrs (callPackage ../os-specific/gnu { - inherit platform crossSystem; + inherit platform; }); hwdata = callPackage ../os-specific/linux/hwdata { }; @@ -11156,13 +11140,13 @@ in linuxHeaders = linuxHeaders_4_4; - linuxHeaders24Cross = forceNativeDrv (callPackage ../os-specific/linux/kernel-headers/2.4.nix { - cross = assert crossSystem != null; crossSystem; - }); + linuxHeaders24Cross = forcedNativePackages.callPackage ../os-specific/linux/kernel-headers/2.4.nix { + cross = assert targetPlatform != buildPlatform; targetPlatform; + }; - linuxHeaders26Cross = forceNativeDrv (callPackage ../os-specific/linux/kernel-headers/4.4.nix { - cross = assert crossSystem != null; crossSystem; - }); + linuxHeaders26Cross = forcedNativePackages.callPackage ../os-specific/linux/kernel-headers/4.4.nix { + cross = assert targetPlatform != buildPlatform; targetPlatform; + }; linuxHeaders_3_18 = callPackage ../os-specific/linux/kernel-headers/3.18.nix { }; @@ -11173,8 +11157,8 @@ in else if ver == "2.6" then linuxHeaders26Cross else throw "Unknown linux kernel version"; - linuxHeadersCross = assert crossSystem != null; - linuxHeadersCrossChooser crossSystem.platform.kernelMajor; + linuxHeadersCross = assert targetPlatform != buildPlatform; + linuxHeadersCrossChooser targetPlatform.platform.kernelMajor; kernelPatches = callPackage ../os-specific/linux/kernel/patches.nix { }; @@ -11839,7 +11823,7 @@ in uclibcCross = lowPrio (callPackage ../os-specific/linux/uclibc { linuxHeaders = linuxHeadersCross; gccCross = gccCrossStageStatic; - cross = assert crossSystem != null; crossSystem; + cross = assert targetPlatform != buildPlatform; targetPlatform; }); udev = systemd; diff --git a/pkgs/top-level/default.nix b/pkgs/top-level/default.nix index a146dad63bc..3c67d316f7c 100644 --- a/pkgs/top-level/default.nix +++ b/pkgs/top-level/default.nix @@ -83,7 +83,24 @@ in let boot = import ../stdenv/booter.nix { inherit lib allPackages; }; stages = stdenvStages { - inherit lib system platform crossSystem config overlays; + # One would think that `localSystem` and `crossSystem` overlap horribly with + # the three `*Platforms` (`buildPlatform`, `hostPlatform,` and + # `targetPlatform`; see `stage.nix` or the manual). Actually, those + # identifiers I, @Ericson2314, purposefully not used here to draw a subtle + # but important distinction: + # + # While the granularity of having 3 platforms is necessary to properly + # *build* packages, it is overkill for specifying the user's *intent* when + # making a build plan or package set. A simple "build vs deploy" dichotomy + # is adequate: the "sliding window" principle described in the manual shows + # how to interpolate between the these two "end points" to get the 3 + # platform triple for each bootstrapping stage. + # + # Also, less philosophically but quite practically, `crossSystem` should be + # null when one doesn't want to cross-compile, while the `*Platform`s are + # always non-null. `localSystem` is always non-null. + localSystem = { inherit system platform; }; + inherit lib crossSystem config overlays; }; pkgs = boot stages; diff --git a/pkgs/top-level/haskell-packages.nix b/pkgs/top-level/haskell-packages.nix index 5b14af145e9..7309121486e 100644 --- a/pkgs/top-level/haskell-packages.nix +++ b/pkgs/top-level/haskell-packages.nix @@ -1,4 +1,4 @@ -{ pkgs, callPackage, stdenv, crossSystem }: +{ pkgs, callPackage, stdenv, buildPlatform, targetPlatform }: rec { @@ -55,7 +55,7 @@ rec { ghcHEAD = callPackage ../development/compilers/ghc/head.nix rec { bootPkgs = packages.ghc7103; inherit (bootPkgs) alex happy; - inherit crossSystem; + inherit buildPlatform targetPlatform; selfPkgs = packages.ghcHEAD; }; ghcjs = packages.ghc7103.callPackage ../development/compilers/ghcjs { diff --git a/pkgs/top-level/release-cross.nix b/pkgs/top-level/release-cross.nix index f582bcf3b32..48f183162cf 100644 --- a/pkgs/top-level/release-cross.nix +++ b/pkgs/top-level/release-cross.nix @@ -32,8 +32,10 @@ let in { - # These `nativeDrv`s should be identical to their vanilla ones --- cross - # compiling should not affect the native derivation. + # These derivations from a cross package set's `buildPackages` should be + # identical to their vanilla equivalents --- none of these package should + # observe the target platform which is the only difference between those + # package sets. ensureUnaffected = let # Absurd values are fine here, as we are not building anything. In fact, # there probably a good idea to try to be "more parametric" --- i.e. avoid @@ -47,8 +49,12 @@ in # good idea lest there be some irrelevant pass-through debug attrs that # cause false negatives. testEqualOne = path: system: let - f = attrs: builtins.toString (lib.getAttrFromPath path (allPackages attrs)); - in assert f { inherit system; } == f { inherit system crossSystem; }; true; + f = path: attrs: builtins.toString (lib.getAttrFromPath path (allPackages attrs)); + in assert + f path { inherit system; } + == + f (["buildPackages"] ++ path) { inherit system crossSystem; }; + true; testEqual = path: systems: forAllSupportedSystems systems (testEqualOne path); diff --git a/pkgs/top-level/release-lib.nix b/pkgs/top-level/release-lib.nix index 5fe87711c01..04a57ef8063 100644 --- a/pkgs/top-level/release-lib.nix +++ b/pkgs/top-level/release-lib.nix @@ -76,8 +76,9 @@ rec { * parameter for allPackages, defining the target platform for cross builds, * and triggering the build of the host derivation (cross built - crossDrv). */ mapTestOnCross = crossSystem: mapAttrsRecursive - (path: systems: testOnCross crossSystem systems - (pkgs: addMetaAttrs { maintainers = crossMaintainers; } (getAttrFromPath path pkgs))); + (path: systems: testOnCross crossSystem systems (pkgs: addMetaAttrs + { maintainers = crossMaintainers; } + (getAttrFromPath path pkgs.splicedPackages))); /* Recursively map a (nested) set of derivations to an isomorphic diff --git a/pkgs/top-level/splice.nix b/pkgs/top-level/splice.nix new file mode 100644 index 00000000000..7afbd956d5b --- /dev/null +++ b/pkgs/top-level/splice.nix @@ -0,0 +1,83 @@ +# The `splicedPackages' package set, and its use by `callPackage` +# +# The `buildPackages` pkg set is a new concept, and the vast majority package +# expression (the other *.nix files) are not designed with it in mind. This +# presents us with a problem with how to get the right version (build-time vs +# run-time) of a package to a consumer that isn't used to thinking so cleverly. +# +# The solution is to splice the package sets together as we do below, so every +# `callPackage`d expression in fact gets both versions. Each# derivation (and +# each derivation's outputs) consists of the run-time version, augmented with a +# `nativeDrv` field for the build-time version, and `crossDrv` field for the +# run-time version. +# +# We could have used any names we want for the disambiguated versions, but +# `crossDrv` and `nativeDrv` were somewhat similarly used for the old +# cross-compiling infrastructure. The names are mostly invisible as +# `mkDerivation` knows how to pull out the right ones for `buildDepends` and +# friends, but a few packages use them directly, so it seemed efficient (to +# @Ericson2314) to reuse those names, at least initially, to minimize breakage. +# +# For performance reasons, rather than uniformally splice in all cases, we only +# do so when `pkgs` and `buildPackages` are distinct. The `actuallySplice` +# parameter there the boolean value of that equality check. +lib: pkgs: actuallySplice: + +let + defaultBuildScope = pkgs.buildPackages // pkgs.buildPackages.xorg; + # TODO(@Ericson2314): we shouldn't preclude run-time fetching by removing + # these attributes. We should have a more general solution for selecting + # whether `nativeDrv` or `crossDrv` is the default in `defaultScope`. + pkgsWithoutFetchers = lib.filterAttrs (n: _: !lib.hasPrefix "fetch" n) pkgs; + defaultRunScope = pkgsWithoutFetchers // pkgs.xorg; + + splicer = buildPkgs: runPkgs: let + mash = buildPkgs // runPkgs; + merge = name: { + inherit name; + value = let + defaultValue = mash.${name}; + buildValue = buildPkgs.${name} or {}; + runValue = runPkgs.${name} or {}; + augmentedValue = defaultValue + // (lib.optionalAttrs (buildPkgs ? ${name}) { nativeDrv = buildValue; }) + // (lib.optionalAttrs (runPkgs ? ${name}) { crossDrv = runValue; }); + # Get the set of outputs of a derivation + getOutputs = value: + lib.genAttrs (value.outputs or []) (output: value.${output}); + in + # Certain *Cross derivations will fail assertions, but we need their + # nativeDrv. We are assuming anything that fails to evaluate is an + # attrset (including derivation) and thus can be unioned. + if !(builtins.tryEval defaultValue).success then augmentedValue + # The derivation along with its outputs, which we recur + # on to splice them together. + else if lib.isDerivation defaultValue then augmentedValue + // splicer (getOutputs buildValue) (getOutputs runValue) + # Just recur on plain attrsets + else if lib.isAttrs defaultValue then splicer buildValue runValue + # Don't be fancy about non-derivations. But we could have used used + # `__functor__` for functions instead. + else defaultValue; + }; + in lib.listToAttrs (map merge (lib.attrNames mash)); + + splicedPackages = + if actuallySplice + then splicer defaultBuildScope defaultRunScope + else pkgs // pkgs.xorg; + +in + +{ + splicedPackages = splicedPackages // { recurseForDerivations = false; }; + + # We use `callPackage' to be able to omit function arguments that can be + # obtained `pkgs` or `buildPackages` and their `xorg` package sets. Use + # `newScope' for sets of packages in `pkgs' (see e.g. `gnome' below). + callPackage = pkgs.newScope {}; + + callPackages = lib.callPackagesWith splicedPackages; + + newScope = extra: lib.callPackageWith (splicedPackages // extra); +} diff --git a/pkgs/top-level/stage.nix b/pkgs/top-level/stage.nix index cbf65870eb7..6febedb79f3 100644 --- a/pkgs/top-level/stage.nix +++ b/pkgs/top-level/stage.nix @@ -9,8 +9,45 @@ import `pkgs/default.nix` or `default.nix`. */ -{ # The system (e.g., `i686-linux') for which to build the packages. - system +{ ## Misc parameters kept the same for all stages + ## + + # Utility functions, could just import but passing in for efficiency + lib + +, # Use to reevaluate Nixpkgs; a dirty hack that should be removed + nixpkgsFun + + ## Platform parameters + ## + ## The "build" "host" "target" terminology below comes from GNU Autotools. See + ## its documentation for more information on what those words mean. Note that + ## each should always be defined, even when not cross compiling. + ## + ## For purposes of bootstrapping, think of each stage as a "sliding window" + ## over a list of platforms. Specifically, the host platform of the previous + ## stage becomes the build platform of the current one, and likewise the + ## target platform of the previous stage becomes the host platform of the + ## current one. + ## + +, # The platform on which packages are built. Consists of `system`, a + # string (e.g.,`i686-linux') identifying the most import attributes of the + # build platform, and `platform` a set of other details. + buildPlatform + +, # The platform on which packages run. + hostPlatform + +, # The platform which build tools (especially compilers) build for in this stage, + targetPlatform + + ## Other parameters + ## + +, # The package set used at build-time. If null, `buildPackages` will + # be defined internally as the produced package set as itself. + buildPackages , # The standard environment to use for building packages. stdenv @@ -21,21 +58,19 @@ allowCustomOverrides , # Non-GNU/Linux OSes are currently "impure" platforms, with their libc - # outside of the store. Thus, GCC, GFortran, & co. must always look for - # files in standard system directories (/usr/include, etc.) - noSysDirs ? (system != "x86_64-freebsd" && system != "i686-freebsd" - && system != "x86_64-solaris" - && system != "x86_64-kfreebsd-gnu") + # outside of the store. Thus, GCC, GFortran, & co. must always look for files + # in standard system directories (/usr/include, etc.) + noSysDirs ? buildPlatform.system != "x86_64-freebsd" + && buildPlatform.system != "i686-freebsd" + && buildPlatform.system != "x86_64-solaris" + && buildPlatform.system != "x86_64-kfreebsd-gnu" , # The configuration attribute set config -, overlays # List of overlays to use in the fix-point. - -, crossSystem -, platform -, lib -, nixpkgsFun +, # A list of overlays (Additional `self: super: { .. }` customization + # functions) to be fixed together in the produced package set + overlays }: let @@ -50,11 +85,28 @@ let }; stdenvBootstappingAndPlatforms = self: super: { - stdenv = stdenv // { inherit platform; }; - inherit - system platform crossSystem; + buildPackages = (if buildPackages == null then self else buildPackages) + // { recurseForDerivations = false; }; + inherit stdenv + buildPlatform hostPlatform targetPlatform; }; + # The old identifiers for cross-compiling. These should eventually be removed, + # and the packages that rely on them refactored accordingly. + platformCompat = self: super: let + # TODO(@Ericson2314) this causes infinite recursion + #inherit (self) buildPlatform hostPlatform targetPlatform; + in { + stdenv = super.stdenv // { + inherit (buildPlatform) platform; + } // lib.optionalAttrs (targetPlatform != buildPlatform) { + cross = targetPlatform; + }; + inherit (buildPlatform) system platform; + }; + + splice = self: super: import ./splice.nix lib self (buildPackages != null); + allPackages = self: super: let res = import ./all-packages.nix { inherit lib nixpkgsFun noSysDirs config; } @@ -83,8 +135,10 @@ let # The complete chain of package set builders, applied from top to bottom toFix = lib.foldl' (lib.flip lib.extends) (self: {}) ([ stdenvBootstappingAndPlatforms + platformCompat stdenvAdapters trivialBuilders + splice allPackages aliases stdenvOverrides