Note that a bunch of non-python packages use this attribute already.
Some of those are clearly unaware of the fact that this attribute does
not exists in stdenv because they define it but don't to add it to
their `bulidInputs` :)
Also note that I use `buildInputs` here and only handle regular
builds because python and haskell builders do it this way and I'm not
sure how to properly handle the cross-compilation case.
upstream issue:
https://bugs.python.org/issue31940
There are two PR's proposed to fix this,
but both seem to be stalling waiting for review.
I previously used what appears to be the favored
of the two approaches[1] to fix this,
with plan of keeping it musl-only until PR was merged.
However, while writing up a commit message
explaining the problem and why it needed fixing...
I investigated a bit and found it increasingly
hard to justify anything other than ...
simply not using lchmod.
Here's what I found:
* lchmod is non-POSIX, seems BSD-only these days
* Functionality of lchmod isn't supported on Linux
* best scenario on Linux would be an error
* POSIX does provide lchmod-esque functionality
with fchmodat(), which AFAICT is generally preferred.
* Python intentionally overlooks fchmodat()[2]
electing instead to use lchmod() behavior
as a proxy for whether fchmodat() "works".
I'm not sure I follow their reasoning...
* both glibc and musl provide lchmod impls:
* glibc returns ENOSYS "not implemented"
* musl implements lchmod with fchmodat(),
and so returns EOPNOTSUPP "op not supported"
* Python doesn't expect EOPNOTSUPP from lchmod,
since it's not valid on BSD's lchmod.
* "configure" doesn't actually check lchmod usefully,
instead checks for glibc preprocessor defines
to indicate if the function is just a stub[3];
somewhat fittingly, if the magic macros are defined
then the next line of the C source is "choke me",
causing the compiler to trip, fall, and point
a finger at whatever is near where it ends up.
(somewhat amusing, but AFAIK effective way to get an error :P)
I'm leaving out links to threads on mailing lists and such,
but for now I hope I've convinced you
(or to those reading commit history: explained my reasons)
that this is a bit of a mess[4].
And so instead of making a big mess messier,
and with hopes of never thinking about this again,
I propose we simply tell Python "don't use lchmod" on Linux.
[1] https://github.com/python/cpython/pull/4783
[2] 28453feaa8/Lib/os.py (L144)
[3] 28453feaa8/configure (L2198)
[4] Messes happen, no good intention goes unpunished :).
Documents the reason why it's needed and also prevents the
ensureNewerSourcesHook call being evaluated again and again for every
single Python package.
This hardcoded list of CFFI extension modules gets stale when PyPy adds
more, but fortunately the main translation step already builds these now
(hack_for_cffi_modules in pypy/goal/targetpypystandalone.py).
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Wrappers already included `PYTHONNOUSERSITE=1`, but now this env var is
also set in the Python setup hook. This improves purity in case of
non-sandboxes builds and nix-shell.
When a Python script has the extension `.py`, bytecode is generated.
Typically, executables in bin have no extension, so no bytecode is
generated. However, some packages do provide executables with
extensions, and thus bytecode is generated.
28299f669a introduced the first Python
packages having multiple outputs. The required outputs were not picked
up by `python.buildEnv` (#31857).
This commit modifies `python.buildEnv` so that it always includes the
$out output and thus fixes#31857.
Python libraries or modules now have an attribute `pythonModule = interpreter;` to indicate
they provide Python modules for the specified `interpreter`.
The package set provides the following helper functions:
- hasPythonModule: Check whether a derivation provides a Python module.
- requiredPythonModules: Recurse into a list of Python modules, returning all Python modules that are required.
- makePythonPath: Create a PYTHONPATH from a list of Python modules.
Also included in this commit is:
- disabledIf: Helper function for disabling non-buildPythonPackage functions.
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.
This is an incremental step to fully keeping the dependencies separate
in all cases.
In the maintenance release bump in
90059701a8 a certain change to /test/ was
backported from Python 3:
- bpo-30207: To simplify backports from Python 3, the test.test_support
module was converted into a package and renamed to test.support. The
test.script_helper module was moved into the test.support package.
Names test.test_support and test.script_helper are left as aliases to
test.support and test.support.script_helper.
libffi needs a patch to actually work on aarch64 (or the cffi Python package
fails its testsuite). Of course the bundled version of libffi has the
same bug, so don't use the buggy version on aarch64.
Python3 already uses the system libffi on all platforms. I don't know
why Python2 doesn't.
While we tell pip not to fetch (with the `--no-index` option),
`setuptools` can do so itself. In the past we used a `distutils.cfg`
with `allow-hosts = None` to prevent setuptools from fetching itself.
This was removed when we started building wheels in
2562f94de4e4fd2ddc677187fa2e2848L69.
The `dist-utils.cfg` code was still there, and adding it to
`buildInputs` is sufficient.
Tested with python.pkgs.passlib by removing the `checkInputs` / `nose`.
test.{support, regrtest} are the internal packages cpython
developers use to write tests.
Although they are not public and the API may change/break
some developers use these packages to write tests for their
(3rd party) software.
The derivations for cpython now only remove the actual tests
but leave the packages in place that are used to write them.
Discussion: https://github.com/NixOS/nixpkgs/pull/28540
Symbolic links were added pointing to the executables that end with 3 as
part of the Python 2 to 3 migration. At some point I disabled ensurepip
but forgot to remove this symbolic link.
Symbolic links were added pointing to the executables that end with 3 as
part of the Python 2 to 3 migration. At some point I disabled ensurepip
but forgot to remove this symbolic link.
Symbolic links were added pointing to the executables that end with 3 as
part of the Python 2 to 3 migration. At some point I disabled ensurepip
but forgot to remove this symbolic link.
* pkgs: refactor needless quoting of homepage meta attribute
A lot of packages are needlessly quoting the homepage meta attribute
(about 1400, 22%), this commit refactors all of those instances.
* pkgs: Fixing some links that were wrongfully unquoted in the previous
commit
* Fixed some instances
Python envs did not pass through any of the properties the Python
interpreter has. That could be annoying, especially not having
`python.interpreter` which is the path to the interpreter. This commit
fixes the situation and inherit python.passthru.
Thus far all executables in a derivation were wrapped. This commit
only wraps executables in $out/bin. If other scripts need to be wrapped
as well, then one can use wrapPythonProgramsIn.
When tests are disabled, we do not want to pass checkInputs to
stdenv.mkDerivation. This reduces the build requirements and, more
importantly, helps cutting cycles.
The Python interpreters are patched so they can build .pyc bytecode free
of certain indeterminism.
When building Python packages we currently set
```
compiling python files.
in nix store.
DETERMINISTIC_BUILD=1;
PYTHONHASHSEED = 0;
```
Instead if setting these environment variables in the function that
builds the package, this commit sets the variables instead in the Python
setup hook. That way, whenever Python is included in a derivation, these
variables are set.
See also the issue https://github.com/NixOS/nixpkgs/issues/25707.
This commit fixes several issues:
- as reported in https://github.com/NixOS/nixpkgs/issues/24924 it was
possible that the file _sysconfigdata.pyc was generated after the actual
build of the CPython interpreter. We forgot to regenerate that file
during the build. This is now fixed
- the expression of the 3.3 interpreter now also includes some of the
determinism patches even though the output isn't yet reproducible. The
reason for adding them is that this makes the expressions of the
different interpreters more similar.
- references to -dev packages are now also removed in the 3.6 package,
thereby reducing its closure size
Python does add the script's directory into "sys.path". For the case of
"catch_conflicts.py" this means "/nix/store" is added to "sys.path". This can
result in very long delays if the store contains a lot of entries.
(moved from master commit 76213d102c)
From the manual:
> This attribute should be a number, with a higher value denoting a
lower priority. The default priority is 0.
Just passing -5 or -10 wasn't sufficient, so let's make it -100.
- Windows installers are indeterministic and we don't need them.
- since Python 3 ensurepip is installed by default. pip is indeteministic and we don't need it.
- rebuild bytecode to ensure its deterministic
Certain programs, like zim, calibre and now also apparently mercurial,
rely on sys.argv[0] providing not just the script name but the full
path.
The Python docs [1] state the following on the matter:
> argv[0] is the script name (it is operating system dependent whether
this is a full pathname or not).
Therefore, scripts should not expect to receive a full path.
Unfortunately some do. While this can be considered a bug, there doesn't
seem any reason not to provide the full path. Therefore we now provide
the full path.
[1]
https://docs.python.org/3.5/library/sys.html?highlight=sys.argv#sys.argv
A package set is constructed for a specific interpreter. Therefore, we add the
possibility to override the package set to the interpreter. This should make it
easier to override the interpreter and the package set at the same time.
In #19309 a separate output for tkinter was added.
Several dependencies of Python depend indirectly on Python. We have the
following two paths:
```
‘python-2.7.12’ - ‘tk-8.6.6’ - ‘libXft-2.3.2’ - ‘libXrender-0.9.10’ -
‘libX11-1.6.4’ - ‘libxcb-1.12’ - ‘libxslt-1.1.29’- ‘libxml2-2.9.4’ -
‘python-2.7.12’
‘python-2.7.12’ - ‘tk-8.6.6’ - ‘libXft-2.3.2’ - ‘fontconfig-2.12.1’ -
‘dejavu-fonts-2.37’ - ‘fontforge-20160404’ - ‘python-2.7.12’
```
Because only `tkinter` needs this, I added
```
pythonSmall = python.override {x11Support = false;};
```
to break the infinite recursion. We also still have the output
`tkinter`.
However, we might as well build without x11Support by default. Then we build with x11Support as well so we get the tkinter module and put that in a separate package.
python.buildenv is used to build an env that provides binaries that can
import all modules that were passed in to the env.
Before this change it filtered the propagatedBuildInputs to remove all
non-Python packages, thereby possibly reducing the amount of packages
that were referenced. However, Python packages often don't have non-
Python packages as propagatedBuildInputs. And occasionally, we do want
to be able to add other packages to the env.
It's a long build and generally painful to split into smaller commits,
so I apologize for lumping many changes into one commit but this is far
easier.
There are still several outdated parts of the darwin stdenv but these
changes should bring us closer to the goal.
Fixes#18461
Compiling python with "-Wl,-stack_size,1000000" causes problems when
compiling for example pygobject3. pygobject3 uses "python3.x-config
--ldflags" during installation and then fails when
"-Wl,-stack_size,1000000" is present. Maybe we should investigate
removing this during the build of pyobject3, but this stack_size flag is
also not used on the popular darwin homebrew-core channel for python3.5,
so it seems safe to remove it.
This one was already merged into release-16.09, so let's not have the
stable branch is ahead of master and confuse things. In addition to
that, currently we have an odd situation that master has less things
actually finished building than in staging.
Conflicts:
pkgs/data/documentation/man-pages/default.nix
The previous commit revealed that Python wasn't actually using
Berkeley DB; it only had it in its closure due to the build-time flag
dump in Makefile and _sysconfigdata.py. When Python detects both GNU
gdbm and Berkeley DB at build time, it will use the former.
This reduces Python's closure size from 200 MiB to 129 MiB. Even
better would be to get move tkinter to a separate output or package
(since that would get rid of all X11 stuff), but that's a bit harder.
In line with the Nixpkgs manual.
A mechanical change, done with this command:
find pkgs -name "*.nix" | \
while read f; do \
sed -e 's/description\s*=\s*"\([a-z]\)/description = "\u\1/' -i "$f"; \
done
I manually skipped some:
* Descriptions starting with an abbreviation, a user name or package name
* Frequently generated expressions (haskell-packages.nix)
The location of this 'regen' script (whatever is does) depends on the
platform. AFAICT it always exists due to this:
$(srcdir)/Lib/$(PLATDIR):
mkdir $(srcdir)/Lib/$(PLATDIR)
cp $(srcdir)/Lib/plat-generic/regen $(srcdir)/Lib/$(PLATDIR)/regen
Currently, when constructing a buildEnv and adding packages via
extraLibs, then binaries in extraLibs cannot access the other Python
modules. An example is having ipython/jupyter in extraLibs; in that case
ipython cannot import any other modules.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
Without this, python will silently build without native extensions (e.g
openssl); this causes wget to fail its tests for instance.
vcunat added python-3.5.
Fixes regression from 086e801d51.
The switch tu UCS-4 only happened for the main Python interpreter and
libraries, but the extension modules were using the same source but
without any configureFlags, so the extensions still referred to UCS-2
symbols.
Tested module builds of Python 2.7 and all modules except crypto fail.
Also tested against sqlite3 module of Python 2.6, although for other
modules there still seems to be an unrelated build failure with Tcl/Tk
libraries, so we might need to fix that later.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
A build output of such a failure can be found here:
http://hydra.nixos.org/build/21994789/nixlog/1/raw
The build_ext command doesn't return non-zero if it fails but instead
produces a file with a "_failed.so" suffix, which we're now checking
after the build_ext command.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
There was a typo that failed them.
Also, drop pkgconfig again, as it seems unused; and fix python26.
Now all python*.modules build, except for python26.tkinter
which fails on master as well.
- test/test_support.py module is used by many libraries which backport python
3.0 functionality to 2.6/2.7
- enable subprocess32 tests (now passing for python 2.6/2.7)
1) Make the core python libraries deterministic.
2) Make the python libraries created by glib deterministic.
@vcunat put back the removal of share/gtk-doc in glib.
The default setting for extraLibs used to be the set of modules that come with
python by default but aren't usually enabled in our standard python derivation
because they require additional libraries. This meant that users who want to
*add* libraries to that set had to use a fairly complicated override, to add
more entries without loosing the ones set by default.
After this patch, the "standard libraries" such as "curses' are listed in
stdLibs while the extraLibs argument remains empty by default. This allows
users to override extraLibs without overriding the standard libraries.
Furthermore, the wrapper environment can be messed around with in an
additional 'postBuild' step. One nice application of this build step is
to patch scripts and binaries to use the wrapped python interpreter
instead of the pristine one, thereby enabling them to pick up all
modules that have been configured. The following example shows how this
is done for the 'pylint' utility:
pkgs.python27Full.override {
extraLibs = [pkgs.pylint];
postBuild = ''
cd ${pkgs.pylint}/bin
for i in *; do
rm $out/bin/$i
sed -r -e "s|^exec |exec $out/bin/python -- |" <$i >$out/bin/$i
chmod +x $out/bin/$i
done;
'';
};
The new wrapper creates an environment that contains all files from
Python and the extra libraries that have been specified. All files are
found at run-time by means of the $PYTHONHOME variable; the wrapper no
longer uses $PYTHONPATH.
http://bugs.python.org/issue15833
When Python 3.3.0 attempts to compile python bytecode in the system
directories it raises and exception and stops. Since Python 3.3 is
only required by the latest Blender, I hope it's OK to use the RC
until the final release.
- updated to version 3.2.3
- fixed version numbers in setup-hook.sh
- sort input variable list alphabetically
- tk support needs x11 support as well
- deleted obsolete version 3.1
Furthermore, all CPPFLAGS and LDFLAGS required to find the build inputs are now
explicitly passed to ./configure. The python 3.x build systems tries to ignore
the environment as much as possible to provide deterministic builds. This means
that our magic environment variable stuff won't work, and previously built
python3 binaries lacked all kinds of extensions, such as zlib, etc.
svn path=/nixpkgs/trunk/; revision=33937
The build expression for python contains code that patches all occurrences of
impure paths like "/usr" and "/opt" out of "setup.py". The same code must be
run when building a python module, too.
svn path=/nixpkgs/trunk/; revision=27164
remove the pythonModules attribute. The built-in modules are now
accessible as (e.g.) "python.modules.ssl" or "pythonPackages.ssl".
svn path=/nixpkgs/branches/modular-python/; revision=26559
differs from the mtime of the source file. This doesn't work in Nix
because Nix changes the mtime of files in the Nix store to 1. So
treat that as a special case.
Yes, this should make Python programs faster.
svn path=/nixpkgs/branches/modular-python/; revision=25387
additional dependencies (e.g. SQLite, X11, or Tcl/Tk) outside the
main Python package (i.e., pythonBase). This makes pythonFull
unnecessary: you can just pass the additional modules as
buildInputs to packages that require them, e.g.
buildInputs = [ pythonModules.sqlite3 ];
svn path=/nixpkgs/branches/modular-python/; revision=25364
The python wrapper expression expects a list of Python modules, $extraLibs,
which are added to $PYTHONPATH before executing the actual Python interpreter.
svn path=/nixpkgs/trunk/; revision=23194
If a build expressions has set "enableParallelBuilding = true", then the
generic builder may utilize more than one CPU core to build that particular
expression. This feature works out of the box for GNU Make. Expressions that
use other build drivers like Boost.Jam or SCons have to specify appropriate
flags such as "-j${NIX_BUILD_CORES}" themselves.
svn path=/nixpkgs/trunk/; revision=23042
Python version 3.x is really quite different from 2.x. Using a new name
for the package prevents Nix from performing unintended updates to the
3.x version.
svn path=/nixpkgs/trunk/; revision=21264
- cleanup python libraries:
* moving all python libraries into a attr set into a directory
so that expressions can be used for both: python 2.5 and 2.6 easily
* disabling packages which don't build
svn path=/nixpkgs/trunk/; revision=21142
* Dropped "nolongdouble.patch". The patch no longer applies to Python 2.6, and
apparently isn't required anymore either.
* Added access to native Darwin arch utility. Python tries to run 'arch' in
the configure stage, but that binary reside in /usr/bin. To make it
available to the expression, the small wrapper darwinArchUtility is added as
a buildInput if appropriate.
* Don't pass --enable-shared. The build fails if we try to enable building of
shared libraries, apparently because some required libraries aren't linked,
i.e. the linker call isn't right.
TODO:
* Figure out how to enable shared linking.
* The resulting binary on Darwin seem to lack the binascii module.
svn path=/nixpkgs/trunk/; revision=17894
The build succeeds on i686-linux. Other platforms look good, too,
because there were hardly any changes necessary to update the expression
from 2.5.
svn path=/nixpkgs/trunk/; revision=17889
builders. These are redundant now.
* Inlined some trivial builders.
* Removed a few explicit setup-hook creations. This is done
automatically now if setupHook is set.
* Deleted the initscripts package. NixOS doesn't use it anymore.
svn path=/nixpkgs/branches/stdenv-updates/; revision=15276
Orivej Desh
Dmitry Kalinkin
Vladimír Čunát
Jörg Thalheim
Andy Dirnberger
Frederik Rietdijk
Timo Kaufmann
Matthew Justin Bauer
Michael Raskin
Jan Malakhovski
Will Dietz
Josef Kemetmüller
Ben Gamari
Tuomas Tynkkynen
Anders Kaseorg
John Ericson
John Ericson
Frederik Rietdijk
Shea Levy
Shea Levy
Dan Peebles
Jon Banafato
Nikolay Amiantov
Tuomas Tynkkynen
Maximilian Güntner
Robin Gloster
Herwig Hochleitner
Silvan Mosberger
David McFarland
Robert Scott
Johannes Bornhold
Domen Kožar
Tim Cuthbertson
Charles Strahan
Symphorien Gibol
Daiderd Jordan
Aneesh Agrawal
Louis Taylor
Eelco Dolstra
Josef Kemetmueller
Franz Pletz
Bjørn Forsman
Benno Fünfstück
Peter Simons
Rok Garbas
Luca Bruno
Jude Taylor
Asko Soukka
William A. Kennington III
James Cook
Florian Friesdorf
aszlig
Dario Bertini
John Wiegley
Luca Bruno
Jascha Geerds
Alexander Kjeldaas
Charles Strahan
Ricardo M. Correia
Georges Dubus
![](https://secure.gravatar.com/avatar/2f93d08a773b27d6548598c5f45aa3a9?d=identicon&s=56 "Jason \\"Don\\" O'Conal"){kind=small}
Jason \"Don\" O'Conal
Evgeny Egorochkin
Cillian de Róiste
Lluís Batlle i Rossell
Ludovic Courtès
Yury G. Kudryashov
Alexander Tsamutali
Cillian de Roiste
Marc Weber
Lluís Batlle i Rossell
Rob Vermaas
Armijn Hemel