public
/
nixpkgs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
295,481 Commits 1 Branch 0 Tags 1.5 GiB
Commit Graph

122 Commits

Author SHA1 Message Date
Timothy DeHerrera 781b1f8e3a create-amis.sh: fix typo 
(cherry picked from commit 3988440b124fc0576fda52d9753a640c1304dcd1)
 2021-10-07 09:43:49 -07:00
Timothy DeHerrera db82eef3ec create-amis.sh: use status message 
The progress ID is fairly useless. Status message is more useful for
humans.

(cherry picked from commit 4410ccc211c9127f343d8dceada26e9ba0f72687)
 2021-10-07 09:43:49 -07:00
Timothy DeHerrera ff568a49ec create-amis.sh: add support for the ZFS AMIs 
(cherry picked from commit 46bbf738eaebf82cf59391d38f5a9d77ecf53049)
 2021-10-07 09:43:49 -07:00
Timothy DeHerrera 9e5f907d1d create-amis.sh: allow uploading private AMIs 
(cherry picked from commit 423a70d4ee31bc9b1bbac3ccbb87b7085826fd81)
 2021-10-07 09:43:49 -07:00
Timothy DeHerrera e33873f610 create-amis.sh: make vars overridable from env 
(cherry picked from commit 0ffd7d80cab0e5e3e152d678e0107be3438cfda8)
 2021-10-07 09:43:49 -07:00
Graham Christensen 337eb213cf amazon images: extend the image-info.json to have a disks object 
Having a disks object with a dictionary of all the disks and their
properties makes it easier to process multi-disk images.

Note the rename of `label` to `system_label` is because `$label`i
is something of a special token to jq.

(cherry picked from commit 71b3d18181974523ef7264106a4bf213a3103439)
 2021-09-13 16:20:47 +00:00
Graham Christensen a989fd1885 NixOS/amazonImageZfs: init 
Introduce an AWS EC2 AMI which supports aarch64 and x86_64 with a ZFS
root.

This uses `make-zfs-image` which implies two EBS volumes are needed
inside EC2, one for boot, one for root. It should not matter which
is identified `xvda` and which is `xvdb`, though I have always
uploaded `boot` as `xvda`.

(cherry picked from commit bd38b059eae05871579b2dfd51cd41d058b6a1ec)
 2021-09-13 16:20:46 +00:00
Luke Granger-Brown 87c3b7e767 amazonImage: make statically sized again 
For reasons we haven't been able to work out, the aarch64 EC2 image now
regularly exceeds the output image size on hydra.nixos.org. As a
workaround, set this back to being statically sized again.

The other images do seem to build - it's just a case of the EC2 image
now being too large (occasionally non-determinstically).
 2021-05-01 02:19:42 +00:00
lassulus 5aa4273e4f treewide: use auto diskSize for make-disk-image 
(cherry picked from commit f3aa040bcb)
 2021-04-24 14:49:07 -04:00
Luke Granger-Brown 4fb91cbafe Revert "treewide: use auto diskSize for make-disk-image" 
This reverts commit f3aa040bcb.
 2021-04-24 02:38:36 +00:00
Luke Granger-Brown f521b12b0e Revert "nixos/amazon-image: (temporarily) use fixed disk size again" 
This reverts commit 6a8359a92a.
 2021-04-24 02:38:25 +00:00
Luke Granger-Brown 6a8359a92a nixos/amazon-image: (temporarily) use fixed disk size again 
As a temporary workaround for #120473 while the image builder is patched
to correctly look up disk sizes, partially revert
f3aa040bcb for EC2 disk images only.

We retain the type allowing "auto" but set the default back to the
previous value.
 2021-04-24 00:43:47 +00:00
lassulus f3aa040bcb treewide: use auto diskSize for make-disk-image 2021-04-22 19:52:49 +02:00
AmineChikhaoui 606b49721f
add new Google Cloud image for the current release 
update the create-gce.sh script with the ability to create public images
out of a GS object.
 2021-03-21 14:04:09 -04:00
Graham Christensen 7092dd52f8
amazonImage: Upload disks as GP3 for cheaper & faster IO (#109027) 
GP3 is always faster and cheaper than GP2, so sticking to GP2 is
leaving money on the table.

https://cloudwiry.com/ebs-gp3-vs-gp2-pricing-comparison/
 2021-01-11 13:54:40 -05:00
Graham Christensen 38a394bdee
Merge pull request #102174 from grahamc/ami-root-use-gpt 
AMI root partition table: use GPT to support >2T partitions
 2020-10-30 16:14:37 -04:00
Graham Christensen d77ddf2a40
nixos.amazonAmi: use legacy+gpt disk images to support partitions >2T 2020-10-30 15:50:25 -04:00
Graham Christensen 74a577b293
create-amis: improve wording around the service name's IAM role 
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2020-10-30 12:40:17 -04:00
Graham Christensen 2bf1fc0345
create-amis: allow customizing the service role name 
The complete setup on the AWS end can be configured
with the following Terraform configuration. It generates
a ./credentials.sh which I just copy/pasted in to the
create-amis.sh script near the top. Note: the entire stack
of users and bucket can be destroyed at the end of the
import.

    variable "region" {
      type = string
    }
    variable "availability_zone" {
      type = string
    }

    provider "aws" {
      region = var.region
    }

    resource "aws_s3_bucket" "nixos-amis" {
      bucket_prefix = "nixos-amis-"
      lifecycle_rule {
        enabled = true
        abort_incomplete_multipart_upload_days = 1
        expiration {
          days = 7
        }
      }
    }

    resource "local_file" "credential-file" {
      file_permission = "0700"
      filename = "${path.module}/credentials.sh"
      sensitive_content = <<SCRIPT
    export service_role_name="${aws_iam_role.vmimport.name}"
    export bucket="${aws_s3_bucket.nixos-amis.bucket}"
    export AWS_ACCESS_KEY_ID="${aws_iam_access_key.uploader.id}"
    export AWS_SECRET_ACCESS_KEY="${aws_iam_access_key.uploader.secret}"
    SCRIPT
    }

    # The following resources are for the *uploader*
    resource "aws_iam_user" "uploader" {
      name = "nixos-amis-uploader"
    }

    resource "aws_iam_access_key" "uploader" {
      user = aws_iam_user.uploader.name
    }

    resource "aws_iam_user_policy" "upload-to-nixos-amis" {
      user = aws_iam_user.uploader.name

      policy = data.aws_iam_policy_document.upload-policy-document.json
    }

    data "aws_iam_policy_document" "upload-policy-document" {
      statement {
        effect = "Allow"

        actions = [
          "s3:ListBucket",
          "s3:GetBucketLocation",
        ]

        resources = [
          aws_s3_bucket.nixos-amis.arn
        ]
      }

      statement {
        effect = "Allow"

        actions = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
        ]

        resources = [
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }

      statement {
        effect = "Allow"
        actions = [
          "ec2:ImportSnapshot",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:RegisterImage",
          "ec2:DescribeImages"
        ]
        resources = [
          "*"
        ]
      }
    }

    # The following resources are for the *vmimport service user*
    # See: https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#vmimport-role
    resource "aws_iam_role" "vmimport" {
      assume_role_policy = data.aws_iam_policy_document.vmimport-trust.json
    }

    resource "aws_iam_role_policy" "vmimport-access" {
      role = aws_iam_role.vmimport.id
      policy = data.aws_iam_policy_document.vmimport-access.json
    }

    data "aws_iam_policy_document" "vmimport-access" {
      statement {
        effect = "Allow"
        actions = [
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
        ]
        resources = [
          aws_s3_bucket.nixos-amis.arn,
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }
      statement {
        effect = "Allow"
        actions = [
          "ec2:ModifySnapshotAttribute",
          "ec2:CopySnapshot",
          "ec2:RegisterImage",
          "ec2:Describe*"
        ]
        resources = [
          "*"
        ]
      }
    }

    data "aws_iam_policy_document" "vmimport-trust" {
      statement {
        effect = "Allow"
        principals {
          type = "Service"
          identifiers = [ "vmie.amazonaws.com" ]
        }

        actions = [
          "sts:AssumeRole"
        ]

        condition {
          test = "StringEquals"
          variable = "sts:ExternalId"
          values = [ "vmimport" ]
        }
      }
    }
 2020-10-30 12:12:08 -04:00
Graham Christensen e253de8a77
create-amis.sh: log the full response if describing the import snapshot tasks fails 2020-10-30 12:08:01 -04:00
Graham Christensen f92a883ddb
nixos ec2/create-amis.sh: shellcheck: $ is not needed in arithmetic 2020-10-30 12:08:01 -04:00
Graham Christensen 7dac8470cf
nixos ec2/create-amis.sh: shellcheck: explicitly make the additions to block_device_mappings single strings 2020-10-30 12:08:00 -04:00
Graham Christensen a66a22ca54
nixos ec2/create-amis.sh: shellcheck: read without -r mangles backslashes 2020-10-30 12:08:00 -04:00
Graham Christensen baf7ed3f24
nixos ec2/create-amis.sh: shellcheck: SC2155: Declare and assign separately to avoid masking return values. 2020-10-30 12:07:59 -04:00
Graham Christensen f5994c208d
nixos ec2/create-amis.sh: shellcheck: quote state_dir reference 2020-10-30 12:07:59 -04:00
Graham Christensen c76692192a
nixos ec2/create-amis.sh: shellcheck: quote region references 2020-10-30 12:07:49 -04:00
Lassulus d08b81c3b5
Merge pull request #89116 from wagdav/fix-args-create-amis 
nixos/maintainers/scripts/ec2/create-amis.sh: fix argument check
 2020-08-22 16:47:54 +02:00
Jörg Thalheim 0cb79c953d nixos/ec2: remove dependency on NIX_PATH 
This is required when migrating to flakes
 2020-08-16 10:56:44 +00:00
zowoq 2b5659c700 nixos/maintainers/*: editorconfig fixes 2020-08-04 00:23:54 +10:00
David Wagner 3b1ed035c3 create-amis: fix argument check 
Because this script enables `set -u` when no arguments are provided bash
exits with the error:

    $1: unbound variable

instead of the helpful usage message.
 2020-05-28 17:41:45 +02:00
Cole Mickens 7c7e76450b
nixos/azure-new: use local nixpkgs 2020-04-27 02:11:10 -07:00
Cole Mickens 1992768157 nixos/azure: clarify how users work in basic example 2020-03-29 13:56:55 -07:00
Cole Mickens 627ae7e057 nixos/azure: upload-image.sh cleanup $1 handling 2020-03-29 13:56:55 -07:00
Cole Mickens a5de97f21e nixos/azure: upload-image names the image better 2020-03-29 13:56:55 -07:00
Cole Mickens c2b2cc6dbd nixos/azure: simplify example image 2020-03-29 13:56:55 -07:00
Cole Mickens 20f981de08 azure: init nixos/maintainers/scripts/azure-new 2020-03-29 13:56:55 -07:00
Benjamin Hipple 129176452c nixos-ami: update nvme_core.io_timeout for linux kernel >= 4.15 
NixOS 20.03 is built on kernel 5.4 and 19.09 is on 4.19, so we should update
this option to the highest value possible, per linked upstream instructions from
Amazon.
 2020-03-22 00:35:56 -04:00
adisbladis 4e5b0571ed
create-amis: Add eu-north-1 2020-03-05 18:00:28 +00:00
Alyssa Ross 65dcd244bc maintainers/create-azure.sh: run from anywhere 
I'm not really sure how the line directly after ended up with this,
but this line didn't...
 2020-01-09 20:54:28 +00:00
Andrew Childs bd61216f55 ec2/create-amis.sh: register root device as /dev/xvda 
For the case of blkfront drives, there appears to be no difference
between /dev/sda1 and /dev/xvda: the drive always appears as the
kernel device /dev/xvda.

For the case of nvme drives, the root device typically appears as
/dev/nvme0n1.  Amazon provides the 'ec2-utils' package for their first
party linux ("Amazon Linux"), which configures udev to create symlinks
from the provided name to the nvme device name. This name is
communicated through nvme "Identify Controller" response, which can be
inspected with:

  nvme id-ctrl --raw-binary /dev/nvme0n1 | cut -c3073-3104 | hexdump -C

On Amazon Linux, where the device is attached as "/dev/xvda", this
creates:

- /dev/xvda  -> nvme0n1
- /dev/xvda1 -> nvme0n1p1

On NixOS where the device is attach as "/dev/sda1", this creates:

- /dev/sda1  -> nvme0n1
- /dev/sda11 -> nvme0n1p1

This is odd, but not inherently a problem.

NixOS unconditionally configures grub to install to `/dev/xvda`, which
fails on an instance using nvme storage. With the root device name set
to xvda, both blkfront and nvme drives are accessible as /dev/xvda,
either directly or by symlink.
 2019-11-02 05:58:58 +09:00
AmineChikhaoui dc13a7f26a
ec2-amis.nix: add 19.09 amis 
replace /home/deploy -> $HOME to allow running the script from outside
the bastion.
 2019-10-28 14:04:20 -04:00
Johan Thomsen b263e57c57 scripts/gce: make image name configurable 2019-10-25 10:10:42 +02:00
Andrew Childs 84742e2293 amazon-image.nix: upload prebuilt images 2019-09-05 00:52:21 +09:00
Andrew Childs 25bee972e9 amazon-image.nix: add hydra-build-products and improve metadata 2019-09-05 00:52:20 +09:00
Andrew Childs 027e5560bd amazon-image.nix: default to vpc formatted images 
These can be imported without converison.
 2019-09-05 00:52:20 +09:00
Andrew Childs 5501274b5f amazon-image.nix: add EFI support, enable by default for aarch64 2019-09-05 00:52:17 +09:00
danbst 0f8596ab3f mass replace "flip map -> forEach" 
See `forEach`-introduction commit.
```
rg 'flip map ' --files-with-matches | xargs sed -i 's/flip map /forEach /g'
```
 2019-08-05 14:03:38 +03:00
danbst 91bb646e98 Revert "mass replace "flip map -> foreach"" 
This reverts commit 3b0534310c.
 2019-08-05 14:01:45 +03:00
danbst 3b0534310c mass replace "flip map -> foreach" 
See `foreach`-introduction commit.
```
rg 'flip map ' --files-with-matches | xargs sed -i 's/flip map /foreach /g'
```
 2019-07-14 13:46:10 +03:00
Daniel Schaefer 786f02f7a4 treewide: Remove usage of isNull 
isNull "is deprecated; just write e == null instead" says the Nix manual
 2019-04-29 14:05:50 +02:00