For local development we use LocalStack.cloud. It is utilized as a local emulation of S3 during development. This might be beneficial for testing and development purposes before transitioning to actual S3 service like in a production environment.

​
AWS S3

​
Configuration

AWS S3 requires the following environment variables to be set:
  1. S3_REGION:
  2. S3_LOCAL_STACK (optional): S3 endpoint to connect to, leave blank if your connecting directly to AWS
  3. S3_BUCKET_NAME: Name of the bucket to use
  4. STORAGE_SERVICE: Name of the storage service to use, ‘AWS’ or ‘LOCALSTACK’ for S3.

​
Bucket Configuration

In general, access to the S3 bucket is very permissive. The following process is used to manage access to the S3 bucket. General design:
  • public read access (for logo and email attachments—all data is open for read to anyone)
  • write access is via pre-signed url (client-side PUT to S3 after server-side request)
  • private write access (get, put, delete)
  • objects are written with organisationIdTenancy/objectId (at least for brand logos)
Specifics for access:
  • CORS access on bucket (this is documented)
  • bucket object do not need versioning (as all uploads are deemed unique)
  • ACL on buckets is ‘public-read’ (as part of the pre-signed URLs)
  • image uploads (as pre-signed) could also have rules to limit (DENY policy) on extension types (although UI also does this but leaves open a vector)
  • AWS (specific): ensure that Allow owenership controls is ‘BucketOwnerPreferred’ to allow ACL access (rather than role/user) (ie PutObjectAcl)

​
Example Terraform

/**
 * This sets up an S3 bucket for notifications assets (eg photos, documents). The bucket is
 * PUBLIC for read access and needs to be opened up to the API (which is deployed as an ECS task).
 *
 * It also needs to be opened up to worker (to create pre-signed urls for uploads)
 *
 * This bucket is:
 *  * public read access
 *  * private write access to the credentials (added onto the user rather than onto the bucket)
 */

#
# WARNING: this bucket uses IAM Policies and ACL for access
#
#  - @see https://binaryguy.tech/aws/s3/iam-policies-vs-s3-policies-vs-s3-bucket-acls/#:~:text=The%20biggest%20advantage%20of%20using,well%20as%20objects%20in%20it.
#  - @see https://aws.amazon.com/blogs/security/iam-policies-and-bucket-policies-and-acls-oh-my-controlling-access-to-s3-resources/
#  - @see https://stackoverflow.com/questions/47815526/s3-bucket-policy-vs-access-control-list
#
#
# TF: https://www.terraform.io/docs/providers/aws/r/s3_bucket.html
# AWS: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html
# AWS CLI: http://docs.aws.amazon.com/cli/latest/reference/s3api/create-bucket.html
resource "aws_s3_bucket" "notifications" {
  bucket = "${var.environment}-notifications-data"
  # removed for upgrade to aws provider 4.0
  # see https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#acl-argument
  # acl    = "private"

  tags = {
    Environment = var.environment
    Terraform   = true
  }
}


# TF: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_public_access_block
resource "aws_s3_bucket_public_access_block" "notifications" {
  bucket                  = aws_s3_bucket.notifications.id
  block_public_acls       = false
  # This feature protects your bucket from accidentally getting a policy that would enable public access
  # We WANT public access for read
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}


# TF: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls
# this is required when setting the 'public-read' acl—without it it fails on apply
resource "aws_s3_bucket_ownership_controls" "notifications" {
  bucket = aws_s3_bucket.notifications.id

  rule {
    object_ownership = "BucketOwnerPreferred"
  }
}

# TF: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_acl
resource "aws_s3_bucket_acl" "notifications" {
  bucket = aws_s3_bucket.notifications.id
  # Defaults to private
  # ["private" "public-read" "public-read-write" "authenticated-read" "aws-exec-read" "log-delivery-write"]
  acl    = "public-read"

  depends_on = [
    aws_s3_bucket_ownership_controls.notifications,
    aws_s3_bucket_public_access_block.notifications,
  ]
}

#
# Note: no point on having vesrioning enabled
#

#
# Restricted CORS policy but increased headers exposed (that could be further restrictred)
#
# TF: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_cors_configuration
# AWS: https://docs.amplify.aws/lib/storage/getting-started/q/platform/js/#amazon-s3-bucket-cors-policy-setup
resource "aws_s3_bucket_cors_configuration" "notifications" {
  bucket = aws_s3_bucket.notifications.id

  cors_rule {
    allowed_headers = ["*"]
    allowed_methods = [
      "GET",
      "HEAD",
      "PUT",
    ]
    allowed_origins = ["/* OWN FQDN */"]
    expose_headers  = [
      "x-amz-server-side-encryption",
      "x-amz-request-id",
      "x-amz-id-2",
      "x-amz-meta-tag",
      "ETag"
    ]
    max_age_seconds = 3000
  }
}

resource "aws_s3_bucket_policy" "notifications" {
  bucket = aws_s3_bucket.notifications.id
  policy = data.aws_iam_policy_document.notifications-user.json
}

# [Data] IAM policy to define S3 permissions
#
# Restricting users files types to avoid problems (note: hand helds love to upload PDFs)
#
# TF: https://www.terraform.io/docs/providers/aws/d/iam_policy_document.html
# AWS: http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html
# AWS CLI: http://docs.aws.amazon.com/cli/latest/reference/iam/create-policy.html
data "aws_iam_policy_document" "notifications-user" {
  statement {
    sid    = "DenyNonImage"
    effect = "Deny"
    principals {
      identifiers = [aws_iam_user.notifications.arn]
      type        = "AWS"
    }
    actions = [
      # NOTE: this is an ACL permissions
      "s3:PutObjectAcl",
    ]
    not_resources = [
      "${aws_s3_bucket.notifications.arn}/*.png",
      "${aws_s3_bucket.notifications.arn}/*.jpg",
      "${aws_s3_bucket.notifications.arn}/*.jpeg",
      "${aws_s3_bucket.notifications.arn}/*.gif"
    ]
  }
  # NOTE: these rules exist on the user policy
/*
  statement {
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_user.notifications.arn]
    }
    actions = [
      "s3:GetObject",
      "s3:DeleteObject",
      "s3:ListBucket",
      "s3:PutObject",
      # this is the magic spice, because the perms are created via ACL (rather than roles)
      "s3:PutObjectAcl"
    ]
    resources = [
      "${aws_s3_bucket.notifications.arn}*/
/*",
    ]
  }
*/
}

#######################

# TF: https://www.terraform.io/docs/providers/aws/r/iam_policy.html
# AWS: http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html
# AWS CLI: http://docs.aws.amazon.com/cli/latest/reference/iam/create-policy.html
resource "aws_iam_policy" "s3" {
  name        = "ecs-app-${var.environment}-s3"
  description = "Access from the ecs tasks to s3 buckets"
  policy      = data.aws_iam_policy_document.s3.json
}

# [Data] IAM policy to define S3 permissions
# TF: https://www.terraform.io/docs/providers/aws/d/iam_policy_document.html
# AWS: http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html
# AWS CLI: http://docs.aws.amazon.com/cli/latest/reference/iam/create-policy.html
data "aws_iam_policy_document" "s3" {
  statement {
    sid     = ""
    effect  = "Allow"
    actions = [
      "s3:DeleteObject",
      "s3:GetObject",
      "s3:PutObject",
      # this is the magic spice, because the perms are created via ACL (rather than roles)
      "s3:PutObjectAcl",
      "s3:ListBucket"
    ]
    resources = [
      "arn:aws:s3:::${var.assets_bucket}/*"
    ]
  }
}

######################

# Attaches a managed IAM policy to an IAM role for the ecs tasks
# TF: https://www.terraform.io/docs/providers/aws/r/iam_role_policy_attachment.html
# AWS: http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html
# AWS CLI: http://docs.aws.amazon.com/cli/latest/reference/iam/attach-role-policy.html
resource "aws_iam_role_policy_attachment" "s3" {
  role       = aws_iam_role.app_task.name  # need your role attached to whatever is doing the execution (eg workload task)
  policy_arn = aws_iam_policy.s3.arn
}

​
Azure Blob Storage Configuration

​
Configuration

Azure Blob Storage requires the following environment variables to be set:

​
Required

  1. STORAGE_SERVICE: Name of the storage service to use, ‘AZURE’ Azure Blob.
  2. AZURE_CONTAINER_NAME: The name of the container in your Azure Storage account that you wish to use for your blob storage.
  3. AZURE_ACCOUNT_NAME: The name of your Azure Storage account. This is used to form the URL at which your blob storage is accessible.
  4. AZURE_ACCOUNT_KEY: The access key for your Azure Storage account. This is used to authenticate requests made against your blob storage.
  5. AZURE_HOST_NAME: The host name of your Azure Storage account . This is used to form the URL at which your blob storage is accessible. Ref.

​
Bucket Configuration

TBD

​
Google Cloud Storage Configuration

​
Configuration

Google Cloud Storage requires the following environment variables to be set:
  1. GCS_BUCKET_NAME: Name of the bucket to use.
  2. GOOGLE_APPLICATION_CREDENTIALS: Path to the service account key file.
  3. STORAGE_SERVICE: Name of the storage service to use, ‘GCS’ for Google Cloud Storage.

​
Bucket Configuration

TBD