Storage: S3, EBS & EFS

The object model

Every S3 object has a key (e.g. uploads/2024/invoice.pdf), a value (0 bytes to 5 TB), and metadata (system + user-defined). Buckets are globally unique names in the s3.amazonaws.com namespace — choose names carefully; you cannot rename a bucket without creating a new one and copying objects.

Storage classes — comparison

Storage class determines durability, availability, minimum storage duration, retrieval fees, and access latency. Match the class to access pattern — not every object belongs in Standard.

Lifecycle rules

Lifecycle configurations automatically transition objects between storage classes or expire (delete) them based on age, prefix, tags, or current storage class. Rules run once per day — not real-time.

• Transition — move to IA after 30 days, Glacier after 90 days, Deep Archive after 365 days
• Expiration — delete objects or noncurrent versions after N days
• Abort incomplete multipart uploads — after 7 days to stop paying for orphaned parts
• Filter — apply rules to prefix (logs/) or object tags

Versioning

With versioning enabled, every PUT creates a new version; DELETE adds a delete marker (doesn't remove data). Restore by deleting the delete marker or copying a previous version. Pair with lifecycle rules to expire noncurrent versions after N days — otherwise storage grows forever.

Replication — CRR and SRR

S3 Replication copies objects from a source bucket to a destination bucket automatically. Requires versioning on both buckets. IAM role must allow replication actions.

CRR copies across regions (DR, compliance); SRR within the same region (log aggregation). Both require versioning on source and destination. RTC adds a 15-minute replication SLA.

Production bucket — encryption + lifecycle

aws s3api create-bucket --bucket my-app-artifacts-prod-eu \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1

aws s3api put-bucket-versioning --bucket my-app-artifacts-prod-eu \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-encryption --bucket my-app-artifacts-prod-eu \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "alias/my-app-s3-key"
      },
      "BucketKeyEnabled": true
    }]
  }'

aws s3api put-public-access-block --bucket my-app-artifacts-prod-eu \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

cat > /tmp/lifecycle.json <<'EOF'
{
  "Rules": [{
    "ID": "archive-and-expire",
    "Status": "Enabled",
    "Filter": { "Prefix": "uploads/" },
    "Transitions": [
      { "Days": 30, "StorageClass": "STANDARD_IA" },
      { "Days": 90, "StorageClass": "GLACIER" }
    ],
    "NoncurrentVersionExpiration": { "NoncurrentDays": 30 },
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
  }]
}
EOF

aws s3api put-bucket-lifecycle-configuration --bucket my-app-artifacts-prod-eu \
  --lifecycle-configuration file:///tmp/lifecycle.json

resource "aws_s3_bucket" "artifacts" {
  bucket = "my-app-artifacts-prod-eu"
}

resource "aws_s3_bucket_versioning" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.s3.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "artifacts" {
  bucket                  = aws_s3_bucket.artifacts.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_lifecycle_configuration" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
  rule {
    id     = "archive-and-expire"
    status = "Enabled"
    filter { prefix = "uploads/" }
    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }
    transition {
      days          = 90
      storage_class = "GLACIER"
    }
    noncurrent_version_expiration { noncurrent_days = 30 }
    abort_incomplete_multipart_upload { days_after_initiation = 7 }
  }
}

import * as s3 from 'aws-cdk-lib/aws-s3';
import * as kms from 'aws-cdk-lib/aws-kms';
import { Duration, RemovalPolicy } from 'aws-cdk-lib';

const key = kms.Key.fromLookup(this, 'S3Key', { aliasName: 'alias/my-app-s3-key' });

new s3.Bucket(this, 'ArtifactsBucket', {
  bucketName: 'my-app-artifacts-prod-eu',
  versioned: true,
  encryption: s3.BucketEncryption.KMS,
  encryptionKey: key,
  bucketKeyEnabled: true,
  blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
  enforceSSL: true,
  lifecycleRules: [{
    id: 'archive-and-expire',
    prefix: 'uploads/',
    transitions: [
      { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: Duration.days(30) },
      { storageClass: s3.StorageClass.GLACIER, transitionAfter: Duration.days(90) },
    ],
    noncurrentVersionExpiration: Duration.days(30),
    abortIncompleteMultipartUploadAfter: Duration.days(7),
  }],
  removalPolicy: RemovalPolicy.RETAIN,
});

Block Public Access

Four account-level and bucket-level settings that override any policy making a bucket public. Enable at the account level in Organizations — defense in depth with SCPs and bucket policies. Even with Block Public Access on, a misconfigured bucket policy can still grant overly broad access to authenticated AWS principals — BPA only blocks anonymous/public access.

Bucket policies vs IAM policies

S3 bucket policies are resource-based — they attach to the bucket and can grant cross-account access without an IAM policy on the other side (though the other principal often still needs permission to call S3). Use bucket policies for: CloudFront OAC, cross-account read, denying unencrypted uploads, requiring VPC endpoint access.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyUnencryptedUploads",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::my-app-artifacts-prod-eu/*",
    "Condition": {
      "StringNotEquals": {
        "s3:x-amz-server-side-encryption": "aws:kms"
      }
    }
  }]
}

Presigned URLs and POST

Presigned URLs grant temporary access to a specific object (GET or PUT) without making the bucket public. Generated by signing with IAM credentials — expiry from seconds to 7 days (SigV4). Presigned POST allows browser direct upload via HTML form — common for user file uploads from a Spring Boot app without proxying bytes through your API.

Generate a presigned download URL

# Presigned GET — share invoice download for 15 minutes
aws s3 presign s3://my-app-artifacts-prod-eu/invoices/2024/inv-001.pdf \
  --expires-in 900 \
  --region eu-west-1

# Presigned PUT — client uploads directly (Spring returns URL to browser)
aws s3 presign s3://my-app-artifacts-prod-eu/uploads/user-42/doc.pdf \
  --expires-in 3600 \
  --region eu-west-1 \
  --http-method PUT

# Verify the URL works
curl -I "$(aws s3 presign s3://my-app-artifacts-prod-eu/invoices/2024/inv-001.pdf --expires-in 300)"

# Presign at runtime — grant s3:GetObject on the prefix to the app role.
resource "aws_iam_role_policy" "presign" {
  role = aws_iam_role.api.id
  policy = jsonencode({
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:GetObject"]
      Resource = "${aws_s3_bucket.artifacts.arn}/invoices/*"
    }]
  })
}

taskRole.addToPolicy(new iam.PolicyStatement({
  actions: ['s3:GetObject'],
  resources: [`${bucket.bucketArn}/invoices/*`],
}));
// Runtime: S3Presigner.create().presignGetObject(...) — 15 min expiry

Encryption — SSE-S3, SSE-KMS, SSE-C

Prefix strategy and request rate

S3 scales automatically, but extremely hot prefixes (millions of PUTs/sec to one prefix) can throttle. For high-throughput workloads, use hex hash prefixes: uploads/a3/f9/object-id instead of uploads/2024/06/10/object-id when date clustering creates hotspots. For most backend apps, date-based prefixes are fine.

Multipart upload

Required for objects > 5 GB; recommended for > 100 MB. Upload parts in parallel (5 MB–5 GB each, max 10,000 parts). Failed uploads leave parts that cost money — lifecycle rule to abort incomplete uploads after 7 days is mandatory. Complete with CompleteMultipartUpload; list in-progress with ListMultipartUploads.

S3 Select and events

• S3 Select — SQL on CSV/JSON/Parquet in-place; filter without full download
• Event notifications — Lambda, SQS, SNS, EventBridge on object create/delete (at-least-once)

Object Lock (WORM)

Retention or legal hold prevents deletion — Governance (admins can override) vs Compliance (nobody deletes until expiry). Requires versioning; bucket created with Object Lock enabled.

Block storage fundamentals

EBS volumes live in a single Availability Zone. An EC2 instance must be in the same AZ to attach. You can detach and reattach to another instance in the same AZ (stop instance first for root volumes on Nitro). Size and type can be modified online for most volume types. Max 128 volumes per instance (quota increaseable).

Volume types

Snapshots and encryption

EBS snapshots are incremental backups in S3 (AWS-managed). Copy cross-region for DR; use DLM for automated schedules. Enable account-level EBS encryption by default — all new volumes use KMS; no performance penalty on Nitro.

Multi-Attach (io2 only)

One volume on up to 16 instances in the same AZ — requires cluster-aware FS (Oracle RAC, GFS2). Standard ext4/xfs without clustering will corrupt. Use EFS for shared POSIX without cluster software.

EBS vs instance store

Create a gp3 volume and attach

aws ec2 create-volume \
  --availability-zone eu-west-1a \
  --size 100 \
  --volume-type gp3 \
  --iops 3000 \
  --throughput 125 \
  --encrypted \
  --kms-key-id alias/my-app-ebs-key \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=order-db-data}]'

# Attach to running instance (data volume — not root)
aws ec2 attach-volume \
  --volume-id vol-0abc123def456 \
  --instance-id i-0fedcba987654 \
  --device /dev/sdf

# On the instance: mkfs, mount, fstab
# sudo mkfs -t xfs /dev/nvme1n1
# sudo mount /dev/nvme1n1 /data

resource "aws_ebs_volume" "order_db_data" {
  availability_zone = "eu-west-1a"
  size              = 100
  type              = "gp3"
  iops              = 3000
  throughput        = 125
  encrypted         = true
  kms_key_id        = aws_kms_key.ebs.arn

  tags = { Name = "order-db-data" }
}

resource "aws_volume_attachment" "order_db" {
  device_name = "/dev/sdf"
  volume_id   = aws_ebs_volume.order_db_data.id
  instance_id = aws_instance.order_db.id
}

new ec2.Volume(this, 'OrderDbData', {
  availabilityZone: 'eu-west-1a',
  size: cdk.Size.gibibytes(100),
  volumeType: ec2.EbsDeviceVolumeType.GP3,
  iops: 3000, throughput: 125,
  encrypted: true, encryptionKey: key,
});

NFS fundamentals

Mount EFS with standard NFS clients (mount -t nfs4 or EFS mount helper). Access via security groups (port 2049) and EFS access points for per-application POSIX identities. Works across all AZs in a region — unlike EBS, no AZ lock-in.

Performance modes

Throughput modes

• Bursting — throughput scales with filesystem size; burst credits for spikes (default, cost-effective for small FS)
• Provisioned — set throughput independent of size; for consistent high throughput on small datasets
• Elastic (recommended) — automatically scales throughput up/down; pay for peak usage; replaces bursting for most new workloads

EFS storage classes

Standard for active files; EFS-IA moves files not accessed for 30 days (lifecycle policy). One Zone variants for dev/test — lower cost, no cross-AZ redundancy.

Selection guide — EFS vs EBS vs S3

Backup strategy — 3-2-1 on AWS

Three copies (prod + snapshot + cross-region), two media types (EBS block + S3 object), one offsite (CRR or AWS Backup vault copy).

Cross-region DR with S3

Primary bucket with versioning + CRR to a DR region; lifecycle moves DR copies to Standard-IA. On regional failure, point config/DNS at the DR bucket. Enable RTC for RPO < 15 minutes.

Spring Boot static assets on S3 + CloudFront

Don't serve static JS/CSS from your JVM — offload to S3 behind CloudFront with Origin Access Control (OAC). CI uploads hashed assets; Spring templates reference CDN URLs in prod.

• CI runs aws s3 sync with long cache on hashed assets, short cache on index.html
• CloudFront OAC → private S3 origin; ACM cert on custom domain
• Spring Boot serves API only in prod; templates reference CDN URLs, not classpath static

# CI deploy after ./mvnw package
aws s3 sync target/classes/static/ s3://my-app-static-prod/assets/ \
  --cache-control "public, max-age=31536000, immutable" --exclude "index.html"
aws s3 cp target/classes/static/index.html s3://my-app-static-prod/index.html \
  --cache-control "public, max-age=60"
aws cloudfront create-invalidation --distribution-id E1234567890ABC \
  --paths "/index.html" "/assets/*"

Operational checklist

• Account-level Block Public Access + SCP deny on public bucket policies; S3 Access Analyzer weekly
• Versioning on prod buckets; lifecycle expires noncurrent versions; abort incomplete MPU after 7 days
• CRR for critical buckets; EBS snapshot cross-region copy via DLM or AWS Backup
• EBS encryption by default; S3 SSE-KMS with Bucket Key; org-wide Storage Lens for cost anomalies