Ankusa.BlobStore.GCS (ankusa v0.7.0)

Copy Markdown View Source

Ankusa.BlobStore backed by Google Cloud Storage's JSON API, via Req. Works against real GCS or the floci-gcp emulator in docker-compose.integration.yml.

This adapter deliberately carries no OAuth2 dependency: :token_provider is a callback you point at whatever your deployment already uses (Goth, ADC), rather than a bundled OAuth2 client. On GCE/GKE the built-in Ankusa.BlobStore.GCS.MetadataServer needs no configuration.

opts:

  • :bucket — required
  • :endpoint — default "https://storage.googleapis.com"; point
                     at `http://localhost:4588` for floci-gcp
  • :token_provider — {module, fun, args}, applied per request, must
                     return `{:ok, bearer_token} | :error`. Omit for the
                 emulator (unauthenticated). **Required against real
                 GCS**: `{Ankusa.BlobStore.GCS.MetadataServer, :token, []}`
                 on GCE/GKE, or your own source (Goth, ADC). On a
                 `401` the adapter calls the module's `invalidate/n`
                 (same arguments) when it exports one and sends the
                 request once more with the fresh token
  • :timeout_ms — default 10_000, for both connect and response of
                     one request that carries at most 1 MiB
  • :upload_timeout_ms, :op_deadline_ms, :multipart_threshold_bytes, :multipart_part_bytes, :breaker_*, :pool_* — one deadline per
                     operation, resumable upload above a size, a breaker
                 per bucket and a connection pool per endpoint. See
                 `Ankusa.BlobStore.Options`
  • :req_options — transport options for the HTTP client, e.g. a custom
                     Finch pool, a proxy, or `plug:` for `Req.Test` in
                 tests. See `Ankusa.HttpClient` for the accepted keys
  • :retries — default 2: extra attempts after a transport error
                     or a `408`/`429`/`5xx`, each with a fresh token from
                 `:token_provider`. See `Ankusa.BlobStore.Retry`
  • :sse — %{type: :kms, key_id: key_resource_name}: every upload
                     names the Cloud KMS key that encrypts the object
                 (`x-goog-encryption-kms-key-name` and `kmsKeyName`).
                 GCS encrypts everything at rest anyway; this picks
                 the key. See `Ankusa.BlobStore.SSE`

put_if_absent/4 is ifGenerationMatch=0 (a 412 is :already_exists); every upload carries x-goog-hash: md5=… and a GET verifies the x-goog-hash md5 it is sent (Ankusa.BlobStore.Checksum); a body over multipart_threshold_bytes is a resumable upload in multipart_part_bytes chunks.

Local dev

config :ankusa,
  storage: %{
    blob_store:
      {Ankusa.BlobStore.GCS, bucket: "ankusa-segments-dev", endpoint: "http://localhost:4588"}
  }

docker compose -f docker-compose.integration.yml up -d floci-gcp gcs-bootstrap brings up the emulator and creates the bucket.