Object data-plane performance

The primary workload is 10 MiB–1 GiB Parquet objects. The layout and I/O paths below make PostgreSQL practical as the byte store for native buckets, shared read fills and durable origin staging. The page-oriented layout is also used by pgvfs.

design.md defines ownership and acknowledgement; this page explains the mechanisms, not a durability or capacity benchmark. Use paired acceptance for per-path evidence and the data-plane proof/capacity ledger for replica, recovery and resource measurements. The roadmap keeps the remaining deployment-specific limits explicit.

Speed of light

just sol measures one client stream through the gateway against the floors that bound it on the same host. The HTTP floor is a loopback server that only moves bytes. The PostgreSQL floor is a direct COPY (write) or 8-connection SELECT (read) of the identical 8,120-byte rows in a table with the same storage settings, using the gateway’s chunk query and chunk-pool planner settings, assembled into 256 KiB chunks (the minimum any implementation must do). A perfectly pipelined gateway would approach min(HTTP, PostgreSQL). Payloads are incompressible; caches are warm; the gateway-local read cache is off; each cell is the median of five interleaved trials.

Size Op pgvs3 MiB/s HTTP floor PG floor (1 conn) % of floor gateway CPU ns/B
10 MiB PUT 188 3,455 330 57% 2.97
10 MiB GET 2,168 2,444 4,986 (1,009) 89% 0.56
10 MiB 8 MiB range 1,982 3,717 5,121 (1,187) 53% 0.60
256 MiB PUT 384 3,555 461 83% 2.31
256 MiB GET 3,311 4,011 5,767 (1,227) 83% 0.53
256 MiB 8 MiB range 1,476 3,145 2,556 (873) 58% 0.76
1 GiB PUT 309 4,063 406 76% 2.98
1 GiB GET 2,077 3,086 2,468 (811) 84% 0.74
1 GiB 8 MiB range 984 2,978 1,777 (755) 55% 0.94

Intel Core Ultra X7 358H laptop (16 hybrid cores, no SMT), NVMe, PostgreSQL 18 in Docker with shared_buffers=1GB, max_wal_size=4GB, gateway default pools. Report: .tmp/verify/sol.LTDw0T/sol.json, SHA-256 61fc82be48534a0d463e047ed8bdb3e0e78f922e59cb1687fd283706b4820534. Single runs on this host move by about ±15%; the loopback HTTP floor alone ranged from 2.4 to 3.2 GiB/s for 10 MiB between runs. The 1 GiB rows exceed shared_buffers, so both pgvs3 and the floor read through the operating-system page cache there.

Until 2026-11-02 the PostgreSQL read floor used ORDER BY no on an un-analyzed table. That planned an external merge sort spilling to disk and understated the floor about 2.5× (one connection ~0.45 GiB/s instead of ~0.8–1.3; eight ~2.5 instead of 3.4–5.8). Earlier “% of floor” figures, including those above 100%, compare against that understated floor.

What it says:

  • Writes are at the floor. PUT is 57–83% of a raw PostgreSQL COPY here (77–96% in the earlier desktop run). Faster single-stream writes need a faster PostgreSQL write path (WAL, disk), not gateway work.
  • Reads are bound by HTTP, not PostgreSQL. Eight connections deliver 2.5–5.8 GiB/s and one ~0.8–1.2 GiB/s, so the loopback HTTP floor binds every read. Whole-object GET reaches 2.1–3.3 GiB/s (83–89% of floor).
  • Ranges pay fixed setup. An 8 MiB range reaches 1.0–2.0 GiB/s (53–58%). Metadata, the read guard and each worker’s first rows cost about 1–1.5 ms before the first byte on this host, a quarter to a third of the request.
  • Under concurrency the ceiling is CPU per byte. With 8 clients, uncached reads plateau near 4.3 GiB/s here with either read guard. The gateway spends ~0.8–0.9 ns/byte, mostly kernel socket reads from PostgreSQL and writes to clients. Only the gateway read cache changes that.

Run just sol (defaults -sizes 10MiB,256MiB,1GiB -repeat 5 -range 8MiB -readers 8) on your hardware before quoting any of these numbers.

Read-path changes, paired

The paired gate against the previous binary (snapshot-per-piece window, unbuffered chunk connections) passes every case: native 10 MiB GET 11.7% faster at one worker and 15.8% at four; 100 MiB GET 16.1% and 11.4% faster; HEAD, small, range and PUT cases within ±13%; the cold origin profile within ±9%. Reports: .tmp/verify/load.n64lvy and .tmp/verify/load.F7vjJz, both complete: true with no failed gates.

A direct single-client loop on the same host (median of three interleaved rounds, MiB/s):

Read Previous Current Current, warm read cache
10 MiB GET 1,986 2,132 4,900
8 MiB range of 10 MiB 1,603 1,917 4,163
64 MiB GET 1,844 2,266 5,058
8 MiB range of 256 MiB 1,652 1,644 4,829
256 MiB GET 1,868 2,612 5,508

With 8 concurrent clients the previous and current paths are equal within noise (10 MiB GET ~4.4 GiB/s, 64 MiB ~4.3 GiB/s, 1 MiB ranges 2.7 → 3.0 GiB/s); a warm read cache serves 15.7, 15.1 and 9.5 GiB/s respectively at ~0.3 ns of gateway CPU per byte and no PostgreSQL work.

FIG 03 — 8,192-Byte PostgreSQL Page, 8,120-Byte Chunks & Cut-Through Streaming I/O

FIG 03 — 8,192-Byte PostgreSQL Page, 8,120-Byte Chunks & Cut-Through Streaming I/O

Storage layout

Mechanism How it works
Namespace-local relations s3b_<name> (native bucket), s3c_<name> (shared read cache/RESP values), s3w_<name> (write-back) and s3q_<name> (jobs/streams) have their own tables and settings. They share cluster I/O, WAL and autovacuum capacity, not dedicated workers per namespace. All are WAL-logged.
Inline small objects Native/write-cache objects can inline up to 64 KiB and read-cache/RESP values up to 8,120 bytes in objects.inline_data with file_id NULL. Zstd is used only when it shrinks the body. A primary-key lookup avoids chunks; multipart/pending paths can retain chunked body/outbox publication even when small.
PostgreSQL 18 publication Inline/chunked entry points share publish_value; RETURNING old.file_id, old.parts supplies overwritten-file ownership without a pre-read on unconditional writes. Preconditions, TTL, metadata, lease fencing and garbage publication remain transactional.
Shared range parent metadata A logged schema-local origin_metadata record caches size/validator/headers with validation TTL but never pretends to hold an object body. Write-back/read-cache shadowing resolves in one SQL round trip; read-only workers can use replicated metadata and cached range blocks.
Length-discriminated Zstd 8,120-byte chunk blocks Larger objects split into logical 8,120-byte blocks in chunks (the block-encoding section of internal/store/object.go). Each block is compressed with a reusable Zstd frame if it shrinks (len(stored) < wantLen) or stored raw (len(stored) == wantLen), so incompressible blocks never exceed the 8,160-byte page budget and logical offsets remain no × 8,120 for O(1) byte-range seeking.
Admission control Protocol requests/batches and background fill/upload classes have bounded concurrency. Disk spools share a process-wide byte budget. These are resource controls, not independent PostgreSQL capacity reservations; size the whole fleet’s pools and monitor waits.
No TOAST overhead chunks sets toast_tuple_target = 8160 and STORAGE EXTERNAL, so PostgreSQL never detoasts or chases out-of-line pointers on read.
Immutable files Chunked writes allocate fresh file_ids from s3p.file_ids; overwrites swap object ownership and queue retired files for reaping. Chunk rows are never updated in place.
HTTP validators Ordinary native ETags are content MD5s and multipart ETags are composite values, not file_ids. Origin versions are retained separately for revalidation/conditions. Clients can reuse cached slices using the public validator.
Split connection pools The main pool keeps enable_indexscan = on for fast B-tree lookups on objects and ListObjectsV2; a dedicated chunkPool sets enable_indexscan = off so chunks range reads use PostgreSQL 18 bitmap heap scans with read-ahead (effective_io_concurrency = 32), and reads its sockets through a 256 KiB buffer.
Unpartitioned chunks table per bucket Each bucket’s chunks is a single table (capped at 32 TiB / ~31.7 TiB of payload). Partitioning within a bucket by hash or range was slower on point reads and writes; see partitioning.md.
Aggressive autovacuum Tuned to autovacuum_vacuum_scale_factor = 0.01 and analyze_scale_factor = 0.02 so freed pages are recycled quickly after whole-file deletes.

Read path

  1. Inline fast path: metadata and inline_data come from one primary-key lookup on objects, within the namespace’s inline limit. HEAD queries only header columns and never transfers inline_data.
  2. Small single-query reads: spans up to 1 MiB, or any plan that gets one reader, run as one chunks range query (file_id = $1 AND no BETWEEN $2 AND $3), consistent within its statement.
  3. Ordered pull window: larger spans plan balanced pieces (1 MiB minimum, ~8 MiB for large objects). Up to eight workers pull pieces in order from a shared counter and stream each into its own channel; the consumer drains the channels in piece order, so output stays contiguous and memory is bounded by the workers’ buffers. The window reserves half the chunk pool for other work and shrinks as reads overlap.
  4. Read pins, with a snapshot fallback: before workers start, one guard connection takes shared advisory locks (pg_try_advisory_lock_shared) on every file in the span. The reaper takes the exclusive lock before deleting a file and skips pinned ones, so workers run plain autocommit queries and a pin holds no MVCC snapshot back from vacuum. On a standby (pg_is_in_recovery()), for spans over 64 files, or when a pin finds the reaper deleting that file, the read exports one REPEATABLE READ snapshot that each worker imports once. The exporting transaction ends when every worker has imported it, and worker transactions lift idle_in_transaction_session_timeout locally, so a slow client cannot get a stream cut off mid-object.
  5. Synchronous vs cut-through streaming: Reads up to 8 MiB stream synchronously on the HTTP goroutine. Above 8 MiB, a background fetcher feeds a bounded 32-buffer channel while Body.WriteTo streams 256 KiB buffers to the socket, overlapping database I/O with network writes and backpressuring slow clients.
  6. Buffered chunk connections: chunk-pool connections read the socket through a 256 KiB buffer under pgx’s 8 KiB frame reader, so one read system call carries ~32 rows instead of one. This is what lifted large GETs.
  7. TID-to-row-number reordering: Bitmap heap scans return pages in physical TID order; streamRows buffers any out-of-order rows in memory and emits strict no order.
  8. Gateway read cache (opt-in): see below.
  9. Warm cache hit: the cache tier reuses the metadata row it already read for the TTL/lifecycle decision when opening the body, so a warm cache GET does not repeat the primary-key lookup. Inline cache objects are served directly from that row; larger ones start the normal chunk stream.

Gateway read cache

--read-cache-bytes (default 0, off) gives each gateway a bounded in-memory cache of decoded chunk blocks (32 rows, 253.75 KiB). It changes where hot bytes come from, not who owns them:

  • No authority. Every request still reads metadata from PostgreSQL first; only bytes of the file that metadata currently references are served. Keys are the namespace objects table OID plus file_id and block number. File IDs are never reused within a table and the OID changes on drop/restore, so a key never names different content and nothing needs invalidating.
  • Fills while streaming. Reads aligned to whole blocks store blocks as they pass through. A fully cached span needs no chunk connection, pin or snapshot; partly cached spans serve cached pieces in order and query the rest.
  • Scan resistant. Each shard runs S3-FIFO: new blocks enter a small queue and only blocks read again move to the main queue, so one large scan cannot flush hot footers and row groups.
  • Per gateway. Replicas do not share it; PostgreSQL remains the shared cache across gateways and standbys. Count the budget inside GOMEMLIMIT.

Measured above: warm whole-object reads 2.3–2.6× faster for one client and about 3.5× higher aggregate for eight, with gateway CPU per byte falling from ~0.9 to ~0.3 ns. It helps repeated reads of the same objects (Parquet footers, popular row groups, shared lake files) and does nothing for one-pass scans. Watch pgvs3_read_cache_hit_bytes_total against total read volume before sizing it up.

Write path

  1. Inline writes (≤64 KiB): s3b_<bucket>.put_inline takes a per-key transaction advisory lock, checks If-Match / If-None-Match, queues any replaced file_id for GC, and upserts the row in one round trip. The bucket must already exist in the namespace catalog; a missing or deleted bucket returns NoSuchBucket and is never recreated implicitly. The HTTP handler reuses a bounded 64 KiB request buffer and releases it after the SQL call, so a stream of small PUTs does not allocate a fresh body for every request.
  2. Streaming binary COPY (>64 KiB): internal/store/write.go hashes the request body (MD5 + SHA-256/CRC) while framing 4 MiB binary COPY batches from a sync.Pool into s3b_<bucket>.chunks. Full rows are framed straight from the incoming read buffer; only the trailing partial row is retained.
  3. Atomic publish: In the same transaction as COPY, s3b_<bucket>.publish_object evaluates conditional headers, updates objects, and queues any replaced file_id. If a client disconnects mid-upload, the transaction rolls back and publishes nothing.
  4. Zero-copy multipart, CopyObject, and bucket clones: Multipart parts upload in parallel as independent file_ids; CompleteMultipartUpload validates checksums and records the ordered file_id array without moving bytes. Same-bucket CopyObject and whole-file UploadPartCopy share the source’s immutable file_ids; cross-bucket CopyObject and s3p.clone_namespace copy chunk rows inside PostgreSQL without streaming bytes through the gateway.
  5. Native batch delete: DeleteObjects locks and deletes up to 1,000 keys and queues unreferenced files in two SQL statements instead of 1,000 round trips.

Origin-backed I/O

  • Read fills: a whole-object miss uses one origin GET carrying size/version metadata. Bodies spool to memory (up to 8 MiB) or scratch disk before opening the PostgreSQL body writer. Renewable database leases coalesce cross-gateway fills; publication checks the live owner and expected origin version. Waiters can fall back to origin pass-through after their bounded wait.
  • Warm and ranged reads: within TTL, a hit needs no origin request. Warm GET reuses its already-read metadata. Cold partial reads of objects larger than 8 MiB pass through unless aligned range caching is enabled; logged parent metadata avoids repeated origin HEADs. Read-only gateways can consume replayed metadata/blocks but never fill them.
  • Synchronous mutations: small/conditional PUT/COPY and deletes require origin work before success. Origin mutation and cache publication remain separate commits. Foreground mutations and outbox uploads deliberately retain a PostgreSQL session key lock across origin I/O; read-fill downloads do not.
  • Write-back: by default, unconditional PUT/COPY above 8 MiB and unconditional multipart completion at any size commit logged s3w_ bytes plus an outbox row before acknowledging. Pending bodies have no expiry. Measure acknowledgement latency and backlog drain; early acknowledgement is not upload throughput or a lossless-promotion promise independent of PostgreSQL policy.
  • Listings: merge an origin walk with pending objects, retaining one bounded page in memory. There is no separate listing index or refresh generation.

Read-cache fills are writes to PostgreSQL: they consume disk, WAL and replica replay bandwidth even when client traffic is read-only. Rebuildable origin fills and standalone RESP values share s3c_, but only the former are disposable. See operations.md for provider limits, TTLs, tombstones, retries and spool sizing.

Multi-gateway consistency and connection pooling

  • Stateless gateways: Gateways hold no authoritative in-process object, metadata, namespace, file-ID, or lease state. The namespace catalog in PostgreSQL decides what exists. Gateways sharing a primary read the same committed rows under MVCC, without a gateway invalidation bus. Each PostgreSQL instance has its own shared_buffers; standbys share the logged data through WAL replay, subject to replication lag, not coherent distributed RAM. The optional read cache holds only immutable chunk bytes and is never consulted without fresh metadata, so it needs no invalidation.
  • One maintenance leader: a single writable gateway holds the maintenance advisory lock; the others skip the tick. If the leader exits, another elects itself on a later tick.
  • Conditional cache publishes: read-through fills are fenced by the expected origin version and live fill owner; outbox flushes carry a lease plus a token guard. A stale fill or flush is discarded instead of overwriting a newer write-through row. Write-through deletes leave an expiring tombstone so a fill that started before the delete cannot reinsert the object.
  • MRU connection reuse: pgxpool reuses the most-recently-used idle connection first, keeping TCP connections warm and avoiding slow-start penalties. Connections recycle after a finite lifetime with jitter.
  • Once-per-connection setup: Each connection applies a configurable work_mem (default 16MB), effective_io_concurrency = 32, TCP keepalives, statement/lock/idle-transaction timeouts, and synchronous_commit = on. The chunk pool deliberately disables the statement timeout for long streaming reads; upload transactions disable it around COPY.

Reaping and vacuum

  • Read grace window: Deleted or overwritten files remain in their namespace’s chunks table for --reap-grace (default 10 minutes), allowing metadata/body lookups to bridge a concurrent overwrite. Multi-piece reads additionally pin their files (or hold an imported snapshot on a standby) until they finish, so a read that outlives the grace window still completes. The reaper skips pinned files and counts them in pgvs3_reap_pinned_skips_total.
  • Bounded, chained GC passes: Every minute, one writable gateway acquires a database-scoped pg_try_advisory_lock and deletes up to 65,536 chunk rows (~508 MiB) per pass; a saturated pass immediately chains another. If the leader exits, another writable gateway acquires the lock on a later tick.
  • Bounded TTL expiry: expire_objects(4096) repeats until a batch is short or the per-tick round budget is reached; expiry is not an unbounded full-table sweep.
  • Space reuse: Freed pages are reused by subsequent writes once autovacuum makes dead-tuple space reusable. Long-running snapshots hold back vacuum horizon; avoid multi-hour open transactions on the storage database.

Tuning DuckDB / DuckLake clients

File layout, projection and client caches can dominate analytics speed. Use the executable extraction/configuration recipe and just bench tuning, not a universal 8,192-row/64 MB preset. Small row groups can help selective pruning but hurt scan/compression/request overhead; file targets are not exact physical sizes. Persisted catalog options affect future writes, and tiny DuckLake inserts can be inlined in the catalog rather than creating objects. The guide tests both configuration syntax and inlining/flush behavior; it does not declare that candidate the fastest layout.

DuckDB 2.0 Parquet uses async I/O. Inspect async_threads, read-ahead and HTTP pool capacity as well as execution threads, and budget aggregate client pressure against gateway/cluster limits. Its remote-file and Parquet metadata caches are client-private, not pgvs3 shared PostgreSQL rows. Disable both when attributing shared hits; use fresh processes for a genuinely fresh native-client cache. See measurement labels.

Companion paths

RESP2 pipelines coalesce eligible GET/SET batches into Cache.MGet/Cache.MSet against logged s3c_ values. OJS jobs use database-clock leases and SKIP LOCKED claims without per-queue subtransactions. Streams use snapshot-delimited batches and consumer-pinned rotation of three event partitions; retry/dead-letter state has its own lifecycle. See queue-stream.md.

These paths retain their targeted tests and benchmarks, but are not a reason to reshape the specialized object COPY/streaming loops into a universal data model.