Module streaming 

Streaming operation types and concurrent executor.

StreamExecutor processes a stream of (idx, Result<Operation, E>) tuples concurrently within a bounded window. Errors in the input stream pass through unchanged; successful operations are executed against TieredStorage directly, with [tokio::spawn] for panic isolation and run-to-completion guarantees.

Window and Permit Reservation

The concurrency window is derived from the service’s available permits at the time StorageService::stream is called: ceil(tasks_available × 0.10). The executor pre-acquires exactly window permits from the service’s ConcurrencyLimiter as a single bulk reservation. The reservation is shared (via Arc) with every spawned task, so permits are released only after every in-flight task completes — even if the output stream is dropped early.

This means:

• If the service is at capacity, StorageService::stream fails immediately with Error::AtCapacity before any operations are read.
• During execution, operations call the storage backend directly without acquiring additional per-operation permits.

Concurrency Model

StreamExecutor::execute uses buffer_unordered to drive up to window operations concurrently. The input stream is pulled lazily — at most window operations are in-flight at once, bounding memory to roughly window × max_operation_size. Results are yielded in completion order.

Each operation is wrapped in a [tokio::spawn] for panic isolation: a panic in one operation surfaces as Error::Panic for that item and does not affect the others.

Future Scope

The window fraction (10%) is hard-coded. Configurable fractions, adaptive window sizing, and backend-level optimizations (e.g. BigTable multi-read, GCS batch API) are out of scope for the current implementation.

Structs

Delete

A delete operation: removes an object by key.

Get

A get operation: retrieves an existing object by key.

Insert

An insert operation: stores an object at the given key.

StreamExecutor

Executes streaming operations with bounded concurrency.

Enums

OpResponse

The response of a single executed streaming operation.

Operation

A single streaming operation.