Skip to main content

TieredStorage

Struct TieredStorage 

Source
pub struct TieredStorage { /* private fields */ }
Expand description

Two-tier storage backend that routes objects by size.

TieredStorage implements Backend and is intended to be used inside a StorageService, which wraps it with task spawning and panic isolation.

§Size-Based Routing

Objects are routed at write time based on their size relative to a 1 MiB threshold:

  • Objects ≤ 1 MiB go to the high_volume backend — optimized for low-latency reads and writes of small objects (e.g. BigTable).
  • Objects > 1 MiB go to the long_term backend — optimized for cost-efficient storage of large objects (e.g. GCS).
  • Resumable uploads at or below 1 MiB are declined; larger uploads go to long_term.

§Redirect Tombstones

Because the ObjectId is backend-independent, reads must be able to find an object without knowing which backend stores it. A naive approach would check the long-term backend on every read miss in the high-volume backend — but that is slow and expensive.

Instead, when an object is stored in the long-term backend, a redirect tombstone is written in the high-volume backend. It acts as a signpost: “the real data lives in the other backend at this target.” On reads, a single high-volume lookup either returns the object directly or follows the tombstone to long-term storage, without probing both backends.

How tombstones are physically stored is determined by the HighVolumeBackend implementation — refer to the backend’s own documentation for storage format details.

§Consistency

Consistency across the two backends is maintained through compare-and-swap operations on the high-volume backend (see HighVolumeBackend::compare_and_write), not distributed locks. Each mutating operation reads the current high-volume revision, performs its work, and then atomically swaps the high-volume entry only if the revision is still current — rolling back on conflict. Cleanup of unreferenced LT blobs runs in background tasks so the caller returns as soon as the commit point is reached. Call Backend::join during shutdown to wait for outstanding cleanup.

See the module-level documentation for per-operation diagrams.

§Concurrency

Optional per-tier limits keep stalls in one backend from using all capacity.

High-volume work that continues an operation after long-term storage access has high priority; other work has normal priority. The high-volume concurrency is shared between these two priority classes. This lets operations already in progress finish first. Permits cover calls to their own backend, not waits for the other tier. Long-term downloads retain their permit while the response stream is active.

§Usage

TieredStorage handles routing, consistency, and optional per-tier limits. Wrap it in a StorageService to add task spawning, panic isolation, and the total service concurrency limit.

Implementations§

Source§

impl TieredStorage

Source

pub fn new( high_volume: Box<dyn HighVolumeBackend>, long_term: Box<dyn Backend>, changelog: Box<dyn ChangeLog>, ) -> Self

Creates a new TieredStorage with the given backends and change log.

Source

pub fn with_concurrency(self, config: TieredConcurrencyConfig) -> Self

Enables per-tier concurrency limits.

Trait Implementations§

Source§

impl Backend for TieredStorage

Source§

fn name(&self) -> &'static str

The backend name, used for diagnostics.
Source§

fn upload_granularity(&self) -> u64

Returns the upload granularity for sessions opened by this backend, in bytes. Read more
Source§

fn create_upload_session<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, id: &'life1 ObjectId, metadata: &'life2 Metadata, upload_length: NonZeroU64, ) -> Pin<Box<dyn Future<Output = Result<Option<BackendToken>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Creates a resumable upload session for the object at id. Read more
Source§

fn put_chunk<'life0, 'life1, 'async_trait>( &'life0 self, session: &'life1 Session, offset: u64, content_length: u64, stream: ClientStream, ) -> Pin<Box<dyn Future<Output = Result<UploadProgress>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Writes a chunk of content_length bytes at offset into an open session. Read more
Source§

fn upload_offset<'life0, 'life1, 'async_trait>( &'life0 self, session: &'life1 Session, ) -> Pin<Box<dyn Future<Output = Result<UploadProgress>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Reports how far the session has progressed. Read more
Source§

fn cancel_upload<'life0, 'life1, 'async_trait>( &'life0 self, session: &'life1 Session, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Cancels an upload session, discarding whatever was uploaded. Read more
Source§

fn put_object<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, id: &'life1 ObjectId, metadata: &'life2 Metadata, stream: ClientStream, access_time: Timestamp, ) -> Pin<Box<dyn Future<Output = Result<PutResponse>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Stores an object at the given path with the given metadata.
Source§

fn get_object<'life0, 'life1, 'async_trait>( &'life0 self, id: &'life1 ObjectId, access_time: Timestamp, range: Option<ByteRange>, ) -> Pin<Box<dyn Future<Output = Result<GetResponse>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Retrieves (part of) an object at the given path, returning its metadata, a description of the part being returned, and the payload.
Source§

fn get_metadata<'life0, 'life1, 'async_trait>( &'life0 self, id: &'life1 ObjectId, access_time: Timestamp, ) -> Pin<Box<dyn Future<Output = Result<MetadataResponse>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Retrieves only the metadata for an object, without the payload.
Source§

fn set_expiry<'life0, 'life1, 'async_trait>( &'life0 self, id: &'life1 ObjectId, target: ExpiryUpdate, access_time: Timestamp, ) -> Pin<Box<dyn Future<Output = Result<SetExpiryResponse>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Extends the deadline of an existing object with expiration policy. Read more
Source§

fn delete_object<'life0, 'life1, 'async_trait>( &'life0 self, id: &'life1 ObjectId, access_time: Timestamp, ) -> Pin<Box<dyn Future<Output = Result<DeleteResponse>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Deletes the object at the given path.
Source§

fn join<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Waits for any outstanding background operations to complete before shutdown. Read more
Source§

impl Debug for TieredStorage

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
§

impl<T> IntoRequest<T> for T

§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
§

impl<L> LayerExt<L> for L

§

fn named_layer<S>(&self, service: S) -> Layered<<L as Layer<S>>::Service, S>
where L: Layer<S>,

Applies the layer to a service and wraps it in [Layered].
§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more