Cloud Object Storage

Relevant source files

• core/src/services/azblob/backend.rs
• core/src/services/azblob/core.rs
• core/src/services/azblob/writer.rs
• core/src/services/cos/backend.rs
• core/src/services/cos/core.rs
• core/src/services/cos/writer.rs
• core/src/services/gcs/backend.rs
• core/src/services/gcs/core.rs
• core/src/services/gcs/writer.rs
• core/src/services/obs/backend.rs
• core/src/services/obs/config.rs
• core/src/services/obs/core.rs
• core/src/services/obs/writer.rs
• core/src/services/oss/backend.rs
• core/src/services/oss/core.rs
• core/src/services/oss/writer.rs
• core/src/services/s3/backend.rs
• core/src/services/s3/core.rs
• core/src/services/s3/writer.rs

This page covers OpenDAL's support for cloud object storage services, including AWS S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible services. For file system storage backends, see File Systems. For database storage backends, see Key-Value Stores and Databases.

Overview

OpenDAL provides unified access to major cloud object storage services through a consistent Access trait implementation. Each service follows a common architectural pattern while handling service-specific authentication, request signing, and API differences.

Supported Services

Service	Builder	Backend	Core
AWS S3	S3Builder	S3Backend	S3Core
Google Cloud Storage	GcsBuilder	GcsBackend	GcsCore
Azure Blob Storage	AzblobBuilder	AzblobBackend	AzblobCore
Alibaba Cloud OSS	OssBuilder	OssBackend	OssCore
Tencent Cloud COS	CosBuilder	CosBackend	CosCore
Huawei Cloud OBS	ObsBuilder	ObsBackend	ObsCore

Sources: core/src/services/s3/backend.rs89-97 core/src/services/gcs/backend.rs59-66 core/src/services/azblob/backend.rs73-79 core/src/services/oss/backend.rs59-65 core/src/services/cos/backend.rs57-63 core/src/services/obs/backend.rs57-63

Architecture Pattern

Service Implementation Pattern

Sources: core/src/services/s3/backend.rs718-721 core/src/services/gcs/backend.rs236-239 core/src/services/azblob/backend.rs303-306 core/src/services/oss/backend.rs320-323

Request Signing and Authentication

Sources: core/src/services/s3/core.rs125-158 core/src/services/gcs/core.rs91-133 core/src/services/azblob/core.rs90-128 core/src/services/oss/core.rs97-127

AWS S3 and Compatible Services

Configuration

The S3Builder supports extensive configuration options including:

• Basic Setup: bucket(), endpoint(), region()
• Authentication: access_key_id(), secret_access_key(), session_token()
• Advanced Auth: role_arn(), external_id(), customized_credential_load()
• Features: enable_virtual_host_style(), enable_versioning(), allow_anonymous()
• Server-Side Encryption: Multiple SSE methods (SSE-S3, SSE-KMS, SSE-C)

Sources: core/src/services/s3/backend.rs108-400 core/src/services/s3/backend.rs344-384

S3 Core Operations

The S3Core struct handles all HTTP request construction and signing:

Sources: core/src/services/s3/core.rs364-436 core/src/services/s3/core.rs539-573 core/src/services/s3/core.rs213-289

S3 Capabilities

The S3 backend provides comprehensive capabilities:

Sources: core/src/services/s3/backend.rs910-998

Google Cloud Storage

Authentication and Token Management

GCS uses Google Cloud authentication with OAuth2 tokens:

Sources: core/src/services/gcs/backend.rs265-305 core/src/services/gcs/core.rs91-133

GCS Request Patterns

GCS uses both JSON API and XML API depending on the operation:

• JSON API: Used for most operations via /storage/v1/ endpoints
• XML API: Used for presigned URLs and certain compatibility operations
• Upload API: Supports both simple and multipart upload via /upload/storage/v1/

Sources: core/src/services/gcs/core.rs182-343 core/src/services/gcs/core.rs387-439

Azure Blob Storage

Authentication Methods

Azure Blob Storage supports multiple authentication methods:

Sources: core/src/services/azblob/backend.rs127-231

Azure Storage Features

Azure Blob Storage provides unique features like append blobs and customer-provided encryption:

• Blob Types: Block blobs and append blobs
• Server-Side Encryption: Customer-provided keys (CPK)
• Batch Operations: Batch delete operations
• SAS Token Support: Shared Access Signature authentication

Sources: core/src/services/azblob/core.rs140-169 core/src/services/azblob/writer.rs66-122

Service-Specific Features

Multipart Upload Support

All cloud object storage services support multipart uploads with service-specific constraints:

Service	Min Part Size	Max Part Size	Max Parts
S3	5 MiB	5 GiB	10,000
GCS	5 MiB	5 GiB	10,000
Azure	N/A	N/A	50,000
OSS	100 KiB	5 GiB	10,000
COS	1 MiB	5 GiB	10,000
OBS	5 MiB	5 GiB	10,000

Sources: core/src/services/s3/backend.rs963-971 core/src/services/gcs/backend.rs344-352 core/src/services/oss/backend.rs455-463 core/src/services/cos/backend.rs263-271 core/src/services/obs/backend.rs268-276

Versioning Support

Services that support object versioning:

Sources: core/src/services/s3/backend.rs491-495 core/src/services/oss/backend.rs107-112 core/src/services/cos/backend.rs134-139 core/src/services/obs/backend.rs135-140

Common Operations Implementation

Write Operations

All cloud storage services implement multipart writing through the oio::MultipartWriter:

Sources: core/src/services/s3/writer.rs32 core/src/services/gcs/writer.rs30 core/src/services/azblob/writer.rs31

Error Handling

Each service implements service-specific error parsing while maintaining consistent error types:

Sources: core/src/services/s3/error.rs core/src/services/gcs/error.rs core/src/services/azblob/error.rs core/src/services/oss/error.rs