Building and Testing

Relevant source files

• .github/actions/setup/action.yaml
• .github/services/rocksdb/rocksdb/action.yml
• .github/workflows/test_edge.yml

This page provides comprehensive instructions for building OpenDAL, setting up the development environment, and running tests. It covers dependency installation, build configuration, and the various test categories available for validating OpenDAL functionality.

Development Environment Setup

Setting up a development environment for OpenDAL requires installing Rust, system dependencies, and various tools used for building and testing. The environment setup follows the same pattern used in the CI/CD pipeline.

Development Environment Setup Process

Sources: .github/actions/setup/action.yaml38-77

Required Environment Variables

The following environment variables should be configured for optimal development experience:

Variable	Value	Purpose
RUSTFLAGS	-C force-frame-pointers=yes -C debuginfo=1	Optimize debug builds and enable backtraces
RUST_BACKTRACE	1	Enable detailed error backtraces
RUST_LOG	opendal=trace	Enable detailed logging for OpenDAL components
CARGO_REGISTRIES_CRATES_IO_PROTOCOL	sparse	Use sparse registry protocol for faster builds

Sources: .github/actions/setup/action.yaml44-50

Building OpenDAL

OpenDAL can be built using standard Rust tooling with cargo. The build process supports various feature flags to enable or disable specific storage services and functionality.

Basic Build Commands

Build Dependencies

OpenDAL requires several system dependencies depending on which services you want to build:

System Dependencies Build Flow

Sources: .github/actions/setup/action.yaml55-58 .github/actions/setup/action.yaml94-135

Installing System Dependencies

On Ubuntu/Debian systems:

Building Service-Specific Dependencies

Some services require additional build steps:

RocksDB Setup

Sources: .github/actions/setup/action.yaml94-111

Development Tools

Several specialized tools are used for OpenDAL development and testing:

Installing Development Tools

Tool Configuration

Development tools can be configured through input parameters:

Development Tools Setup Flow

Sources: .github/actions/setup/action.yaml20-33 .github/actions/setup/action.yaml67-78

Running Tests

OpenDAL uses cargo-nextest as the primary test runner for better performance and reporting. Tests are organized into several categories based on functionality and scope.

Basic Test Commands

Test Categories and Execution Flow

Sources: .github/actions/setup/action.yaml67-71 .github/workflows/test_edge.yml28-33

Edge Case Testing

OpenDAL includes specialized edge case tests that verify behavior under unusual conditions:

Sources: .github/workflows/test_edge.yml57-58 .github/workflows/test_edge.yml104-107 .github/workflows/test_edge.yml131-132

Service-Specific Testing

Each storage service requires specific environment configuration for testing. The environment variables and setup procedures vary depending on the service being tested.

RocksDB Testing Configuration

RocksDB tests require specific environment setup:

RocksDB Test Environment Setup

Sources: .github/services/rocksdb/rocksdb/action.yml27-31

S3 Testing with MinIO

For S3 testing, OpenDAL uses MinIO as a local S3-compatible server:

Sources: .github/workflows/test_edge.yml90-101

WebAssembly Testing Environment

For WebAssembly testing, additional setup is required:

Sources: .github/workflows/test_edge.yml72-107

Local Development Testing

Running tests locally requires proper environment setup and dependency installation. Follow these steps for a complete local testing environment:

Step-by-Step Local Setup

1. Install Rust and basic tools:

2. Set environment variables:

3. Install system dependencies (Ubuntu/Debian):

4. Run tests:

Testing Specific Services

Different services may require additional setup:

Service	Additional Requirements	Environment Variables
FileSystem	None	OPENDAL_FS_ROOT
Memory	None	None
S3	MinIO server or AWS credentials	OPENDAL_S3_* variables
RocksDB	RocksDB library installation	ROCKSDB_LIB_DIR, OPENDAL_ROCKSDB_*
FoundationDB	FoundationDB installation	OPENDAL_FOUNDATIONDB_*

Sources: .github/actions/setup/action.yaml44-50 .github/actions/setup/action.yaml55-58

Behavior Testing

OpenDAL uses behavior tests to ensure consistent functionality across all storage service implementations. These tests verify that each service correctly implements the core OpenDAL operations.

Behavior Test Structure

Adding New Service Tests

When implementing a new storage service, create behavior tests that verify:

1. Core Operations Compliance:

   • Accessor::read() returns correct data
   • Accessor::write() stores data properly
   • Accessor::stat() returns accurate metadata
   • Accessor::list() enumerates entries correctly
   • Accessor::delete() removes data as expected

2. Error Handling:

   • Appropriate ErrorKind for different failure scenarios
   • Consistent error behavior across operations

3. Service-Specific Configuration:

   • Create test configuration in .github/services/your_service/
   • Define required environment variables
   • Set up any external dependencies

Sources: .github/services/rocksdb/rocksdb/action.yml17-32

Quality Assurance and Code Checks

OpenDAL includes several automated quality checks that run during testing and CI/CD:

Dependency Auditing

Code Quality Checks

The CI pipeline includes several automated checks:

Quality Assurance Pipeline

Environment Configuration for CI

The CI environment is configured with specific settings for optimal testing:

Configuration	Value	Purpose
RUSTFLAGS	-C force-frame-pointers=yes -C debuginfo=1	Enable backtraces and debugging
RUST_BACKTRACE	1	Show detailed error backtraces
RUST_LOG	opendal=trace	Enable detailed logging
Cache Protocol	sparse	Faster dependency resolution

Sources: .github/actions/setup/action.yaml74-77 .github/actions/setup/action.yaml44-50

Summary

OpenDAL's testing framework is designed to ensure consistent behavior across a diverse range of storage services. The behavior-driven approach, combined with comprehensive setup actions and continuous integration, provides confidence in the reliability and correctness of the code. By focusing on behaviors rather than implementations, the tests help maintain compatibility as the codebase evolves and new storage services are added.