On-demand migration
Pull missing objects into a RustFS bucket from an external source on a read or a backfill, with Cloudflare R2 as the worked example.
This page explains on-demand migration, what you need before you start, and a migration walkthrough. The walkthrough uses Cloudflare R2 as the worked example.
Overview
On-demand migration is an inbound pull. When a client reads a key that is not in the local bucket, RustFS reads that object from the configured source, returns it, and stores it locally. You can also start a backfill to pull keys that no client has requested yet. The enable form states that RustFS never writes or deletes objects on the source. The migration page states that new PUT and DELETE operations in RustFS are not synchronized to the source, and that later source changes do not update local copies. On-demand migration is not continuous replication from the source.
| Goal | Use |
|---|---|
| Pull objects into RustFS from a provider in the list below, on a read miss or by backfill | On-demand migration |
| Push object versions from a RustFS bucket to another bucket | Bucket replication |
| Synchronize site configuration and identity data across deployments | Site replication |
Bucket replication is an outbound copy of object versions and requires versioning on both buckets. After a key is stored locally, on-demand migration does not keep following the source.
Source connection → Provider lists these sources. The names below are the Console labels. Google Cloud Storage is two entries, HMAC and Native.
- Amazon S3
- S3-compatible storage
- MinIO
- RustFS
- Cloudflare R2
- Google Cloud Storage (HMAC)
- Azure Blob Storage (Native)
- Google Cloud Storage (Native)
If you need a provider that is not in this list, open a pull request in the RustFS repository for official review.
Do not close a replication loop
Do not select a bucket that is already a replication target of this local bucket as the on-demand migration source.
Enabling the source does not copy data
Migration active means the bucket is attached to the source. Runtime counters stay at zero until a client reads a missing key or you start a backfill.
Requirement
- An installed RustFS instance. Use the Linux quick start to install RustFS, sign in to the Console, and create a bucket. This walkthrough used Console v1.0.3 on port
9001. - The RustFS CLI,
rc, on the host that will read the local bucket. - A Cloudflare R2 bucket for this example.
Migration
This section uses Cloudflare R2 as the example. A migration from any other provider in the Overview list follows the same flow: create an empty local bucket, enable on-demand migration, choose Provider, select Test connection, select Enable migration, then pull on a GET miss or with a backfill. Only the endpoint, credential, and bucket fields differ by provider.
Screenshots in this walkthrough use placeholders for the R2 account id and access key. The terminal prompt host is localhost.
The credential and endpoint here are for the Cloudflare R2 example. For another provider, enter that system's endpoint, read credential, and bucket name in the same Console fields.
- A RustFS Console that includes On-demand migration. This walkthrough used Console v1.0.3, on port
9001. - A Console account that can open the local bucket's Settings.
- The RustFS CLI,
rc, on the host that will read the local bucket. - A Cloudflare R2 API token that can read objects in the source bucket. Prefer a token scoped to that bucket. Public access on the R2 bucket is not part of this flow.
- The R2 S3 endpoint for the source bucket,
https://<account-id>.r2.cloudflarestorage.com. If the bucket uses a jurisdiction, copy the endpoint from the R2 dashboard.
Point rc at the local RustFS S3 API, not at R2:
rc alias set rustfs http://localhost:9000 <your-access-key> <your-secret-key> --region us-east-1 --bucket-lookup pathThe local bucket in this walkthrough is migration. The R2 bucket is dl-rustfs. Versioning was enabled on migration, and Object Lock was disabled. The walkthrough did not need Object Lock to enable on-demand migration, and it did not treat versioning as a requirement for enabling it. With versioning enabled, a local delete marker can hide the source key.
Use a read-only source token
Create an R2 token that can read the source bucket and cannot write or delete there. The form states that RustFS stores the secret and does not return it to the Console.
Create an empty local bucket
In the Console, open Browser → Buckets, create a bucket named migration, and confirm that it has no objects. Open Settings from that bucket when you are ready to attach the source.
Open on-demand migration
- Open bucket
migrationand select Settings. - Select Automation.
- Find On-demand migration.
- When it is not configured, select Open migration settings or + Enable on-demand migration. When it is already active, select Manage migration.
The unconfigured page in this walkthrough says that no migration source is configured and offers + Enable on-demand migration.
After the source is enabled, Automation shows Migration active and Manage migration.
Point the bucket at Cloudflare R2
In Enable on-demand migration, set the source as follows. Replace the account id and keys with your own R2 token. Leave the session token empty unless the token requires one. Leave Local prefix empty to expose the whole source namespace. This walkthrough saved no extra prefix (All keys, No added prefix).
| Field | Value in this walkthrough |
|---|---|
| Source | Cloudflare R2 |
| Endpoint | https://<account-id>.r2.cloudflarestorage.com |
| Region | us-east-1 (the form also accepts auto) |
| Source Bucket | dl-rustfs |
| Addressing style | Automatic |
| Access Key | R2 access key id |
| Secret Key | R2 secret |
| Session token | Empty |
| Local prefix | Empty |
Test the connection and enable migration
Select Test connection. A successful check reports Connection test passed and shows a sample object key from the source bucket. Then select Enable migration.
The same result also appears as a toast:
The configuration status becomes Migration active. Until something is read or backfilled, Runtime status stays at zero migrated objects and zero migrated data. Latest job stays No backfill yet.
Pull one object with rc cp
Read the local bucket with rc. Do not send this request to the R2 endpoint. In this walkthrough the local bucket was still empty, and one missing object was pulled. The first miss returns the object to the client and stores it locally. A later read of the same key is served from RustFS.
rc cp rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb /tmp/rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb -> /tmp/rustfs-nightly-2026-11-02.deb (71.34 MiB)Use another key if the source bucket does not contain this object. The terminal session shows that rc cp and later rc ls checks of the local nightly prefix. The listing is the local prefix after objects were present. The single rc cp above is the 71.34 MiB pull.
The same nightly prefix on the R2 bucket, as it appeared in the Cloudflare dashboard:
The bucket-size figure on that dashboard is Cloudflare's total for the bucket. The RustFS counters later on this page are what the migration job reported.
After the single-object pull, the Console Browser shows that object in the local bucket:
Backfill a prefix
Start backfill scans the source and pulls keys that are not already local. The dialog in this walkthrough used these settings:
| Option | Meaning | Walkthrough value |
|---|---|---|
| Source listing prefix | List only this prefix | artifacts/rustfs/packages/nightly/ |
| Existing local objects | What to do when the key is already local | Always skip |
| Dry run | List matches without queueing or writing objects | On for the rehearsal, then off for the pull |
Dry run lists matching objects without queueing or writing any object. Confirm the prefix, then turn Dry run off before you pull.
A prefix listing recorded during the walkthrough follows. The first key was truncated in the lab record. The full key has the form artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-29.deb. Timestamps are the lab record, about UTC. The full list is longer than this excerpt.
…ghtly-2026-10-29.deb
[2026-11-02 03:25:08] 70.36 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-30.deb
[2026-11-02 03:25:08] 70.65 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-31.deb
[2026-11-02 03:25:08] 71.15 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-01.deb
[2026-11-02 03:14:39] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb
[2026-11-02 03:25:08] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-latest.debWhile a backfill is writing objects, the Console shows listed and pulled counts. It does not show a completion percentage while the source scan is still running.
Run a dry run, then backfill the source namespace
Leave Source listing prefix empty to scan the configured source namespace. Run Dry run first and confirm the listed set. Then turn Dry run off and start the backfill.
A dry run in this walkthrough finished with objects listed and nothing pulled:
| Listed | Pulled | Skipped | Failed objects | Data |
|---|---|---|---|---|
| 128 | 0 | 0 | 0 | 0 B |
The full backfill that followed, with an empty prefix, finished as follows:
| Listed | Pulled | Skipped | Failed objects | Data |
|---|---|---|---|---|
| 128 | 127 | 1 | 0 | 8.0 GiB |
Verify the bucket
On the on-demand migration page, check:
- Runtime status — migrated objects and migrated data are no longer zero, and no pull failures are recorded.
- Background backfill — the latest job shows the listed, pulled, skipped, and failed counts.
- Browser and
rc— list the prefix withrc ls, as in the terminal session above, and read a local key again withrc cp.
In this walkthrough the runtime panel showed Migrated data 8.1 GiB and Migrated objects 128. The latest backfill showed Listed 128, Pulled 127, Skipped 1, Failed objects 0, and Data 8.0 GiB.
The bucket list after the walkthrough shows migration with 128 objects and 8.1 GiB:
The local copy does not track the source
A later change to the same key on R2 is not pulled over the local object. A local PUT of that key is not written back to the source, and it stays in front of the source key. On-demand migration is not continuous replication from the source.
Next steps
- Configure bucket replication when you need an outbound copy
- Install and use rc
- Configure site replication for site configuration and identity data