Skip to content

Here is the complete development documentation for using AWS S3 JS SDK with RustFS, including SDK installation, initialization configuration, basic operations (upload, download, delete, list), presigned URLs and multipart uploads, suitable for Node.js environments.

RustFS AWS S3 JS SDK Documentation (for Node.js)

I. Overview

RustFS is an S3 protocol-compatible object storage system, accessible through AWS's official JavaScript SDK (v3). This guide will show you how to use JS to connect to RustFS and perform common object storage operations.

II. Preparation

2.1 SDK Installation

Install the required AWS SDK v3 modules with NPM:

bash
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

2.2 RustFS Example Configuration

Assume the RustFS instance is deployed as follows:

Endpoint: http://192.168.1.100:9000
Access Key: rustfsadmin
Secret Key: rustfssecret

III. S3 Client Initialization

js
import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@smithy/node-http-handler";

const s3 = new S3Client({
 endpoint: "http://192.168.1.100:9000", // RustFS endpoint
 region: "us-east-1", // can be filled arbitrarily
 credentials: {
 accessKeyId: "rustfsadmin",
 secretAccessKey: "rustfssecret",
 },
 forcePathStyle: true, // Must be enabled for RustFS compatibility
 requestHandler: new NodeHttpHandler({
 connectionTimeout: 3000,
 socketTimeout: 5000,
 }),
});

IV. Basic Operations

4.1 Create Bucket

js
import { CreateBucketCommand } from "@aws-sdk/client-s3";

await s3.send(new CreateBucketCommand({ Bucket: "my-bucket" }));
console.log("Bucket created");

4.2 Upload Object

js
import { PutObjectCommand } from "@aws-sdk/client-s3";
import { readFileSync } from "fs";

const data = readFileSync("hello.txt");

await s3.send(
 new PutObjectCommand({
 Bucket: "my-bucket",
 Key: "hello.txt",
 Body: data,
 })
);

console.log("File uploaded");

4.3 Download Object

js
import { GetObjectCommand } from "@aws-sdk/client-s3";
import { writeFile } from "fs/promises";

const response = await s3.send(
 new GetObjectCommand({ Bucket: "my-bucket", Key: "hello.txt" })
);

const streamToBuffer = async (stream) => {
 const chunks = [];
 for await (const chunk of stream) chunks.push(chunk);
 return Buffer.concat(chunks);
};

const buffer = await streamToBuffer(response.Body);
await writeFile("downloaded.txt", buffer);

console.log("File downloaded");

4.4 List Objects

js
import { ListObjectsV2Command } from "@aws-sdk/client-s3";

const res = await s3.send(new ListObjectsV2Command({ Bucket: "my-bucket" }));
res.Contents?.forEach((obj) => console.log(`${obj.Key} (${obj.Size} bytes)`));

4.5 Delete Object

js
import { DeleteObjectCommand } from "@aws-sdk/client-s3";

await s3.send(new DeleteObjectCommand({ Bucket: "my-bucket", Key: "hello.txt" }));
console.log("File deleted");

V. Advanced Features

5.1 Generate Presigned URLs

Allows frontend or third parties to use temporary links for uploading/downloading files

Download (GET)

js
import { GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const url = await getSignedUrl(
 s3,
 new GetObjectCommand({ Bucket: "my-bucket", Key: "hello.txt" }),
 { expiresIn: 600 }
);

console.log("Presigned GET URL:", url);

Upload (PUT)

js
import { PutObjectCommand } from "@aws-sdk/client-s3";

const url = await getSignedUrl(
 s3,
 new PutObjectCommand({ Bucket: "my-bucket", Key: "upload.txt" }),
 { expiresIn: 600 }
);

console.log("Presigned PUT URL:", url);

5.2 Multipart Upload

js
import {
 CreateMultipartUploadCommand,
 UploadPartCommand,
 CompleteMultipartUploadCommand,
 AbortMultipartUploadCommand,
} from "@aws-sdk/client-s3";
import { createReadStream } from "fs";

const bucket = "my-bucket";
const key = "large-file.zip";
const filePath = "./large-file.zip";
const partSize = 5 * 1024 * 1024; // 5 MB

// 1. Create upload task
const createRes = await s3.send(
 new CreateMultipartUploadCommand({ Bucket: bucket, Key: key })
);
const uploadId = createRes.UploadId;

// 2. Segmented upload
import { statSync, openSync, readSync, closeSync } from "fs";

const fileSize = statSync(filePath).size;
const fd = openSync(filePath, "r");
const parts = [];

for (let partNumber = 1, offset = 0; offset < fileSize; partNumber++) {
 const buffer = Buffer.alloc(Math.min(partSize, fileSize - offset));
 readSync(fd, buffer, 0, buffer.length, offset);

 const uploadPartRes = await s3.send(
 new UploadPartCommand({
 Bucket: bucket,
 Key: key,
 UploadId: uploadId,
 PartNumber: partNumber,
 Body: buffer,
 })
 );

 parts.push({ ETag: uploadPartRes.ETag, PartNumber: partNumber });
 offset += partSize;
}

closeSync(fd);

// 3. Complete upload
await s3.send(
 new CompleteMultipartUploadCommand({
 Bucket: bucket,
 Key: key,
 UploadId: uploadId,
 MultipartUpload: { Parts: parts },
 })
);

console.log("Multipart upload completed");

VI. Common Issues and Notes

ProblemCauseSolution
SignatureDoesNotMatchWrong signature versionJS SDK v3 uses v4 by default, ensure RustFS supports v4
EndpointConnectionErrorEndpoint address misconfigured or not startedCheck if RustFS address is accessible
NoSuchKeyFile does not existCheck if Key is spelled correctly
InvalidAccessKeyId / SecretCredentials misconfiguredCheck accessKeyId / secretAccessKey configuration
Upload failure (path issue)Path-style not enabledSet forcePathStyle: true

VII. Appendix: Frontend Upload Adaptation

Using presigned URLs allows browsers to upload files directly without passing AccessKey.

Frontend (HTML+JS) upload example:

html
<input type="file" id="fileInput" />
<script>
 document.getElementById("fileInput").addEventListener("change", async (e) => {
 const file = e.target.files[0];
 const url = await fetch("/api/presigned-put-url?key=" + file.name).then((r) =>
 r.text()
 );

 const res = await fetch(url, {
 method: "PUT",
 body: file,
 });

 if (res.ok) alert("Uploaded!");
 });
</script>

Released under the Apache License 2.0.