500 Internal Server Error when making request to Web3.Storage API endpoint

[rockyessel]

Issue Description

Summary:
Experiencing a 500 Internal Server Error when making requests to the Web3.Storage API, both during file upload and while retrieving CIDs.

Details:

1. Initial Request:

   • When the getCIDs function is called initially, it receives a status of 200 OK.

2. Upload Attempt:

   • Using the handler function along with uploadToWeb3Storage fails intermittently. No error, but got an authorization message. Then fixed the authorization by adding store/add, then tried again, and got 500 Internal Server Error.

3. Subsequent Requests:

   • After a failed upload attempt, subsequent requests to getCIDs also return a 500 Internal Server Error.

4. Code Behavior:

   • The uploadToWeb3Storage function creates a CAR file and sends a JSON payload to https://up.web3.storage/bridge.
   • The getCIDs function attempts to get a list of CIDs from the same endpoint.

working code:

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car';
import { Blob } from 'buffer';
import axios from 'axios';
import dotenv from 'dotenv';
import express from 'express';
import fs from 'fs';
import multer from 'multer';

const app = express();
const port = process.env.PORT || 4000;
const upload = multer();
dotenv.config({ path: '.env' });

async function uploadToWeb3Storage(fileBuffer, spaceKey, authSecret, authToken) {
    const file = new Blob([fileBuffer], { type: 'application/octet-stream' });
    let rootCID;
    const chunks = [];
    await createFileEncoderStream(file)
        .pipeThrough(new TransformStream({
            transform(block, controller) {
                rootCID = block.cid;
                controller.enqueue(block);
            }
        }))
        .pipeThrough(new CAREncoderStream())
        .pipeTo(new WritableStream({
            write(chunk) {
                chunks.push(chunk);
            }
        }));

    const carBuffer = new Uint8Array(chunks.reduce((acc, chunk) => acc + chunk.length, 0));
    let offset = 0;
    for (const chunk of chunks) {
        carBuffer.set(chunk, offset);
        offset += chunk.length;
    }

    const carSize = carBuffer.length;
    const storeJson = {
        tasks: [
            [
                "upload/add",
                spaceKey,
                {
                    link: { "/": rootCID.toString() },
                    size: carSize
                }
            ]
        ]
    };

    try {
        const response = await axios.post('https://up.web3.storage/bridge', storeJson, {
            headers: {
                'X-Auth-Secret': authSecret,
                'Authorization': authToken,
                'Content-Type': 'application/json'
            }
        });

        console.log('Upload successful. Response:', response.data);
        return response.data;
    } catch (error) {
        console.error('Upload failed:', error.response ? error.response.data : error.message);
        throw error;
    }
}

export async function handler(fileBuffer) {
    const W3S_SPACE = process.env.W3S_SPACE;
    const X_AUTH_SECRET = process.env.X_AUTH_SECRET;
    const AUTH_HEADER = process.env.AUTH_HEADER;

    try {
        const result = await uploadToWeb3Storage(fileBuffer, W3S_SPACE, X_AUTH_SECRET, AUTH_HEADER);
        return {
            statusCode: 200,
            body: JSON.stringify(result)
        };
    } catch (error) {
        return {
            statusCode: 500,
            body: JSON.stringify({ error: error.message })
        };
    }
}

app.post('/upload', upload.single('file'), async (req, res) => {
    try {
        const fileBuffer = req.file.buffer;
        const carFile = await handler(fileBuffer);
        res.status(200).json({ carFile });
    } catch (err) {
        console.error('Error creating CAR file:', err);
        res.status(500).send('Error creating CAR file');
    }
});

app.listen(port, () => {
    console.log(`Server running on port ${port}`);
});

const getCIDs = async () => {
    const X_AUTH_SECRET = process.env.X_AUTH_SECRET;
    const AUTH_HEADER = process.env.AUTH_HEADER;
    const W3S_SPACE = process.env.W3S_SPACE;

    try {
        const response = await axios.post('https://up.web3.storage/bridge', {
            "tasks": [
                [
                    "store/list",
                    W3S_SPACE,
                    {}
                ]
            ]
        }, {
            headers: {
                'X-Auth-Secret': X_AUTH_SECRET,
                'Authorization': AUTH_HEADER
            }
        });
        console.log('data-p-fx: ', response.data[0].p.fx);
        console.log('data-p-out: ', response.data[0].p.out);
        console.log('data-p-ran: ', response.data[0].p.ran);
        console.log('data-s: ', response.data[0].s['/']);
    } catch (error) {
        console.error('Failed to retrieve CIDs:', error.message);
    }
};

getCIDs().then(() => console.log('done')).catch((error) => console.log(error));

Error Log:

• 500 Internal Server Error on upload attempts and when calling getCIDs.

Possible Causes:

• Issues with the Web3.Storage API.
• Problems with the JSON payload or headers being sent.
• Server-side issues with the Web3.Storage service.

Actions Taken:

• Verified payload structure and headers.
• Tried multiple requests resulting in consistent 500 errors.

Request for Assistance:

• Diagnose the cause of the 500 Internal Server Error from the Web3.Storage API.
• Confirm if there are known issues with the Web3.Storage API or if there's a need for adjustments in the request format or payload.