实现文件分片上传

站长

2023年03月18日 12:12 · 阅读数 37

实现文件分片上传

在大文件上传的过程中，文件分片上传是一种非常实用的解决方案。通过将文件分成多个较小的片段，我们可以提高上传速度、减少失败率，以及实现断点续传。本文将介绍如何使用React和Koa2实现文件分片上传功能，包括前端分片处理、后端合并分片、为每个文件分配唯一标识符、实现断点续传和上传进度显示（console打印下得了--！）。

一、前端处理：React

1. 创建文件上传组件

首先，我们需要在React中创建一个文件上传组件：

import React, { useState } from 'react';

const FileUploader = () => {
  const [file, setFile] = useState(null);

  const handleFileChange = (e) => {
    setFile(e.target.files[0]);
  };

  const handleUpload = async () => {
    // 分片上传逻辑
  };

  return (
    <div>
      <input type="file" onChange={handleFileChange} />
      <button onClick={handleUpload}>上传</button>
    </div>
  );
};

export default FileUploader;

2. 实现文件分片

接下来，实现文件分片的逻辑。在handleUpload函数中，将文件切分成多个片段，然后分别上传：

const CHUNK_SIZE = 1024 * 1024; // 分片大小：1MB
const fileChunks = [];
const fileSize = file.size;
const chunksCount = Math.ceil(fileSize / CHUNK_SIZE);

for (let i = 0; i < chunksCount; i++) {
  const start = i * CHUNK_SIZE;
  const end = i === chunksCount - 1 ? fileSize : start + CHUNK_SIZE;
  const chunk = file.slice(start, end);

  fileChunks.push({
    index: i,
    chunk,
  });
}

3. 为每个文件分配唯一标识符

为了实现断点续传功能，我们需要为每个文件分配一个唯一的标识符。这里使用文件名和文件大小生成一个简单的标识符：

const generateFileIdentifier = (file) => {
  return `${file.name}-${file.size}`;
};

const fileIdentifier = generateFileIdentifier(file);

4. 上传分片

遍历fileChunks数组，将每个分片上传到服务器。同时，记录上传进度：

let uploadedSize = 0;
const uploadProgress = {};

for (const { index, chunk } of fileChunks) {
  const formData = new FormData();
  formData.append('file', chunk);
  formData.append('index', index);
  formData.append('identifier', fileIdentifier);

  const response = await fetch('http://localhost:4000/upload-chunk', {
    method: 'POST',
    body: formData,
  });

  if (response.ok) {
    uploadedSize += chunk.size;
    uploadProgress[fileIdentifier] = (uploadedSize / fileSize) * 100;

    // 更新进度显示
    console.log(`上传进度：${uploadProgress[fileIdentifier]}%`);
  } else {
    // 上传失败处理
  }
}

二、后端处理：Koa2

1. 安装依赖

在Koa2项目中，我们需要使用koa-body和koa-router中间件来处理文件上传请求。首先安装这两个依赖：

npm install koa-body koa-router

2. 配置中间件

在Koa2应用中引入并配置koa-body和koa-router中间件：

const Koa = require('koa');
const KoaBody = require('koa-body');
const Router = require('koa-router');

const app = new Koa();
const router = new Router();

app.use(
  KoaBody({
    multipart: true,
    formidable: {
      maxFileSize: 10 * 1024 * 1024 * 1024, // 设置最大文件大小：10GB
    },
  })
);

app.use(router.routes());
app.use(router.allowedMethods());

3. 实现文件分片上传接口

接下来，我们需要在后端实现一个文件分片上传的接口。首先创建一个临时目录用于存储上传的文件分片：

const fs = require('fs');
const path = require('path');

const TEMP_DIR = path.resolve(__dirname, 'temp');

if (!fs.existsSync(TEMP_DIR)) {
  fs.mkdirSync(TEMP_DIR);
}

然后，实现文件分片上传接口：

router.post("/upload-chunk", async (ctx) => {
  const file = ctx.request.files.file;
  const index = ctx.request.body.index;
  const identifier = ctx.request.body.identifier;

  const chunkPath = path.join(TEMP_DIR, `${identifier}-${index}`);
  await fs.promises.copyFile(file.path || file.filepath, chunkPath);
  await fs.promises.unlink(file.path || file.filepath);

  ctx.body = {
    success: true,
  };
});

4. 实现文件合并接口

最后，我们需要实现一个文件合并接口，用于将所有分片合并成一个完整的文件：

router.post("/merge-chunks", async (ctx) => {
  const identifier = ctx.request.body.identifier;
  const fileName = ctx.request.body.fileName;
  const outputPath = path.join(__dirname, "temp", fileName);

  const files = await fs.promises.readdir(TEMP_DIR);
  const chunks = files
    .filter((file) => file.startsWith(identifier))
    .sort((a, b) => parseInt(a.split("-")[2]) - parseInt(b.split("-")[2]));

  const writeStream = fs.createWriteStream(outputPath);

  for (const chunk of chunks) {
    const chunkPath = path.join(TEMP_DIR, chunk);
    const readStream = fs.createReadStream(chunkPath);

    await new Promise((resolve) => {
      readStream.pipe(writeStream, { end: false });
      readStream.on("end", () => {
        fs.promises.unlink(chunkPath);
        resolve();
      });
    });
  }

  writeStream.end();

  ctx.body = {
    success: true,
  };
});

三、实现断点续传

为了实现断点续传功能，我们需要在后端实现一个接口，用于检查已上传的文件分片。首先，在前端添加一个checkUploadedChunks函数，用于向后端发送请求：

const checkUploadedChunks = async (identifier) => {
    const response = await fetch('http://localhost:4000/check-uploaded', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            identifier,
        }),
    });

    if (response.ok) {
        const data = await response.json();
        return data.uploadedChunks;
    }

    return [];
};

接下来，在前端的handleUpload函数中，调用checkUploadedChunks函数，跳过已上传的分片：

const uploadedChunks = await checkUploadedChunks(fileIdentifier);

for (const { index, chunk } of fileChunks) {
  if (uploadedChunks.includes(index)) {
    // 跳过已上传的分片
    continue;
  }

  // 上传分片的逻辑...
}

然后，在后端实现/check-uploaded接口，返回已上传的分片索引：

router.post("/check-uploaded", async (ctx) => {
  const identifier = ctx.request.body.identifier;

  const files = await fs.promises.readdir(TEMP_DIR);
  const uploadedChunks = files
    .filter((file) => file.startsWith(identifier))
    .map((file) => parseInt(file.split("-")[2]));

  ctx.body = {
    success: true,
    uploadedChunks,
  };
});

至此，我们已经实现了断点续传功能。当用户重新上传相同的文件时，前端会检查已上传的分片，然后只上传未完成的分片。

四、整合后完整示例

前端：React

import React, { useState } from 'react';

const FileUploader = () => {
    const [file, setFile] = useState(null);

    const handleFileChange = (e) => {
        setFile(e.target.files[0]);
    };

    const generateFileIdentifier = (file) => {
        return `${file.name}-${file.size}`;
    };

    const checkUploadedChunks = async (identifier) => {
        const response = await fetch('http://localhost:4000/check-uploaded', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                identifier,
            }),
        });

        if (response.ok) {
            const data = await response.json();
            return data.uploadedChunks;
        }

        return [];
    };

    const uploadChunk = async (chunk, index, identifier) => {
        const formData = new FormData();
        formData.append('file', chunk);
        formData.append('index', index);
        formData.append('identifier', identifier);

        const response = await fetch('http://localhost:4000/upload-chunk', {
            method: 'POST',
            body: formData,
        });

        return response.ok;
    };

    const mergeChunks = async (identifier, fileName) => {
        const response = await fetch('http://localhost:4000/merge-chunks', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                identifier,
                fileName,
            }),
        });

        return response.ok;
    };

    const handleUpload = async () => {
        const CHUNK_SIZE = 1024 * 1024;
        const fileChunks = [];
        const fileSize = file.size;
        const chunksCount = Math.ceil(fileSize / CHUNK_SIZE);

        for (let i = 0; i < chunksCount; i++) {
            const start = i * CHUNK_SIZE;
            const end = i === chunksCount - 1 ? fileSize : start + CHUNK_SIZE;
            const chunk = file.slice(start, end);

            fileChunks.push({
                index: i,
                chunk,
            });
        }

        const fileIdentifier = generateFileIdentifier(file);
        const uploadedChunks = await checkUploadedChunks(fileIdentifier);
        let uploadedSize = 0;
        const uploadProgress = {};

        for (const { index, chunk } of fileChunks) {
            if (uploadedChunks.includes(index)) {
                continue;
            }

            const success = await uploadChunk(chunk, index, fileIdentifier);
            if (success) {
                uploadedSize += chunk.size;
                uploadProgress[fileIdentifier] = (uploadedSize / fileSize) * 100;

                console.log(`上传进度：${uploadProgress[fileIdentifier]}%`);
            } else {
                // 上传失败处理
            }
        }

        const mergeSuccess = await mergeChunks(fileIdentifier, file.name);
        if (mergeSuccess) {
            console.log('文件上传成功');
        } else {
            console.log('文件合并失败');
        }
    };

    return (
        <div>
            <input type="file" onChange={handleFileChange} />
            <button onClick={handleUpload}>上传</button>
        </div>
    );
};

export default FileUploader;

后端：Koa2

// server.js
const Koa = require("koa");
const KoaBody = require("koa-body");
const Router = require("koa-router");
const fs = require("fs");
const path = require("path");
const cors = require("@koa/cors");

const app = new Koa();
const router = new Router();
app.use(cors());
app.use(
  KoaBody({
    multipart: true,
    formidable: {
      maxFileSize: 10 * 1024 * 1024 * 1024, // 设置最大文件大小：10GB
    },
  })
);

app.use(router.routes());

app.use(router.allowedMethods());

const TEMP_DIR = path.resolve(__dirname, "temp");

if (!fs.existsSync(TEMP_DIR)) {
  fs.mkdirSync(TEMP_DIR);
}

router.post("/upload-chunk", async (ctx) => {
  const file = ctx.request.files.file;
  const index = ctx.request.body.index;
  const identifier = ctx.request.body.identifier;

  const chunkPath = path.join(TEMP_DIR, `${identifier}-${index}`);
  await fs.promises.copyFile(file.path || file.filepath, chunkPath);
  await fs.promises.unlink(file.path || file.filepath);

  ctx.body = {
    success: true,
  };
});

router.post("/merge-chunks", async (ctx) => {
  const identifier = ctx.request.body.identifier;
  const fileName = ctx.request.body.fileName;
  const outputPath = path.join(__dirname, "temp", fileName);

  const files = await fs.promises.readdir(TEMP_DIR);
  const chunks = files
    .filter((file) => file.startsWith(identifier))
    .sort((a, b) => parseInt(a.split("-")[2]) - parseInt(b.split("-")[2]));

  const writeStream = fs.createWriteStream(outputPath);

  for (const chunk of chunks) {
    const chunkPath = path.join(TEMP_DIR, chunk);
    const readStream = fs.createReadStream(chunkPath);

    await new Promise((resolve) => {
      readStream.pipe(writeStream, { end: false });
      readStream.on("end", () => {
        fs.promises.unlink(chunkPath);
        resolve();
      });
    });
  }

  writeStream.end();

  ctx.body = {
    success: true,
  };
});

router.post("/check-uploaded", async (ctx) => {
  const identifier = ctx.request.body.identifier;

  const files = await fs.promises.readdir(TEMP_DIR);
  const uploadedChunks = files
    .filter((file) => file.startsWith(identifier))
    .map((file) => parseInt(file.split("-")[2]));

  ctx.body = {
    success: true,
    uploadedChunks,
  };
});

app.listen(4000, () => {
  console.log("Server is running on port 4000");
});

五、总结

本文详细介绍了如何使用React和Koa2实现文件分片上传功能。我们首先在前端实现文件切片，并为每个文件分配唯一标识符。然后在后端实现分片上传和合并接口。最后，通过检查已上传的分片，实现了断点续传功能。同时，我们还添加了上传进度显示，以便用户了解上传状态。

实际项目中可能需要根据具体需求进行调整，例如添加更多的错误处理、优化上传性能等。但是，本文提供的基本框架已经足够应对大多数文件分片上传的场景。

转载自:https://juejin.cn/post/7211401380770627643

实现文件分片上传

实现文件分片上传

一、前端处理：React

1. 创建文件上传组件

2. 实现文件分片

3. 为每个文件分配唯一标识符

4. 上传分片

二、后端处理：Koa2

1. 安装依赖

2. 配置中间件

3. 实现文件分片上传接口

4. 实现文件合并接口

三、实现断点续传

四、 整合后完整示例

前端：React

后端：Koa2

五、总结

四、整合后完整示例