Object的分块上传

更新时间：2022-10-21

分块上传的场景

除了通过putObject()方法上传文件到BOS以外，BOS还提供了另外一种上传模式：分块上传（Multipart Upload）。用户可以在如下的应用场景内（但不仅限于此），使用分块上传模式，如：

需要支持断点上传。
上传超过5GB大小的文件。
网络条件较差，和BOS的服务器之间的连接经常断开。
需要流式地上传文件。
上传文件之前，无法确定上传文件的大小。

Multipart Upload分块上传流程

假设有一个文件，本地路径为 /path/to/file.zip ，由于文件比较大，使用分块上传其传输到BOS中。

基本流程：

初始化Multipart Upload。
上传分块。
完成分块上传。

初始化Multipart Upload

使用 initiateMultipartUpload 方法来初始化一个分块上传事件：

示例代码：

                Java
                
            

                // 开始Multipart Upload
InitiateMultipartUploadRequest initiateMultipartUploadRequest =
      new InitiateMultipartUploadRequest(<BucketName>, <ObjectKey>);
InitiateMultipartUploadResponse initiateMultipartUploadResponse =
      client.initiateMultipartUpload(initiateMultipartUploadRequest);

// 打印UploadId
System.out.println("UploadId: " + initiateMultipartUploadResponse.getUploadId());
            

说明：initiateMultipartUpload 的返回结果中含有 UploadId ，它是区分分块上传事件的唯一标识，在后面的操作中，我们将用到它。

上传分块

将文件分块上传。

示例代码：

                Java
                
            

                // 设置每块为 5MB
final long partSize = 1024 * 1024 * 5L;

File partFile = new File("/path/to/file.zip");

// 计算分块数目
int partCount = (int) (partFile.length() / partSize);
if (partFile.length() % partSize != 0) {
    partCount++;
}
// 新建一个List保存每个分块上传后的ETag和PartNumber
List<PartETag> partETags = new ArrayList<PartETag>();
// 获取文件流
FileInputStream fis = new FileInputStream(partFile);
for (int i = 0; i < partCount; i++) {

    // 计算每个分块的大小
    long skipBytes = partSize * i;
    long size = partSize < partFile.length() - skipBytes ?
              partSize : partFile.length() - skipBytes;

    byte[] buf = new byte[(int)size];
    int offset = 0;
    while (true) {
        int byteRead = fis.read(buf, offset, (int)size);
        offset += byteRead;
        if (byteRead < 0 || offset >= size) {
           break;
        }
    }
    ByteArrayInputStream bufStream = new ByteArrayInputStream(buf);

    // 创建UploadPartRequest，上传分块
    UploadPartRequest uploadPartRequest = new UploadPartRequest();
    uploadPartRequest.setBucketName(bucketName);
    uploadPartRequest.setKey(objectkey);
    uploadPartRequest.setUploadId(initiateMultipartUploadResponse.getUploadId());
    uploadPartRequest.setInputStream(bufStream);
    uploadPartRequest.setPartSize(size);
    uploadPartRequest.setPartNumber(i + 1);
    // 上传进度回调
    uploadPartRequest.setProgressCallback(new BosProgressCallback<UploadPartRequest>() {
        @Override
        public void onProgress(UploadPartRequest request, long currentSize, long totalSize) {
            Log.e(currentSize + "", totalSize + "");
        }
    });

    UploadPartResponse uploadPartResponse = client.uploadPart(uploadPartRequest);

    // 将返回的PartETag保存到List中。
    partETags.add(uploadPartResponse.getPartETag());

    System.out.println(uploadPartResponse.getPartETag());
  }
  // 关闭文件
  fis.close();
            

注意：上面代码的核心是调用 UploadPart 方法来上传每一个分块，但是要注意以下几点：

UploadPart 方法要求Part大小是1MB的整数倍或大于5MB。但是Upload Part接口并不会立即校验上传Part的大小；只有当Complete Multipart Upload的时候才会校验。

为了保证数据在网络传输过程中不出现错误，建议您在UploadPart后，使用每个分块BOS返回的Content-MD5值分别验证已上传分块数据的正确性。当所有分块数据合成一个Object后，不再含MD5值。

Part号码的范围是1~10000。如果超出这个范围，BOS将返回InvalidArgument的错误码。

每次上传Part时都要把流定位到此次上传块开头所对应的位置。

每次上传Part之后，BOS的返回结果会包含一个 PartETag 对象，它是上传块的ETag与块编号（PartNumber）的组合，在后续完成分块上传的步骤中会用到它，因此需要将其保存起来。一般来讲这些 PartETag 对象将被保存到List中。

进度回调接口的使用可以参考“获取上传进度”一章。

完成分块上传

示例代码

                Java
                
            

                CompleteMultipartUploadRequest completeMultipartUploadRequest =
      new CompleteMultipartUploadRequest(<BucketName>, <ObjectKey>, 
            initiateMultipartUploadResponse.getUploadId(), partETags);

// 完成分块上传
CompleteMultipartUploadResponse completeMultipartUploadResponse =
      client.completeMultipartUpload(completeMultipartUploadRequest);

// 打印Object的ETag
System.out.println(completeMultipartUploadResponse.getETag());
            

说明：上面代码中的 partETags 是第二步中保存的partETag的列表，BOS收到用户提交的Part列表后，会逐一验证每个数据Part的有效性。当所有的数据Part验证通过后，BOS将把这些数据part组合成一个完整的Object。

完整示例

                Java
                
            

                import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.json.JSONException;
import android.app.Activity;
import android.os.Bundle;
import com.baidubce.BceClientException;
import com.baidubce.BceServiceException;
import com.baidubce.auth.DefaultBceCredentials;
import com.baidubce.demo.R;
import com.baidubce.services.bos.BosClient;
import com.baidubce.services.bos.BosClientConfiguration;
import com.baidubce.services.bos.model.CompleteMultipartUploadRequest;
import com.baidubce.services.bos.model.CompleteMultipartUploadResponse;
import com.baidubce.services.bos.model.InitiateMultipartUploadRequest;
import com.baidubce.services.bos.model.InitiateMultipartUploadResponse;
import com.baidubce.services.bos.model.PartETag;
import com.baidubce.services.bos.model.UploadPartRequest;
import com.baidubce.services.bos.model.UploadPartResponse;

public class ExampleActivity extends Activity {

private String bucketName = <BucketName>;
private String objectKey = <ObjectKey>;
​    
@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    new Thread(new Runnable() {
        @Override
        public void run() {
            try {
              BosClientConfiguration config = new BosClientConfiguration();
              config.setCredentials(new DefaultBceCredentials(<AccessKeyID>, <SecretAccessKey>));
              config.setEndpoint(<EndPoint>);
              BosClient client = new BosClient(config);

              // 开始Multipart Upload
              InitiateMultipartUploadRequest initiateMultipartUploadRequest = new InitiateMultipartUploadRequest(<BucketName>, <ObjectKey>);
              InitiateMultipartUploadResponse initiateMultipartUploadResponse =
                        client.initiateMultipartUpload(initiateMultipartUploadRequest);

              // 打印UploadId
              System.out.println("UploadId: " + initiateMultipartUploadResponse.getUploadId());

              // 设置每块为 5MB
              final long partSize = 1024 * 1024 * 5L;

              File partFile = new File("/path/to/file.zip");

              // 计算分块数目
              int partCount = (int) (partFile.length() / partSize);
              if (partFile.length() % partSize != 0) {
                  partCount++;
              }

              // 新建一个List保存每个分块上传后的ETag和PartNumber
              List<PartETag> partETags = new ArrayList<PartETag>();

              // 获取文件流
              FileInputStream fis = new FileInputStream(partFile);
              for (int i = 0; i < partCount; i++) {                       

                  // 计算每个分块的大小
                  long skipBytes = partSize * i;
                  long size = partSize < partFile.length() - skipBytes ?
                          partSize : partFile.length() - skipBytes;

                  byte[] buf = new byte[(int)size];
                  int offset = 0;
                  while (true) {
                      int byteRead = fis.read(buf, offset, (int)size);
                      offset += byteRead;
                      if (byteRead < 0 || offset >= size) {
                          break;
                      }
                  }
                  ByteArrayInputStream bufStream = new ByteArrayInputStream(buf);

                  // 创建UploadPartRequest，上传分块
                  UploadPartRequest uploadPartRequest = new UploadPartRequest();
                  uploadPartRequest.setBucketName(<BucketName>);
                  uploadPartRequest.setKey(<ObjectKey>);
                  uploadPartRequest.setUploadId(initiateMultipartUploadResponse.getUploadId());
                  uploadPartRequest.setInputStream(fis);
                  uploadPartRequest.setPartSize(size);
                  uploadPartRequest.setPartNumber(i + 1);
                  UploadPartResponse uploadPartResponse = client.uploadPart(uploadPartRequest);

                  // 将返回的PartETag保存到List中。
                  partETags.add(uploadPartResponse.getPartETag());

                  System.out.println(uploadPartResponse.getPartETag());                                         
              }
              // 关闭文件
              fis.close();

              CompleteMultipartUploadRequest completeMultipartUploadRequest = new CompleteMultipartUploadRequest(<BucketName>, <ObjectKey>, initiateMultipartUploadResponse.getUploadId(), partETags);

              // 完成分块上传
              CompleteMultipartUploadResponse completeMultipartUploadResponse =
                      client.completeMultipartUpload(completeMultipartUploadRequest);

              // 打印Object的ETag
              System.out.println(completeMultipartUploadResponse.getETag());

            } catch (BceServiceException e) {
                System.out.println("Error ErrorCode: " + e.getErrorCode());
                System.out.println("Error RequestId: " + e.getRequestId());
                System.out.println("Error StatusCode: " + e.getStatusCode());
                System.out.println("Error Message: " + e.getMessage());
                System.out.println("Error ErrorType: " + e.getErrorType());
            } catch (BceClientException e) {
                System.out.println("Error Message: " + e.getMessage());
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            } catch (JSONException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            } 
        }
    }).start();
}}
            

取消分块上传

用户可以使用abortMultipartUpload方法取消分块上传。

示例代码：

                Java
                
                AbortMultipartUploadRequest abortMultipartUploadRequest =
    new AbortMultipartUploadRequest(<BucketName>, <ObjectKey>, <UploadId>);

// 取消分块上传
client.abortMultipartUpload(abortMultipartUploadRequest);

获取未完成的分块上传

用户可以使用 listMultipartUploads 方法获取Bucket内未完成的分块上传事件。

基本流程

创建listMultipartUploadsRequest类的实例,传入<BucketName>参数。
创建BOSClient类的实例,执行BOSClient.listMultipartUploads( )方法。
listMultipartUploads()返回所有已上传part的信息。

示例代码

                Java
                
            

                ListMultipartUploadsRequest listMultipartUploadsRequest = 
    new ListMultipartUploadsRequest(<BucketName>);

// 获取Bucket内所有上传事件
ListMultipartUploadsResponse list = client.listMultipartUploads(listMultipartUploadsRequest);
​    
// 遍历所有上传事件
for (MultipartUploadSummary multipartUpload : list.getMultipartUploads()) {
    System.out.println("Key: " + multipartUpload.getKey() + " UploadId: " + multipartUpload.getUploadId());
}
            

注意：

默认情况下，如果Bucket中的分块上传事件的数目大于1000，则只会返回1000个Object，并且返回结果中IsTruncated的值为True，同时返回NextKeyMarker作为下次读取的起点。

若想获取更多分块上传事件，可以使用KeyMarker参数分次读取。

完整示例

                Java
                
            

                import android.app.Activity;
import android.os.Bundle;
import com.baidubce.BceClientException;
import com.baidubce.BceServiceException;
import com.baidubce.auth.DefaultBceCredentials;
import com.baidubce.demo.R;
import com.baidubce.services.bos.BosClient;
import com.baidubce.services.bos.BosClientConfiguration;
import com.baidubce.services.bos.model.ListMultipartUploadsRequest;
import com.baidubce.services.bos.model.ListMultipartUploadsResponse;
import com.baidubce.services.bos.model.MultipartUploadSummary;

public class ExampleActivity extends Activity {

private String bucketName = <BucketName>;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    new Thread(new Runnable() {
        @Override
        public void run() {
            try {
                BosClientConfiguration config = new BosClientConfiguration();
                config.setCredentials(new DefaultBceCredentials(<AccessKeyID>, <SecretAccessKey>));
                config.setEndpoint(<EndPoint>);
                BosClient client = new BosClient(config);

                ListMultipartUploadsRequest listMultipartUploadsRequest = 
                        new ListMultipartUploadsRequest(<BucketName>);

                    // 获取Bucket内所有上传事件
                ListMultipartUploadsResponse listing = client.listMultipartUploads(listMultipartUploadsRequest);

                 // 遍历所有上传事件
                 for (MultipartUploadSummary multipartUpload : listing.getMultipartUploads()) {
                     System.out.println("Key: " + multipartUpload.getKey() + " UploadId: " + multipartUpload.getUploadId());
                 }

            } catch (BceServiceException e) {
                System.out.println("Error ErrorCode: " + e.getErrorCode());
                System.out.println("Error RequestId: " + e.getRequestId());
                System.out.println("Error StatusCode: " + e.getStatusCode());
                System.out.println("Error ErrorType: " + e.getErrorType());
                System.out.println("Error Message: " + e.getMessage());
            } catch (BceClientException e) {
                System.out.println("Error Message: " + e.getMessage());
            } 
        }
    }).start();
}}
            

获取所有已上传的分块信息

用户可以使用 listParts 方法获取某个上传事件中所有已上传的块。

基本流程

创建ListPartsRequest类的实例,传入<BucketName>,<ObjectKey>, <UploadId>参数。
创建BOSClient类的实例,执行BOSClient.listParts( )方法。
listParts()返回所有已上传part的信息。

示例代码

                Java
                
            

                ListPartsRequest listPartsRequest = new ListPartsRequest(<BucketName>, <ObjectKey>, <UploadId>);
​    
// 获取上传的所有Part信息
ListPartsResponse partListing = client.listParts(listPartsRequest);
​    
// 遍历所有Part
for (PartSummary part : partListing.getParts()) {
    System.out.println("PartNumber: " + part.getPartNumber() + " ETag: " + part.getETag());
}
            

注意：

默认情况下，如果Bucket中的分块上传事件的数目大于1000，则只会返回1000个Object，并且返回结果中IsTruncated的值为True，同时返回NextPartNumberMarker作为下次读取的起点。

若想获取更多已上传的分块信息，可以使用PartNumberMarker参数分次读取。

完整示例

                Java
                
            

                import android.app.Activity;
import android.os.Bundle;
import com.baidubce.BceClientException;
import com.baidubce.BceServiceException;
import com.baidubce.auth.DefaultBceCredentials;
import com.baidubce.demo.R;
import com.baidubce.services.bos.BosClient;
import com.baidubce.services.bos.BosClientConfiguration;
import com.baidubce.services.bos.model.ListPartsRequest;
import com.baidubce.services.bos.model.ListPartsResponse;
import com.baidubce.services.bos.model.PartSummary;

public class ExampleActivity extends Activity {

private String bucketName = <BucketName>;
private String objectKey = <ObjectKey>;
private String uploadId = <UploadId>;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    new Thread(new Runnable() {
        @Override
        public void run() {
            try {
                BosClientConfiguration config = new BosClientConfiguration();
                config.setCredentials(new DefaultBceCredentials(<AccessKeyID>, <SecretAccessKey>));
                config.setEndpoint(<EndPoint>);
                BosClient client = new BosClient(config);

                ListPartsRequest listPartsRequest = new ListPartsRequest(<BucketName>, <ObjectKey>, <UploadId>);
                listPartsRequest.setMaxParts(100);
                listPartsRequest.setPartNumberMarker(50);

                 // 获取上传的所有Part信息
                 ListPartsResponse partListing = client.listParts(listPartsRequest);

                 // 遍历所有Part
                 for (PartSummary part : partListing.getParts()) {
                     System.out.println("PartNumber: " + part.getPartNumber() + " ETag: " + part.getETag());
                 }

            } catch (BceServiceException e) {
                System.out.println("Error ErrorCode: " + e.getErrorCode());
                System.out.println("Error RequestId: " + e.getRequestId());
                System.out.println("Error StatusCode: " + e.getStatusCode());
                System.out.println("Error ErrorType: " + e.getErrorType());
                System.out.println("Error Message: " + e.getMessage());
            } catch (BceClientException e) {
                System.out.println("Error Message: " + e.getMessage());
            } 
        }
    }).start();
}}
            

封装分块上传

在Android SDK中，Bos为用户提供了putSuperObjectFromFile接口，它对分块上传涉及到的initiateMultipartUpload、UploadPart、completeMultipartUpload三个方法进行封装，用户只需调用该接口即可完成分块上传，同时，支持进度同步回调。

简单示例:

                Java
                
                File file = new File("/path/to/file.zip");
PutSuperObjectRequest request = new PutSuperObjectRequest(bucketName, objectKey, file);
bosClient.putSuperObjectFromFile(request);

示例：设置分块大小为2MB，线程数为4，并且每1024个byte同步进度回调

                Java
                
            

                BosClientConfiguration config=new BosClientConfiguration();
config.setUploadSegmentPart(1024);
File file = new File("/path/to/file.zip");
PutSuperObjectRequest request = new PutSuperObjectRequest(bucketName, objectKey,
                                                          file, 1024 * 1024 * 2L, 4);
request.setProgressCallback(new BosProgressCallback<PutSuperObjectRequest>() {
    @Override
    public void onProgress(PutSuperObjectRequest request, long currentSize, long totalSize) {
        Log.e(currentSize + "", totalSize + "");
    }
});
bosClient.putSuperObjectFromFile(request);
            

注意：

默认情况下，分块大小是5MB，线程数是5，进度回调周期是每2048个byte。

若一个大文件耗时很长，用户想结束分块上传，可调用PutSuperObjectRequest中的cancel()方法。

拷贝Object

异常处理

百度智能云

BOS 对象存储

BOS 对象存储

Object的分块上传

分块上传的场景

Multipart Upload分块上传流程

初始化Multipart Upload

上传分块

完成分块上传

示例代码

完整示例

取消分块上传

获取未完成的分块上传

基本流程

示例代码

完整示例

获取所有已上传的分块信息

基本流程

示例代码

完整示例

封装分块上传