From 6334743873221127d5251c4e33cad57e85fe955c Mon Sep 17 00:00:00 2001 From: wubingheng Date: Tue, 18 Feb 2020 21:23:15 +0800 Subject: [PATCH 01/12] optimize some processors' result and optimize docs. --- Makefile | 2 +- docs/asyncfetch.md | 3 +- docs/avinfo.md | 3 +- docs/cdn.md | 12 ++-- docs/censor.md | 9 +-- docs/censorresult.md | 2 +- docs/copy.md | 3 +- docs/datasource.md | 63 ++++++++++++------- docs/downloadfile.md | 3 +- docs/exportts.md | 3 +- docs/fetch.md | 3 +- docs/mime.md | 2 +- docs/move.md | 3 +- docs/pfop.md | 2 +- docs/pfopcmd.md | 2 +- docs/pfopresult.md | 2 +- docs/privateurl.md | 3 +- docs/publicurl.md | 2 +- docs/qhash.md | 3 +- docs/rename.md | 3 +- docs/syncupload.md | 7 ++- docs/uploadfile.md | 3 +- pom.xml | 2 +- .../java/com/qiniu/entry/QSuitsEntry.java | 5 +- .../com/qiniu/process/other/PublicUrl.java | 50 +++++++++------ .../com/qiniu/process/qiniu/PrivateUrl.java | 4 +- .../com/qiniu/process/qiniu/SyncUpload.java | 2 +- .../com/qiniu/process/qiniu/UploadFile.java | 4 +- .../java/com/qiniu/util/ProcessUtils.java | 3 +- version.properties | 2 +- 30 files changed, 129 insertions(+), 81 deletions(-) diff --git a/Makefile b/Makefile index 3dad57ce..593f2e2a 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -version=8.4.0 +version=8.4.1 package_no_test: mvn package -Dmaven.test.skip=true diff --git a/docs/asyncfetch.md b/docs/asyncfetch.md index e512af15..6184fb14 100644 --- a/docs/asyncfetch.md +++ b/docs/asyncfetch.md @@ -56,7 +56,8 @@ check-url= ### 关于 url-index 和 md5-index 当使用 file 数据源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行中含 url 的字段索引，未设置的情况下则使用 -key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)，md5-index 为需要进行 md5 校验时输入 md5 值的字段下标，不设置则无效。 +key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)，md5-index 为需要进行 md5 校验时输入 md5 值的字段下标，不设置则无效。 ### 命令行参数方式 ``` diff --git a/docs/avinfo.md b/docs/avinfo.md index 2f8c3b76..7f5938f8 100644 --- a/docs/avinfo.md +++ b/docs/avinfo.md @@ -24,7 +24,8 @@ url-index= ### 关于 url-index 当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用 -key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 ### 命令行方式 ``` diff --git a/docs/cdn.md b/docs/cdn.md index 3a4431e8..a297e3bf 100644 --- a/docs/cdn.md +++ b/docs/cdn.md @@ -32,7 +32,8 @@ is-dir= #### 关于 url-index 当 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行中存在 url 形式的源文件地址，未设置的情况下则默认从 key 字段 -加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 #### 批量刷新的 url 条数 刷新操作分为目录刷新和普通 url 刷新，目录刷新默认每次提交 10 条，普通 url 刷新默认每次提交 30 条，如果需要其他规模，可以通过 batch-size 参数来 @@ -62,6 +63,11 @@ url-index= |indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| |url-index| 字符串| 通过 url 操作时需要设置的 url 索引（下标），未设置任何索引和 domain 时根据 parse 类型默认为 0 或 "url"| +#### 关于 url-index +当 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行中存在 url 形式的源文件地址，未设置的情况下则默认从 key 字段 +加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 + #### 批量预取的 url 条数 批量预取默认每次提交 30 条，如果需要其他规模，可以通过 batch-size 参数来设置，如 `batch-size=50`，但是批量预取资源的每次提交条数上限是 60。 @@ -89,10 +95,6 @@ url-index= |indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| |url-index| 字符串| 通过 url 操作时需要设置的 url 索引（下标），未设置任何索引和 domain 时根据 parse 类型默认为 0 或 "url"| -#### 关于 url-index -当 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行中存在 url 形式的源文件地址，未设置的情况下则默认从 key 字段 -加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 - #### 批量查询的 url 条数 每次提交查询条数默认 100 条，如果需要其他规模，可以通过 batch-size 参数来设置，如 `batch-size=80`，但是查询的每次提交条数上限是 100。 diff --git a/docs/censor.md b/docs/censor.md index 0ace9b56..ecb4122b 100644 --- a/docs/censor.md +++ b/docs/censor.md @@ -36,6 +36,11 @@ private= |scenes| 审核类型字符串| pulp/terror/politician，鉴黄、鉴暴恐、鉴政，多种类型同时审核可用 `,` 拼接，如：`pulp,terror` 或 `pulp,terror,politician` 等| |private| 数据源私有类型|是否是对私有空间资源进行审核，选择对应的私有类型，参考[私有访问](#资源需要私有签名)| +#### 关于 url-index +当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用 +key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 + #### 关于 queries 参数 queries 参数用于设置 url 的后缀或 ?+参数部分，内容审核可能会出现大图超过尺寸或大小导致失败，因此可以通过一些图片处理样式或参数来设置对处理之后的图 片进行审核。当设置 private（私有签名）的情况下，该参数会使用在 privateurl 操作中（因为 privateurl 操作在前，当前操作在后）。 @@ -76,10 +81,6 @@ private= |check-url| true/false|表示是否在提交任务之前对回调地址进行简单的 post 请求验证（无body的纯post请求），默认为 true，如果无需验证则设置为 false| |private| 数据源私有类型|是否是对私有空间资源进行审核，选择对应的私有类型，参考[私有访问](#资源需要私有签名)| -#### 关于 url-index -当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用 -key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 - ### 资源需要私有签名 当进行图片审核的 url 需要通过私有鉴权访问时（资源来自于存储私有权限的空间），本工具支持串联操作，即先进行对应的私有签名再提交审核，使用如下的 private 参数设置即可，如不需要进行私有访问则不设置，目前支持以下几类签名： diff --git a/docs/censorresult.md b/docs/censorresult.md index 25f48bcd..d9adc83d 100644 --- a/docs/censorresult.md +++ b/docs/censorresult.md @@ -26,4 +26,4 @@ id-index= ``` ## 备注 -censorresult 操作是 file 源下的操作，从 every line of fileinput 的 id-index 索引获取 jobId。 +censorresult 操作是 file 源下的操作，从 every line of fileinput 的 id-index 索引获取 jobId，参见[关于 parse 和索引](datasource.md#关于-parse)。 diff --git a/docs/copy.md b/docs/copy.md index 6bc55f7a..7fa8546e 100644 --- a/docs/copy.md +++ b/docs/copy.md @@ -33,7 +33,8 @@ rm-prefix= ### 关于 toKey-index 指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，但未设置且 add-prefix 不为空 时需要强制指定 prefix-force=true，表明该次重命名操作只添加文件名前缀。由于 rename 操作既需要原始文件名字段也需要新文件名字段，因此 toKey 下 -标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 **注意**：七牛存储空间不支持文件名以 `../`, `./` 开头或者包含 `/../`, `/./` 这种情况，会造成无法访问，因此设置文件名时请注意。 ### 命令行方式 diff --git a/docs/datasource.md b/docs/datasource.md index b0b8371b..05141ce8 100644 --- a/docs/datasource.md +++ b/docs/datasource.md @@ -39,7 +39,7 @@ threads 说明及默认值参考下述[并发处理](#关于并发处理 )，建 多数数据源默认设置为顺序包含 9 个字段：**key,etag,size,datetime,mime,type,status,md5,owner**，参考[文件信息字段](#关于文件信息字段)。但 是对于 file 数据源进行上传操作时或者 `parse=file` 导出文件信息列表时，默认字段为 **key,parent,size,datetime,mime,etag**，key 为解析的文 件名称，字段减少和顺序变化的原因是因为 mime 和 etag 需要经过计算得到，故优先级放低，不是必需的情况下可以舍弃，size 和 datetime 为非关键信息，也 -可以选择舍弃，parent 表示该文件所在的父层路径。 +可以选择舍弃，parent 表示该文件所在的父层路径。而文件本身的 filepath 索引则为默认项，字段为 filepath-index。 2、**indexes 索引的设置方式有三种** （1）使用 `indexes=pre-<索引的前几位个数>`，即按照索引的字段顺序需要前几个默认字段，如 `indexes=pre-3`，表示输入字段需要 key,etag,size，如 @@ -64,15 +64,15 @@ threads 说明及默认值参考下述[并发处理](#关于并发处理 )，建 序会自动识别并扩展出该字段，如 `indexes=key,etag,size,timestamp,mime,type`，此时持久化结果中或者 process 的输入行中将体现 timestamp，亦 可使两者同时存在，如 `indexes=key,etag,size,datetime,timestamp,mime,type,status,md5,owner`。 -（2）当数据源为 [file](#2-file-文本文件行读取) 类型时，默认情况下，程序只从输入行中解析 `key` 字段数据，因此当输入格式为 `tab/csv` 时索引只有 -`0`，输入格式为 `json` 时索引只有 `key`，使用默认值时若存在 [filter](filter.md) 过滤字段则会自动添加过滤字段，需要指定更多字段时可按照[ indexes 规范](#关于-indexes-索引) -设置，例如为数字列表:`0,1,2,3,...` 或者 `json` 的 `key` 名称列表，采用默认字段的设置方式时长度不超过 9，表明取对应顺序的前几个字段，当数据格式 -为 `tab/csv` 时索引必须均为整数，如果输入行中本身只包含部分字段，则可以在缺少字段的顺序位置用 `-1` 索引表示，表示跳过该顺序对应的字段，例如原输入 -行中不包含 mime 字段，则可以设置 `indexes=0,1,2,3,-1,5`。 +（2）当数据源为 [file-文件列表](#2.1-文本文件行读取)类型时，默认情况下，程序只从输入行中解析 `key` 字段数据，因此当输入格式为 `tab/csv` 时索引 +只有 `0`，输入格式为 `json` 时索引只有 `key`，使用默认值时若存在 [filter](filter.md) 过滤字段则会自动添加过滤字段，需要指定更多字段时可按照 +[ indexes 规范](#关于-indexes-索引)设置，例如为数字列表:`0,1,2,3,...` 或者 `json` 的 `key` 名称列表，采用默认字段的设置方式时长度不超过 9， +表明取对应顺序的前几个字段，当数据格式为 `tab/csv` 时索引必须均为整数，如果输入行中本身只包含部分字段，则可以在缺少字段的顺序位置用 `-1` 索引表示， +表示跳过该顺序对应的字段，例如原输入行中不包含 mime 字段，则可以设置 `indexes=0,1,2,3,-1,5`。 -（3）对于 file 数据源，存在的特殊情况是即扫描 path 下的文件进行上传（`process=qupload`）或者导出列表（`parse=file`），对于这种情况，indexes -的设置只允许使用第一种设置方式，即 `indexes=pre-<个数>`，个数不超过 6，如果不设置则默认会取得两项信息：filepath 和 key，key 是属于 indexes -的第一个默认字段，而 filepath 是必备字段且对于 path 下文件的直接扫描不希望更改该字段的索引值，默认即为 filepath。 +（3）对于 file 文件数据源，即扫描 path 下的文件进行上传（`process=qupload`）或者导出列表（`parse=file`），这种数据源参数长度不能超过 6，其 +indexes 的设置通常采用第一种方式，即 `indexes=pre-<个数>`，如果不设置则默认会取得 key，而 filepath 作为必备字段使用单独的 filepath-index +字段来设置索引，但对于 path 下文件的直接扫描不能更改该字段的索引值，默认即为 filepath。 #### 关于并发处理 1、云存储数据源，从存储空间中列举文件，可多线程并发列举，用于支持大量文件的加速列举，线程数在配置文件中指定，自动按照线程数检索前缀并执行并发列举。 @@ -86,19 +86,19 @@ threads 说明及默认值参考下述[并发处理](#关于并发处理 )，建 息可信度会比较高。 #### 关于文件信息字段 -文件信息规范字段及顺序定义为：**key,etag,size,datetime,mime,type,status,md5,owner** (indexes 的默认字段及顺序即使用该字段列表)，默认根 -据七牛存储文件的信息字段进行定义，顺序固定，其释义及其他数据源方式对应关系如下： - -|数据源 |key（文件名） |etag（文件唯一值）|size（文件大小 kb）|datetime（日期时间字符串）|mime（mime-type/content-type）|type（资源存储类型）|status（资源状态）|md5（文件 md5）|owner（终端标识符）| -|-----------|---------------|---------------|-----------------|-----------------------|-----------------------------|-----------------|---------------|-------------|-----------------| -|文件列表输入行|indexes第1个索引|indexes的第2个索引|indexes的第3个索引|indexes的第4个索引 |indexes的第5个索引 |indexes的第6个索引 |indexes的第7个索引|indexes的第8个索引|indexes的第9个索引| -|七牛云存储 |key |eatg（hash） |fsize |putTime 转换而来 |mimeType |type |status |md5 |endUser | -|腾讯云存储 |key |etag |size |lastModified 转换而来 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName| -|阿里云存储 |key |etag |size |lastModified 转换而来 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName| -|AWS云存储/S3|key |etag |size |lastModified 转换而来 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName | -|又拍云存储 |name |无此含义字段 |length |lastModified 转换而来 |type/attribute |无此含义字段 |无此含义字段 |无此含义字段 |无此含义字段 | -|华为云存储 |key |metadata.etag |metadata.contentLength|lastModified 转换而来|metadata.contentType |storageClass |无此含义字段 |metadata.contentMd5|Owner.id | -|百度云存储 |key |etag |size |lastModified 转换而来 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName | +文件信息规范字段默认根据七牛存储文件的信息字段进行定义，其释义及各数据源方式对应关系如下： + +|数据源 |key（文件名） |etag（文件唯一值）|size（文件大小 kb）|datetime（日期时间字符串）|mime（content-type）|type（资源存储类型）|status（资源状态） |md5（文件 md5）|owner（终端标识符）| +|-----------|---------------|---------------|-----------------|----------------------|-------------------|-----------------|-----------------|-------------|----------------| +|文件列表输入行|默认 0 或 "key" |默认 1 或 "etag"|默认 2 或 "size" |默认 3 或 "datetime" |默认 4 或 "mime" |默认 5 或 "type" |默认 6 或 "status"|默认 7 或 "md5"|默认 8 或 "owner"| +|本地文件 |key |etag |length |timestamp 转换 |mime |无此含义字段 |无此含义字段 |无此含义字段 |无此含义字段 | +|七牛云存储 |key |eatg（hash） |fsize |putTime 转换 |mimeType |type |status |md5 |endUser | +|腾讯云存储 |key |etag |size |lastModified 转换 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName| +|阿里云存储 |key |etag |size |lastModified 转换 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName| +|AWS云存储/S3|key |etag |size |lastModified 转换 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName| +|又拍云存储 |name |无此含义字段 |length |lastModified 转换 |type/attribute |无此含义字段 |无此含义字段 |无此含义字段 |无此含义字段 | +|华为云存储 |key |metadata.etag |metadata.contentLength|lastModified 转换 |metadata.contentType|storageClass |无此含义字段 |metadata.contentMd5|Owner.id | +|百度云存储 |key |etag |size |lastModified 转换 |无此含义字段 |storageClass |无此含义字段 |无此含义字段 |Owner.displayName| ### 2 file 本地文件读取 本地文件数据源分为**两种情况：（1）读取文件内容为数据列表按行输入（2）读取路径下的文件本身，包括目录遍历，得到文件信息作为输入** @@ -125,6 +125,25 @@ uri-config= |uris| 字符串|数据源路径下需要读取的文件名列表，如果只想处理部分文件，可使用参数设置列表的方式，以 `,` 号分割文件名，不设置默认读取 path 下全部文本文件| |uri-config| 配置文件路径|表示从该配置中读取文件名作为 file 数据源，同时文件名对应的值表示读取该文件的起始位置，配置文件格式为 json，可参考[ uri-config 配置](#关于-uri-config)| +#### 关于 parse +数据源输入的文件列表为每行一条数据，parse 决定了数据的解析方式以及索引的设置方式，以下做简单举例说明，关于具体的其他索引设置参考对应 process 的文档。 +如文件列表为 \t 分割的行： +``` +IMG_0967.MOV llO8y0gW4GULwF1ry_UcvkA1HXYZ 22257875 2019-01-30T20:15:14.189789900 video/quicktime 0 0 e69957f45a290c0f10b51642b660e6df +IMG_0967.MOVF720.mp4 FoWR3PSyOkXprvMVbQM9SmhDojlO 645768 2019-08-06T18:44:37.193583700 video/mp4 0 0 60618eda5846e623c3cf9028928393bc +IMG_0967F720.mp4 FoWR3PSyOkXprvMVbQM9SmhDojlO 645768 2019-08-06T19:11:13.697041 video/mp4 0 0 60618eda5846e623c3cf9028928393bc +... +``` +则应设置 `parse=tab`，当然 parse=tab 为默认值可以省略，同时可以设置 `indexes=pre-8` 或者 `indexes=0,1,2` 等等，默认值为 `indexes=0`， +或者如在做 copy/move/rename 操作时可以设置 `toKey-index=1`。 +再如文件列表为 json 字符串的行： +``` +{"key":"data/2015/02/01.mp4","id":"NjwAAKG4Gu6bffQV","fops":"avthumb/mp4"} +{"key":"data/2015/02/02.mp4","id":"dUIAAACkEPwNdfQV","fops":"avthumb/mp4"} +``` +则需设置 `parse=json`，indexes 默认值为 `indexes=key`，在做有关 id 的查询时可以 `id-index=id`，或者做 pfop 转码时可以是 `fops-index=fops`， +当然 "id"、"fops" 其实均是对应操作下可以省略的默认值。 + #### 关于 uri-config uri-config 用来设置要读取的文件路径，在 path 为空的情况下，uri-config 中的文件名必须是完整的路径名，path 为目录时，uri-config 中的文件名可 以采取相对 path 路径下的文件名，因此 uri-config 中的文件名必须存在。配置中每一个文件源对应的值表示在一行文本信息，在实际读取数据源过程中，会参照 diff --git a/docs/downloadfile.md b/docs/downloadfile.md index 2c0cf50f..1e9a6280 100644 --- a/docs/downloadfile.md +++ b/docs/downloadfile.md @@ -47,7 +47,8 @@ timeout 参数可以通过全局的 timeout 来设置，参考：[网络设置]( ### 关于 url-index 当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用 -key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 ### 资源需要私有签名 当进行图片审核的 url 需要通过私有鉴权访问时（资源来自于存储私有权限的空间），本工具支持串联操作，即先进行对应的私有签名再提交审核，使用如下的 private diff --git a/docs/exportts.md b/docs/exportts.md index 7b6f9e0f..c91daab8 100644 --- a/docs/exportts.md +++ b/docs/exportts.md @@ -24,7 +24,8 @@ url-index= ### 关于 url-index 当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用 -key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 ### 命令行方式 ``` diff --git a/docs/fetch.md b/docs/fetch.md index 9b9461cf..b61c2601 100644 --- a/docs/fetch.md +++ b/docs/fetch.md @@ -39,7 +39,8 @@ check= ### 关于 url-index 当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用 -key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 ### 命令行参数方式 ``` diff --git a/docs/mime.md b/docs/mime.md index b41a192a..298c460a 100644 --- a/docs/mime.md +++ b/docs/mime.md @@ -27,7 +27,7 @@ indexes= ### 关于 indexes 这里的 indexes 表示取文件名的索引配置，非存储数据源的情况下默认只会包含 key 的索引，会根据 parse 类型设置为 0 或 "key"，如果需要从每一行读取 -mime 用于资源的类型修改，则需要设置 mime 的索引，如 `indexes=0,-1,-1,-1,mime`，具体参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +mime 用于资源的类型修改，则需要设置 mime 的索引，如 `indexes=0,-1,-1,-1,mime`，具体参见[ indexes 索引](datasource.md#关于-indexes-索引)及[关于 parse 和索引](datasource.md#关于-parse)。 ### 命令行参数方式 ``` diff --git a/docs/move.md b/docs/move.md index 5038dfa3..3fe802d9 100644 --- a/docs/move.md +++ b/docs/move.md @@ -31,7 +31,8 @@ rm-prefix= ### 关于 toKey-index 指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，但未设置且 add-prefix 不为空 时需要强制指定 prefix-force=true，表明该次重命名操作只添加文件名前缀。由于 rename 操作既需要原始文件名字段也需要新文件名字段，因此 toKey 下 -标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 **注意**：七牛存储空间不支持文件名以 `../`, `./` 开头或者包含 `/../`, `/./` 这种情况，会造成无法访问，因此设置文件名时请注意。 ### 命令行方式 diff --git a/docs/pfop.md b/docs/pfop.md index 4c6e45cc..984e22a9 100644 --- a/docs/pfop.md +++ b/docs/pfop.md @@ -40,7 +40,7 @@ pfop-config= ### 关于 fops-index 指定输入行中对应转码的命令字段下标，不设置为则无法进行解析。由于转码必须参数包含 key 和 fops，因此输入行中也必须包含 key 字段的值，使用 indexes -参数设置 key 下标，同时 key 下标不能与 fops 下标相同。 +参数设置 key 下标，同时 key 下标不能与 fops 下标相同，参见[ indexes 索引](datasource.md#关于-indexes-索引)及[关于 parse 和索引](datasource.md#关于-parse)。 ### pfop-config 配置文件 ```json diff --git a/docs/pfopcmd.md b/docs/pfopcmd.md index 25c6227b..bf579912 100644 --- a/docs/pfopcmd.md +++ b/docs/pfopcmd.md @@ -26,7 +26,7 @@ combine=true/false |process=pfopcmd| 该操作设置为pfopcmd| 表示根据 avinfo 生成音视频转码指令| |indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| |pfop-config| 文件路径字符串| 进行转码和另存规则设置的 json 配置文件路径，可设置多个转码条件和指令，[配置写法](#pfop-config-配置文件)| -|avinfo-index| 字符串| 读取 avinfo 信息时需要设置的 avinfo 字符串索引（下标），未设置任何索引时根据 parse 类型默认为 1 或 "avinfo"| +|avinfo-index| 字符串| 读取 avinfo 信息时需要设置的 avinfo 字符串索引（下标），未设置任何索引时根据 parse 类型默认为 1 或 "avinfo"，参见[关于 parse 和索引](datasource.md#关于-parse)| |has-duration| true/false| 得到的结果行中是否需要保存 duration（音视频时长）信息，会放在转码指令字段之后 | |has-size| true/false| 得到的结果行中是否需要保存 size（音视频大小）信息，会放在 duration 字段之后| |combine| true/false| 多个 pfop 命令时，是否组合成一个命令来处理，默认为 true，使用 `;` 来拼接多条处理命令，参考[persistentOps 多个命令](https://developer.qiniu.com/dora/api/3686/pfop-directions-for-use#1)| diff --git a/docs/pfopresult.md b/docs/pfopresult.md index 3db6aa13..79e356a2 100644 --- a/docs/pfopresult.md +++ b/docs/pfopresult.md @@ -30,4 +30,4 @@ id-index= ``` ## 备注 -pfopresult 操作是 file 源下的操作，从 every line of fileinput 的 id-index 索引获取 persistentId。 +pfopresult 操作是 file 源下的操作，从 every line of fileinput 的 id-index 索引获取 persistentId，参见[关于 parse 和索引](datasource.md#关于-parse)。 diff --git a/docs/privateurl.md b/docs/privateurl.md index 38894a4e..3eaa31e2 100644 --- a/docs/privateurl.md +++ b/docs/privateurl.md @@ -40,7 +40,8 @@ queries= #### 关于 url-index 当 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行中存在 url 形式的源文件地址，未设置的情况下则默认从 key 字段 -加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 #### 关于 queries 参数 queries 参数用于设置 ?+参数部分（或 url 的后缀），希望在私有签名之前加上参数。 diff --git a/docs/publicurl.md b/docs/publicurl.md index bd2d1586..97f17242 100644 --- a/docs/publicurl.md +++ b/docs/publicurl.md @@ -26,7 +26,7 @@ queries= |queries| 字符串| url 的 query 参数或样式后缀，如 `?v=1.1&time=1565171107845` 或 `-w480`，[关于 queries 参数](#关于-queries-参数)| #### 关于 key -key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)及[关于 parse 和索引](datasource.md#关于-parse)。 #### 关于 queries 参数 queries 参数用于设置 ?+参数部分（或 url 的后缀），希望在 url 上加上参数。 diff --git a/docs/qhash.md b/docs/qhash.md index 5f1bb18a..d804a96c 100644 --- a/docs/qhash.md +++ b/docs/qhash.md @@ -26,7 +26,8 @@ algorithm= ### 关于 url-index 当 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行中存在 url 形式的源文件地址，未设置的情况下则默认从 key 字段 -加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 ### 命令行方式 ``` diff --git a/docs/rename.md b/docs/rename.md index 7b675552..9c495f93 100644 --- a/docs/rename.md +++ b/docs/rename.md @@ -33,7 +33,8 @@ prefix-force= ### 关于 toKey-index 指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，但未设置且 add-prefix 不为空 时需要强制指定 prefix-force=true，表明该次重命名操作只添加文件名前缀。由于 rename 操作既需要原始文件名字段也需要新文件名字段，因此 toKey 下 -标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 **注意**：七牛存储空间不支持文件名以 `../`, `./` 开头或者包含 `/../`, `/./` 这种情况，会造成无法访问，因此设置文件名时请注意。 ### 命令行方式 diff --git a/docs/syncupload.md b/docs/syncupload.md index 8aed8a3c..aab59ce9 100644 --- a/docs/syncupload.md +++ b/docs/syncupload.md @@ -1,7 +1,7 @@ -# 通过 url 同步上传 +# 通过 url 流式同步上传 ## 简介 -将 url 的内容同步上传至存储空间，同时依据原 url 设置目标文件名。上传可参考：[上传策略](https://developer.qiniu.com/kodo/manual/1206/put-policy) +将 url 的内容通过字节流同步上传至存储空间，同时依据原 url 设置目标文件名。上传可参考：[上传策略](https://developer.qiniu.com/kodo/manual/1206/put-policy) 和 [流式上传资源](https://developer.qiniu.com/kodo/sdk/1239/java#upload-stream) 1. **操作需要指定数据源，上传 url 的内容到七牛空间，故需要配置数据源，参考：[配置数据源](datasource.md)** 2. 支持通过 `-a=`/`-d` 使用已设置的账号，则不需要再直接设置密钥，参考：[账号设置](../README.md#账号设置) @@ -53,7 +53,8 @@ timeout 参数可以通过全局的 timeout 来设置，参考：[网络设置]( ### 关于 url-index 当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用 -key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +key 字段加上 domain 的方式访问源文件地址，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 ### 资源需要私有签名 当进行图片审核的 url 需要通过私有鉴权访问时（资源来自于存储私有权限的空间），本工具支持串联操作，即先进行对应的私有签名再提交审核，使用如下的 private diff --git a/docs/uploadfile.md b/docs/uploadfile.md index 9e977830..7197b14b 100644 --- a/docs/uploadfile.md +++ b/docs/uploadfile.md @@ -117,7 +117,8 @@ indexes=1 ### 关于 filepath-index 当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。filepath-index 表示输入行含 filepath 形式的文件路径，未设置的情 -况下则使用 key 字段加上 parent-path 的方式访问文件路径，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)。 +况下则使用 key 字段加上 parent-path 的方式访问文件路径，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) +及[关于 parse 和索引](datasource.md#关于-parse)。 ### 命令行参数方式 ``` diff --git a/pom.xml b/pom.xml index 7db61f5e..d46bb02e 100755 --- a/pom.xml +++ b/pom.xml @@ -6,7 +6,7 @@ com.qiniu qsuits - 8.4.0 + 8.4.1 qsuits qiniu-suits is a efficient tools for qiniu api implemented by java8. https://github.com/NigelWu95/qiniu-suits-java diff --git a/src/main/java/com/qiniu/entry/QSuitsEntry.java b/src/main/java/com/qiniu/entry/QSuitsEntry.java index f4d5dfab..7cc66710 100755 --- a/src/main/java/com/qiniu/entry/QSuitsEntry.java +++ b/src/main/java/com/qiniu/entry/QSuitsEntry.java @@ -761,9 +761,10 @@ private ILineProcess> getPublicUrl(boolean single) throws IO String protocol = entryParam.getValue("protocol", "http").trim(); ParamsUtils.checked(protocol, "protocol", "https?"); String domain = entryParam.getValue("domain", "").trim(); + String urlIndex = indexMap.containsValue("url") ? "url" : null; String queries = entryParam.getValue("queries", "").trim(); - return single ? new PublicUrl(qiniuAccessKey, qiniuSecretKey, protocol, domain, queries) - : new PublicUrl(qiniuAccessKey, qiniuSecretKey, protocol, domain, queries, savePath); + return single ? new PublicUrl(qiniuAccessKey, qiniuSecretKey, protocol, domain, urlIndex, queries) + : new PublicUrl(qiniuAccessKey, qiniuSecretKey, protocol, domain, urlIndex, queries, savePath); } private ILineProcess> getMirrorFile(boolean single) throws IOException { diff --git a/src/main/java/com/qiniu/process/other/PublicUrl.java b/src/main/java/com/qiniu/process/other/PublicUrl.java index 5bf13aa1..5f7c1c18 100755 --- a/src/main/java/com/qiniu/process/other/PublicUrl.java +++ b/src/main/java/com/qiniu/process/other/PublicUrl.java @@ -11,34 +11,42 @@ public class PublicUrl extends Base> { private String protocol; private String domain; + private String urlIndex; private String suffixOrQuery; private ILineProcess> nextProcessor; - public PublicUrl(String accessKey, String secretKey, String protocol, String domain, String suffixOrQuery) throws IOException { + public PublicUrl(String accessKey, String secretKey, String protocol, String domain, String urlIndex, String suffixOrQuery) + throws IOException { super("publicurl", accessKey, secretKey, null); - set(protocol, domain, suffixOrQuery); + set(protocol, domain, urlIndex, suffixOrQuery); } - public PublicUrl(String accessKey, String secretKey, String protocol, String domain, String suffixOrQuery, String savePath, - int saveIndex) throws IOException { + public PublicUrl(String accessKey, String secretKey, String protocol, String domain, String urlIndex, String suffixOrQuery, + String savePath, int saveIndex) throws IOException { super("publicurl", accessKey, secretKey, null, savePath, saveIndex); - set(protocol, domain, suffixOrQuery); + set(protocol, domain, urlIndex, suffixOrQuery); } - public PublicUrl(String accessKey, String secretKey, String protocol, String domain, String suffixOrQuery, String savePath) - throws IOException { - this(accessKey, secretKey, protocol, domain, suffixOrQuery, savePath, 0); + public PublicUrl(String accessKey, String secretKey, String protocol, String domain, String urlIndex, String suffixOrQuery, + String savePath) throws IOException { + this(accessKey, secretKey, protocol, domain, urlIndex, suffixOrQuery, savePath, 0); } - private void set(String protocol, String domain, String suffixOrQuery) throws IOException { + private void set(String protocol, String domain, String urlIndex, String suffixOrQuery) throws IOException { + this.suffixOrQuery = suffixOrQuery == null ? "" : suffixOrQuery; if (domain == null || "".equals(domain)) { - throw new IOException("please set one of domain and url-index."); + if (urlIndex == null || "".equals(urlIndex)) { + throw new IOException("please set one of domain and url-index."); + } else { + this.urlIndex = urlIndex; + if ("".equals(this.suffixOrQuery)) throw new IOException("please set suffix or query if url-index used."); + } } else { this.protocol = protocol == null || !protocol.matches("(http|https)") ? "http" : protocol; RequestUtils.lookUpFirstIpFromHost(domain); this.domain = domain; + this.urlIndex = "url"; } - this.suffixOrQuery = suffixOrQuery == null ? "" : suffixOrQuery; } @Override @@ -61,15 +69,19 @@ protected String resultInfo(Map line) { @Override protected String singleResult(Map line) throws Exception { - String key = line.get("key"); - if (key == null) throw new IOException("key is not exists or empty in " + line); - String url = String.join("", protocol, "://", domain, "/", - key.replace("\\?", "%3f"), suffixOrQuery); - if (nextProcessor != null) { - line.put("url", url); - return nextProcessor.processLine(line); + String url; + if (domain == null) { + url = line.get(urlIndex); + if (nextProcessor == null) return url + suffixOrQuery; + } else { + String key = line.get("key"); + if (key == null) throw new IOException("key is not exists or empty in " + line); + url = String.join("", protocol, "://", domain, "/", + key.replace("\\?", "%3f"), suffixOrQuery); + if (nextProcessor == null) return url; } - return url; + line.put("url", url); + return nextProcessor.processLine(line); } @Override diff --git a/src/main/java/com/qiniu/process/qiniu/PrivateUrl.java b/src/main/java/com/qiniu/process/qiniu/PrivateUrl.java index 7a2459fa..fc7c93fe 100755 --- a/src/main/java/com/qiniu/process/qiniu/PrivateUrl.java +++ b/src/main/java/com/qiniu/process/qiniu/PrivateUrl.java @@ -79,14 +79,14 @@ protected String singleResult(Map line) throws Exception { String url; if (domain == null) { url = line.get(urlIndex); - if (nextProcessor == null) return String.join("\t", url, auth + if (nextProcessor == null) return String.join("\t", auth .privateDownloadUrl(url + suffixOrQuery, expires)); } else { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); url = String.join("", protocol, "://", domain, "/", key.replace("\\?", "%3f"), suffixOrQuery); - if (nextProcessor == null) return String.join("\t", key, auth.privateDownloadUrl(url, expires)); + if (nextProcessor == null) return String.join("\t", auth.privateDownloadUrl(url, expires)); } url = auth.privateDownloadUrl(url, expires); line.put("url", auth.privateDownloadUrl(url, expires)); diff --git a/src/main/java/com/qiniu/process/qiniu/SyncUpload.java b/src/main/java/com/qiniu/process/qiniu/SyncUpload.java index 05ae4aef..8824e299 100755 --- a/src/main/java/com/qiniu/process/qiniu/SyncUpload.java +++ b/src/main/java/com/qiniu/process/qiniu/SyncUpload.java @@ -128,7 +128,7 @@ protected String singleResult(Map line) throws Exception { if (response.statusCode == 200 || response.statusCode == 206) { try { InputStream stream = response.bodyStream(); - return String.join("\t", url, HttpRespUtils.getResult(uploadManager.put(stream, key, + return String.join("\t", key, url, HttpRespUtils.getResult(uploadManager.put(stream, key, auth.uploadToken(bucket, key, expires, policy), params, response.contentType()))); } finally { response.close(); diff --git a/src/main/java/com/qiniu/process/qiniu/UploadFile.java b/src/main/java/com/qiniu/process/qiniu/UploadFile.java index 2fc5a2e5..73fc466d 100755 --- a/src/main/java/com/qiniu/process/qiniu/UploadFile.java +++ b/src/main/java/com/qiniu/process/qiniu/UploadFile.java @@ -159,10 +159,10 @@ protected String singleResult(Map line) throws Exception { key = String.join("", addPrefix, FileUtils.rmPrefix(rmPrefix, key)); if (iFileChecker.check(key) != null) throw new IOException("file exists"); if (filepath.endsWith(FileUtils.pathSeparator)) { - return String.join("\t", filepath, HttpRespUtils.getResult(uploadManager.put(new byte[]{}, key, + return String.join("\t", key, filepath, HttpRespUtils.getResult(uploadManager.put(new byte[]{}, key, auth.uploadToken(bucket, key, expires, policy), params, null, checkCrc))); } else { - return String.join("\t", filepath, HttpRespUtils.getResult(uploadManager.put(filepath, key, + return String.join("\t", key, filepath, HttpRespUtils.getResult(uploadManager.put(filepath, key, auth.uploadToken(bucket, key, expires, policy), params, null, checkCrc))); } } diff --git a/src/main/java/com/qiniu/util/ProcessUtils.java b/src/main/java/com/qiniu/util/ProcessUtils.java index 30808483..b4491c20 100755 --- a/src/main/java/com/qiniu/util/ProcessUtils.java +++ b/src/main/java/com/qiniu/util/ProcessUtils.java @@ -8,6 +8,7 @@ public final class ProcessUtils { public static Set needUrlProcesses = new HashSet(){{ add("asyncfetch"); add("privateurl"); + add("publicurl"); add("qhash"); add("avinfo"); add("exportts"); @@ -127,6 +128,7 @@ public final class ProcessUtils { add("asyncfetch"); add("syncupload"); add("privateurl"); + add("publicurl"); add("imagecensor"); add("videocensor"); add("censorresult"); @@ -134,7 +136,6 @@ public final class ProcessUtils { add("cdnprefetch"); add("refreshquery"); add("prefetchquery"); - add("publicurl"); }}; public static Set dangerousProcesses = new HashSet(){{ add("status"); diff --git a/version.properties b/version.properties index bbbbedc7..60455a7c 100644 --- a/version.properties +++ b/version.properties @@ -1 +1 @@ -version=8.4.0 \ No newline at end of file +version=8.4.1 \ No newline at end of file From 907abbd706c27bfa65a58aacce772cb6c827a1ba Mon Sep 17 00:00:00 2001 From: wubingheng Date: Wed, 19 Feb 2020 11:28:44 +0800 Subject: [PATCH 02/12] optimize some processors' return and optimize process Base's exception handling. And add RestoreArchive processor to restore archival file. --- docs/publicurl.md | 3 +- src/main/java/com/qiniu/process/Base.java | 31 ++-- .../com/qiniu/process/qiniu/AsyncFetch.java | 5 +- .../com/qiniu/process/qiniu/CopyFile.java | 2 +- .../com/qiniu/process/qiniu/FetchFile.java | 3 +- .../com/qiniu/process/qiniu/MoveFile.java | 2 +- .../qiniu/process/qiniu/RestoreArchive.java | 133 ++++++++++++++++++ .../java/com/qiniu/util/ProcessUtils.java | 5 +- .../process/qiniu/RestoreArchiveTest.java | 34 +++++ 9 files changed, 194 insertions(+), 24 deletions(-) create mode 100755 src/main/java/com/qiniu/process/qiniu/RestoreArchive.java create mode 100644 src/test/java/com/qiniu/process/qiniu/RestoreArchiveTest.java diff --git a/docs/publicurl.md b/docs/publicurl.md index 97f17242..229ae239 100644 --- a/docs/publicurl.md +++ b/docs/publicurl.md @@ -14,8 +14,8 @@ sk= protocol= domain= indexes= -expires= queries= +url-index= ``` |参数名|参数值及类型 | 含义| |-----|-------|-----| @@ -24,6 +24,7 @@ queries= |domain| 域名字符串| 用于拼接文件名生成链接的域名（七牛存储空间域名可以使用[ domainsfrom 命令查询](domainsofbucket.md)），当指定 url-index 时无需设置| |indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| |queries| 字符串| url 的 query 参数或样式后缀，如 `?v=1.1&time=1565171107845` 或 `-w480`，[关于 queries 参数](#关于-queries-参数)| +|url-index| 字符串| 通过已有的 url 来添加 queries 时需要设置的 url 索引（下标），未设置任何索引和 domain 时根据 parse 类型默认为 0 或 "url"| #### 关于 key key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)及[关于 parse 和索引](datasource.md#关于-parse)。 diff --git a/src/main/java/com/qiniu/process/Base.java b/src/main/java/com/qiniu/process/Base.java index 68c5d5d6..ce048ae5 100755 --- a/src/main/java/com/qiniu/process/Base.java +++ b/src/main/java/com/qiniu/process/Base.java @@ -149,14 +149,16 @@ protected List parseBatchResult(List processList, String result) throws Ex jsonObject = jsonArray.get(j).getAsJsonObject(); switch (HttpRespUtils.checkStatusCode(jsonObject.get("code").getAsInt())) { case 1: - fileSaveMapper.writeSuccess(resultInfo(processList.get(j)) + "\t" + jsonObject, false); + fileSaveMapper.writeSuccess(String.join("\t", resultInfo(processList.get(j)), + jsonObject.toString()), false); break; case 0: if (retryList == null) retryList = new ArrayList<>(); retryList.add(processList.get(j)); // 放回重试列表 break; case -1: - fileSaveMapper.writeError(resultInfo(processList.get(j)) + "\t" + jsonObject, false); + fileSaveMapper.writeError(String.join("\t", resultInfo(processList.get(j)), + jsonObject.toString()), false); break; } } @@ -199,15 +201,15 @@ private void batchProcess(List lineList, int batchSize, int retryTimes) throw exception = null; } switch (retry) { - case 0: fileSaveMapper.writeError(String.join("\n", processList.stream() - .map(this::resultInfo).collect(Collectors.toList())) + "\t" + message, false); + case 0: fileSaveMapper.writeError(String.join("\t", processList.stream() + .map(this::resultInfo).collect(Collectors.joining("\n")), message), false); processList = null; break; - case -1: fileSaveMapper.writeToKey("need_retry", String.join("\n", processList - .stream().map(this::resultInfo).collect(Collectors.toList())) + "\t" + message, false); + case -1: fileSaveMapper.writeToKey("need_retry", String.join("\t", processList + .stream().map(this::resultInfo).collect(Collectors.joining("\n")), message), false); processList = null; break; - case -2: fileSaveMapper.writeError(String.join("\n", lineList - .subList(batchSize * i, lineList.size()).stream() - .map(this::resultInfo).collect(Collectors.toList())) + "\t" + message, false); + case -2: fileSaveMapper.writeError(String.join("\t", lineList + .subList(batchSize * i, lineList.size()).stream().map(this::resultInfo) + .collect(Collectors.joining("\n")), message), false); if (exception != null) throw exception; } if (exception != null && exception.response != null) exception.response.close(); @@ -262,11 +264,12 @@ private void singleProcess(List lineList, int retryTimes) throws IOException exception = null; } switch (retry) { - case 0: fileSaveMapper.writeError(resultInfo(line) + "\t" + message, false); break; - case -1: fileSaveMapper.writeToKey("need_retry", resultInfo(line) + "\t" + message, false); - break; - case -2: fileSaveMapper.writeError(String.join("\n", lineList.subList(i, lineList.size()) - .stream().map(this::resultInfo).collect(Collectors.toList())) + "\t" + message, false); + case 0: fileSaveMapper.writeError(String.join("\t", resultInfo(line), message), + false); break; + case -1: fileSaveMapper.writeToKey("need_retry", String.join("\t", resultInfo(line), + message), false); break; + case -2: fileSaveMapper.writeError(String.join("\t", lineList.subList(i, lineList.size()) + .stream().map(this::resultInfo).collect(Collectors.joining("\n")), message), false); throw exception; } if (exception != null && exception.response != null) exception.response.close(); diff --git a/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java b/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java index 2bcbb9ff..fc026c09 100755 --- a/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java +++ b/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java @@ -32,7 +32,7 @@ public class AsyncFetch extends Base> { private Configuration configuration; private Auth auth; private Client client; - private String requestUrl; + private static String requestUrl; // private BucketManager bucketManager; public AsyncFetch(String accessKey, String secretKey, Configuration configuration, String bucket, String protocol, @@ -41,7 +41,6 @@ public AsyncFetch(String accessKey, String secretKey, Configuration configuratio this.auth = Auth.create(accessKey, secretKey); this.client = new Client(configuration.clone()); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); - this.requestUrl = configuration.apiHost(auth.accessKey, bucket) + "/sisyphus/fetch"; // this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); // CloudApiUtils.checkQiniu(bucketManager, bucket); set(configuration, protocol, domain, urlIndex, addPrefix, rmPrefix); @@ -54,7 +53,6 @@ public AsyncFetch(String accessKey, String secretKey, Configuration configuratio this.auth = Auth.create(accessKey, secretKey); this.client = new Client(configuration.clone()); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); - this.requestUrl = configuration.apiHost(auth.accessKey, bucket) + "/sisyphus/fetch"; // this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); // CloudApiUtils.checkQiniu(bucketManager, bucket); set(configuration, protocol, domain, urlIndex, addPrefix, rmPrefix); @@ -83,6 +81,7 @@ private void set(Configuration configuration, String protocol, String domain, St } this.addPrefix = addPrefix == null ? "" : addPrefix; this.rmPrefix = rmPrefix; + requestUrl = configuration.apiHost(auth.accessKey, bucket) + "/sisyphus/fetch"; } public void setFetchArgs(String host, String md5Index, String callbackUrl, String callbackBody, diff --git a/src/main/java/com/qiniu/process/qiniu/CopyFile.java b/src/main/java/com/qiniu/process/qiniu/CopyFile.java index ef9b04ff..050ed061 100755 --- a/src/main/java/com/qiniu/process/qiniu/CopyFile.java +++ b/src/main/java/com/qiniu/process/qiniu/CopyFile.java @@ -121,7 +121,7 @@ protected String batchResult(List> lineList) throws IOExcept protected String singleResult(Map line) throws IOException { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); - String toKey = String.join("", addPrefix, FileUtils.rmPrefix(rmPrefix, line.get(toKeyIndex))); + String toKey = addPrefix + FileUtils.rmPrefix(rmPrefix, defaultToKey ? key : line.get(toKeyIndex)); Response response = bucketManager.copy(bucket, key, toBucket, toKey, false); if (response.statusCode != 200) throw new QiniuException(response); response.close(); diff --git a/src/main/java/com/qiniu/process/qiniu/FetchFile.java b/src/main/java/com/qiniu/process/qiniu/FetchFile.java index 1c640ddb..6ab84e40 100755 --- a/src/main/java/com/qiniu/process/qiniu/FetchFile.java +++ b/src/main/java/com/qiniu/process/qiniu/FetchFile.java @@ -95,8 +95,7 @@ protected String singleResult(Map line) throws IOException { } // String check = iFileChecker.check(key); if (iFileChecker.check(key) != null) throw new IOException("file exists"); - Response response = bucketManager.fetchResponse(url, bucket, key); - return String.join("\t", key, url, String.valueOf(response.statusCode), HttpRespUtils.getResult(response)); + return HttpRespUtils.getResult(bucketManager.fetchResponse(url, bucket, key)); } @Override diff --git a/src/main/java/com/qiniu/process/qiniu/MoveFile.java b/src/main/java/com/qiniu/process/qiniu/MoveFile.java index 1de9b937..6be8c340 100755 --- a/src/main/java/com/qiniu/process/qiniu/MoveFile.java +++ b/src/main/java/com/qiniu/process/qiniu/MoveFile.java @@ -143,7 +143,7 @@ protected String batchResult(List> lineList) throws IOExcept protected String singleResult(Map line) throws IOException { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); - String toKey = addPrefix + FileUtils.rmPrefix(rmPrefix, line.get(toKeyIndex)); + String toKey = addPrefix + FileUtils.rmPrefix(rmPrefix, defaultToKey ? key : line.get(toKeyIndex)); if (isRename) { Response response = bucketManager.rename(bucket, key, toKey); if (response.statusCode != 200) throw new QiniuException(response); diff --git a/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java b/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java new file mode 100755 index 00000000..98dc6b73 --- /dev/null +++ b/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java @@ -0,0 +1,133 @@ +package com.qiniu.process.qiniu; + +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.qiniu.common.QiniuException; +import com.qiniu.http.Client; +import com.qiniu.http.Response; +import com.qiniu.process.Base; +import com.qiniu.storage.Configuration; +import com.qiniu.util.*; + +import java.io.IOException; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +public class RestoreArchive extends Base> { + + private int days; + private String condition; + private Configuration configuration; + private Auth auth; + private Client client; + private static String requestUrl; + + public RestoreArchive(String accessKey, String secretKey, Configuration configuration, String bucket, int days, + String condition) throws IOException { + super("restorear", accessKey, secretKey, bucket); + this.days = days; + this.condition = condition; + this.configuration = configuration; + this.auth = Auth.create(accessKey, secretKey); + this.client = new Client(configuration.clone()); + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); + requestUrl = configuration.useHttpsDomains ? "https://rs.qbox.me/restoreAr" : "http://rs.qbox.me/restoreAr"; + } + + public RestoreArchive(String accessKey, String secretKey, Configuration configuration, String bucket, int days, + String condition, String savePath, int saveIndex) throws IOException { + super("restorear", accessKey, secretKey, bucket, savePath, saveIndex); + this.days = days; + this.condition = condition; + this.configuration = configuration; + this.auth = Auth.create(accessKey, secretKey); + this.client = new Client(configuration.clone()); + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); + requestUrl = configuration.useHttpsDomains ? "https://rs.qbox.me/restoreAr" : "http://rs.qbox.me/restoreAr"; + } + + public RestoreArchive(String accessKey, String secretKey, Configuration configuration, String bucket, int days, + String condition, String savePath) throws IOException { + this(accessKey, secretKey, configuration, bucket, days, condition, savePath, 0); + } + + @Override + public RestoreArchive clone() throws CloneNotSupportedException { + RestoreArchive restoreArchive = (RestoreArchive)super.clone(); + restoreArchive.auth = Auth.create(accessId, secretKey); + restoreArchive.client = new Client(configuration.clone()); + return restoreArchive; + } + + @Override + protected String resultInfo(Map line) { + return line.get("key"); + } + + @Override + protected String batchResult(List> lineList) throws IOException { + JsonObject bodyJson = new JsonObject(); + JsonArray entries = new JsonArray(); + JsonObject entry = new JsonObject(); + entry.addProperty("bucket", bucket); + for (Map line : lineList) { + entry.addProperty("key", line.get("key")); + entry.addProperty("freeze_after_days", days); + entry.addProperty("cond", condition); + entries.add(entry); + } + bodyJson.add("entries", entries); + byte[] body = bodyJson.toString().getBytes(); + bodyJson = null; + entries = null; + entry = null; + return HttpRespUtils.getResult(client.post(requestUrl, body, + auth.authorizationV2(requestUrl, "POST", body, Client.JsonMime), Client.JsonMime)); + } + + @Override + protected List> parseBatchResult(List> processList, String result) throws Exception { + // 归档存储解冻操作本身不响应任何 body 内容 +// if (result == null || "".equals(result)) throw new IOException("not valid refresh response."); + if ("".equals(result)) { + fileSaveMapper.writeSuccess(processList.stream().map(this::resultInfo).collect(Collectors.joining("\n")), false); + return null; + } else { + return super.parseBatchResult(processList, result); + } + } + + @Override + protected String singleResult(Map line) throws Exception { + JsonObject bodyJson = new JsonObject(); + JsonArray entries = new JsonArray(); + JsonObject entry = new JsonObject(); + entry.addProperty("bucket", bucket); + String key = line.get("key"); + entry.addProperty("key", key); + entry.addProperty("freeze_after_days", days); + entry.addProperty("cond", condition); + entries.add(entry); + bodyJson.add("entries", entries); + byte[] body = bodyJson.toString().getBytes(); + Response response = client.post(requestUrl, body, + auth.authorizationV2(requestUrl, "POST", body, Client.JsonMime), Client.JsonMime); + if (response.statusCode != 200) throw new QiniuException(response); + response.close(); + bodyJson = null; + entries = null; + entry = null; + body = null; + return String.join("\t", key, "200"); + } + + @Override + public void closeResource() { + super.closeResource(); + condition = null; + configuration = null; + auth = null; + client = null; + } +} diff --git a/src/main/java/com/qiniu/util/ProcessUtils.java b/src/main/java/com/qiniu/util/ProcessUtils.java index b4491c20..2fdfee96 100755 --- a/src/main/java/com/qiniu/util/ProcessUtils.java +++ b/src/main/java/com/qiniu/util/ProcessUtils.java @@ -49,6 +49,7 @@ public final class ProcessUtils { public static Set qiniuProcessesWithBucket = new HashSet(){{ add("status"); add("type"); + add("restorear"); add("lifecycle"); add("delete"); add("copy"); @@ -59,10 +60,12 @@ public final class ProcessUtils { add("qupload"); add("mime"); add("metadata"); + add("mirror"); }}; public static Set canBatchProcesses = new HashSet(){{ add("status"); add("type"); + add("restorear"); add("lifecycle"); add("copy"); add("move"); @@ -102,7 +105,6 @@ public final class ProcessUtils { }}; public static Set needQiniuAuthProcesses = new HashSet(){{ addAll(qiniuProcessesWithBucket); - add("mirror"); add("fetch"); add("asyncfetch"); add("syncupload"); @@ -123,7 +125,6 @@ public final class ProcessUtils { add("exportts"); add("download"); add("filter"); - add("mirror"); add("fetch"); add("asyncfetch"); add("syncupload"); diff --git a/src/test/java/com/qiniu/process/qiniu/RestoreArchiveTest.java b/src/test/java/com/qiniu/process/qiniu/RestoreArchiveTest.java new file mode 100644 index 00000000..99cca919 --- /dev/null +++ b/src/test/java/com/qiniu/process/qiniu/RestoreArchiveTest.java @@ -0,0 +1,34 @@ +package com.qiniu.process.qiniu; + +import com.qiniu.config.PropertiesFile; +import com.qiniu.storage.Configuration; +import org.junit.Test; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.Assert.*; + +public class RestoreArchiveTest { + + @Test + public void testBatchResult() throws IOException { + PropertiesFile propertiesFile = new PropertiesFile("resources/.application.properties"); + String accessKey = propertiesFile.getValue("ak"); + String secretKey = propertiesFile.getValue("sk"); + String bucket = propertiesFile.getValue("bucket"); + RestoreArchive restoreArchive = new RestoreArchive(accessKey, secretKey, new Configuration(), bucket, 1, + null, "~/Downloads/restorear"); + List> list = new ArrayList<>(); + list.add(new HashMap(){{ put("key", "qiniu_success_1.txt"); }}); + list.add(new HashMap(){{ put("key", "qiniu_success_2.txt"); }}); + list.add(new HashMap(){{ put("key", "qiniu_success_3.txt"); }}); + list.add(new HashMap(){{ put("key", "qiniu_success_9.txt"); }}); + restoreArchive.setBatchSize(5); + restoreArchive.processLine(list); + restoreArchive.closeResource(); + } +} \ No newline at end of file From 0f0c75b541f4d48db932f860f68081c9de4e305f Mon Sep 17 00:00:00 2001 From: wubingheng Date: Wed, 19 Feb 2020 14:49:14 +0800 Subject: [PATCH 03/12] add RestoreArchive enable option in ChangeType process for archive file's type changing. --- .../com/qiniu/interfaces/IFileCaller.java | 9 +++++ .../com/qiniu/process/qiniu/ChangeType.java | 33 +++++++++++++++++-- .../com/qiniu/process/qiniu/FetchFile.java | 1 - .../qiniu/process/qiniu/RestoreArchive.java | 12 ++++++- 4 files changed, 50 insertions(+), 5 deletions(-) create mode 100644 src/main/java/com/qiniu/interfaces/IFileCaller.java diff --git a/src/main/java/com/qiniu/interfaces/IFileCaller.java b/src/main/java/com/qiniu/interfaces/IFileCaller.java new file mode 100644 index 00000000..9de7de12 --- /dev/null +++ b/src/main/java/com/qiniu/interfaces/IFileCaller.java @@ -0,0 +1,9 @@ +package com.qiniu.interfaces; + +import java.io.IOException; +import java.util.List; + +public interface IFileCaller { + + String call(List list) throws IOException; +} diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeType.java b/src/main/java/com/qiniu/process/qiniu/ChangeType.java index 7ff25935..cb9c6371 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeType.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeType.java @@ -2,6 +2,7 @@ import com.qiniu.common.QiniuException; import com.qiniu.http.Response; +import com.qiniu.interfaces.IFileCaller; import com.qiniu.process.Base; import com.qiniu.storage.BucketManager; import com.qiniu.storage.BucketManager.*; @@ -21,16 +22,23 @@ public class ChangeType extends Base> { private StorageType storageType; private BatchOperations batchOperations; private List> lines; + private Auth auth; private Configuration configuration; private BucketManager bucketManager; + private int days; + private String condition; + private RestoreArchive restoreArchive; + private IFileCaller> caller; public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type) throws IOException { super("type", accessKey, secretKey, bucket); storageType = type == 0 ? StorageType.COMMON : StorageType.INFREQUENCY; + this.auth = Auth.create(accessKey, secretKey); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(auth, configuration.clone()); CloudApiUtils.checkQiniu(bucketManager, bucket); + caller = list -> null; } public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type, @@ -41,8 +49,10 @@ public ChangeType(String accessKey, String secretKey, Configuration configuratio this.batchOperations = new BatchOperations(); this.lines = new ArrayList<>(); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.auth = Auth.create(accessKey, secretKey); + this.bucketManager = new BucketManager(auth, configuration.clone()); CloudApiUtils.checkQiniu(bucketManager, bucket); + caller = list -> null; } public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type, @@ -50,12 +60,25 @@ public ChangeType(String accessKey, String secretKey, Configuration configuratio this(accessKey, secretKey, configuration, bucket, type, savePath, 0); } + public void enableRestoreArchive(int days, String condition) { + this.days = days; + this.condition = condition; + restoreArchive = new RestoreArchive(auth, secretKey, configuration, bucket, days, condition); + caller = restoreArchive::batchResult; + if (batchSize > 100) batchSize = 100; // 因为 restore archive 操作一次只能 100 个 entry + } + @Override public ChangeType clone() throws CloneNotSupportedException { ChangeType changeType = (ChangeType)super.clone(); - changeType.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + changeType.auth = Auth.create(accessId, secretKey); + changeType.bucketManager = new BucketManager(changeType.auth, configuration.clone()); changeType.batchOperations = new BatchOperations(); changeType.lines = new ArrayList<>(); + if (restoreArchive != null) { + changeType.restoreArchive = new RestoreArchive(changeType.auth, secretKey, configuration, bucket, days, condition); + changeType.caller = restoreArchive::batchResult; + } return changeType; } @@ -69,6 +92,7 @@ protected List> putBatchOperations(List> batchOperations.clearOps(); lines.clear(); String key; + caller.call(processList); for (Map map : processList) { key = map.get("key"); if (key != null) { @@ -90,6 +114,7 @@ protected String batchResult(List> lineList) throws IOExcept protected String singleResult(Map line) throws IOException { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); + if (restoreArchive != null) restoreArchive.singleResult(line); Response response = bucketManager.changeType(bucket, key, storageType); if (response.statusCode != 200) throw new QiniuException(response); response.close(); @@ -101,7 +126,9 @@ public void closeResource() { super.closeResource(); batchOperations = null; lines = null; + auth = null; configuration = null; bucketManager = null; + caller = null; } } diff --git a/src/main/java/com/qiniu/process/qiniu/FetchFile.java b/src/main/java/com/qiniu/process/qiniu/FetchFile.java index 6ab84e40..3249afaf 100755 --- a/src/main/java/com/qiniu/process/qiniu/FetchFile.java +++ b/src/main/java/com/qiniu/process/qiniu/FetchFile.java @@ -1,6 +1,5 @@ package com.qiniu.process.qiniu; -import com.qiniu.http.Response; import com.qiniu.interfaces.IFileChecker; import com.qiniu.process.Base; import com.qiniu.storage.BucketManager; diff --git a/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java b/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java index 98dc6b73..6302f0a0 100755 --- a/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java +++ b/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java @@ -52,6 +52,16 @@ public RestoreArchive(String accessKey, String secretKey, Configuration configur this(accessKey, secretKey, configuration, bucket, days, condition, savePath, 0); } + RestoreArchive(Auth auth, String secretKey, Configuration configuration, String bucket, int days, String condition) { + super("restorear", auth.accessKey, secretKey, bucket); + this.auth = auth; + this.days = days; + this.condition = condition; + this.configuration = configuration; + this.client = new Client(configuration.clone()); + requestUrl = configuration.useHttpsDomains ? "https://rs.qbox.me/restoreAr" : "http://rs.qbox.me/restoreAr"; + } + @Override public RestoreArchive clone() throws CloneNotSupportedException { RestoreArchive restoreArchive = (RestoreArchive)super.clone(); @@ -99,7 +109,7 @@ protected List> parseBatchResult(List> p } @Override - protected String singleResult(Map line) throws Exception { + protected String singleResult(Map line) throws IOException { JsonObject bodyJson = new JsonObject(); JsonArray entries = new JsonArray(); JsonObject entry = new JsonObject(); From 357993309e65403371279c163d5ae90eb107e4d6 Mon Sep 17 00:00:00 2001 From: wubingheng Date: Wed, 19 Feb 2020 16:03:28 +0800 Subject: [PATCH 04/12] fix UploadFile's keepPath and optimize qiniu configuration use. --- .../qiniu/datasource/QiniuQosContainer.java | 3 +- .../com/qiniu/process/other/DownloadFile.java | 6 +-- .../com/qiniu/process/other/ExportTS.java | 6 +-- .../qiniu/process/other/HttpDownloader.java | 2 +- .../com/qiniu/process/other/M3U8Manager.java | 4 +- .../com/qiniu/process/qiniu/AsyncFetch.java | 6 +-- .../com/qiniu/process/qiniu/CdnHelper.java | 7 +--- .../qiniu/process/qiniu/CdnUrlProcess.java | 30 ++++++--------- .../com/qiniu/process/qiniu/CdnUrlQuery.java | 38 ++++++++----------- .../qiniu/process/qiniu/CensorManager.java | 2 +- .../qiniu/process/qiniu/ChangeLifecycle.java | 6 +-- .../com/qiniu/process/qiniu/ChangeStatus.java | 6 +-- .../com/qiniu/process/qiniu/ChangeType.java | 6 +-- .../com/qiniu/process/qiniu/CopyFile.java | 6 +-- .../com/qiniu/process/qiniu/DeleteFile.java | 6 +-- .../qiniu/process/qiniu/DomainsOfBucket.java | 6 +-- .../com/qiniu/process/qiniu/FetchFile.java | 6 +-- .../com/qiniu/process/qiniu/FileChecker.java | 2 +- .../com/qiniu/process/qiniu/ImageCensor.java | 6 +-- .../com/qiniu/process/qiniu/MediaManager.java | 2 +- .../com/qiniu/process/qiniu/MirrorFile.java | 6 +-- .../com/qiniu/process/qiniu/MoveFile.java | 6 +-- .../com/qiniu/process/qiniu/PfopCommand.java | 6 +-- .../com/qiniu/process/qiniu/QiniuPfop.java | 6 +-- .../com/qiniu/process/qiniu/QueryAvinfo.java | 6 +-- .../process/qiniu/QueryCensorResult.java | 6 +-- .../com/qiniu/process/qiniu/QueryHash.java | 6 +-- .../qiniu/process/qiniu/QueryPfopResult.java | 6 +-- .../com/qiniu/process/qiniu/StatFile.java | 6 +-- .../com/qiniu/process/qiniu/SyncUpload.java | 15 +++----- .../com/qiniu/process/qiniu/UploadFile.java | 27 ++++++------- .../com/qiniu/process/qiniu/VideoCensor.java | 6 +-- 32 files changed, 120 insertions(+), 138 deletions(-) diff --git a/src/main/java/com/qiniu/datasource/QiniuQosContainer.java b/src/main/java/com/qiniu/datasource/QiniuQosContainer.java index 233dda3d..a6d02500 100755 --- a/src/main/java/com/qiniu/datasource/QiniuQosContainer.java +++ b/src/main/java/com/qiniu/datasource/QiniuQosContainer.java @@ -84,7 +84,8 @@ protected IResultOutput getNewResultSaver(String order) throws IOException { @Override protected IStorageLister getLister(String prefix, String marker, String start, String end, int unitLen) throws SuitsException { if (marker == null || "".equals(marker)) marker = CloudApiUtils.getQiniuMarker(start); - return new QiniuLister(new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()), bucket, + // BucketManager 中已经做了 configuration.clone() + return new QiniuLister(new BucketManager(Auth.create(accessKey, secretKey), configuration), bucket, prefix, marker, end, unitLen); } } diff --git a/src/main/java/com/qiniu/process/other/DownloadFile.java b/src/main/java/com/qiniu/process/other/DownloadFile.java index c1ce1296..5e17e179 100755 --- a/src/main/java/com/qiniu/process/other/DownloadFile.java +++ b/src/main/java/com/qiniu/process/other/DownloadFile.java @@ -30,14 +30,14 @@ public DownloadFile(Configuration configuration, String protocol, String domain, throws IOException { super("download", "", "", null, savePath, saveIndex); set(configuration, protocol, domain, urlIndex, host, range, suffixOrQuery, addPrefix, rmPrefix, downPath); - downloader = new HttpDownloader(configuration.clone()); + downloader = new HttpDownloader(configuration); } public DownloadFile(Configuration configuration, String protocol, String domain, String urlIndex, String host, int[] range, String suffixOrQuery, String addPrefix, String rmPrefix, String downPath) throws IOException { super("download", "", "", null); set(configuration, protocol, domain, urlIndex, host, range, suffixOrQuery, addPrefix, rmPrefix, downPath); - downloader = new HttpDownloader(configuration.clone()); + downloader = new HttpDownloader(configuration); } public DownloadFile(Configuration configuration, String protocol, String domain, String urlIndex, String host, @@ -88,7 +88,7 @@ private void set(Configuration configuration, String protocol, String domain, St @Override public DownloadFile clone() throws CloneNotSupportedException { DownloadFile downloadFile = (DownloadFile)super.clone(); - downloadFile.downloader = new HttpDownloader(configuration.clone()); + downloadFile.downloader = new HttpDownloader(configuration); return downloadFile; } diff --git a/src/main/java/com/qiniu/process/other/ExportTS.java b/src/main/java/com/qiniu/process/other/ExportTS.java index 2db3fb21..db7d9eea 100755 --- a/src/main/java/com/qiniu/process/other/ExportTS.java +++ b/src/main/java/com/qiniu/process/other/ExportTS.java @@ -20,14 +20,14 @@ public class ExportTS extends Base> { public ExportTS(Configuration configuration, String protocol, String domain, String urlIndex) throws IOException { super("exportts", "", "", null); set(configuration, protocol, domain, urlIndex); - this.m3U8Manager = new M3U8Manager(configuration.clone(), protocol); + this.m3U8Manager = new M3U8Manager(configuration, protocol); } public ExportTS(Configuration configuration, String protocol, String domain, String urlIndex, String savePath, int saveIndex) throws IOException { super("exportts", "", "", null, savePath, saveIndex); set(configuration, protocol, domain, urlIndex); - this.m3U8Manager = new M3U8Manager(configuration.clone(), protocol); + this.m3U8Manager = new M3U8Manager(configuration, protocol); } public ExportTS(Configuration configuration, String protocol, String domain, String urlIndex, String savePath) @@ -54,7 +54,7 @@ private void set(Configuration configuration, String protocol, String domain, St @Override public ExportTS clone() throws CloneNotSupportedException { ExportTS exportTS = (ExportTS)super.clone(); - exportTS.m3U8Manager = new M3U8Manager(configuration.clone(), protocol); + exportTS.m3U8Manager = new M3U8Manager(configuration, protocol); return exportTS; } diff --git a/src/main/java/com/qiniu/process/other/HttpDownloader.java b/src/main/java/com/qiniu/process/other/HttpDownloader.java index 80192844..8fc05e77 100755 --- a/src/main/java/com/qiniu/process/other/HttpDownloader.java +++ b/src/main/java/com/qiniu/process/other/HttpDownloader.java @@ -22,7 +22,7 @@ public HttpDownloader() { } public HttpDownloader(Configuration configuration) { - this.client = new Client(configuration); + this.client = configuration == null ? new Client() : new Client(configuration.clone()); } public File download(String url, String filepath, StringMap headers) throws IOException { diff --git a/src/main/java/com/qiniu/process/other/M3U8Manager.java b/src/main/java/com/qiniu/process/other/M3U8Manager.java index c285f12d..875d4a03 100755 --- a/src/main/java/com/qiniu/process/other/M3U8Manager.java +++ b/src/main/java/com/qiniu/process/other/M3U8Manager.java @@ -24,7 +24,7 @@ public M3U8Manager() { } public M3U8Manager(Configuration configuration) { - this.client = new Client(configuration); + this.client = configuration == null ? new Client() : new Client(configuration.clone()); } public M3U8Manager(String protocol) { @@ -33,7 +33,7 @@ public M3U8Manager(String protocol) { } public M3U8Manager(Configuration configuration, String protocol) { - this.client = new Client(configuration); + this.client = configuration == null ? new Client() : new Client(configuration.clone()); this.protocol = "https".equals(protocol)? "https" : "http"; } diff --git a/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java b/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java index fc026c09..3698a2a7 100755 --- a/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java +++ b/src/main/java/com/qiniu/process/qiniu/AsyncFetch.java @@ -41,7 +41,7 @@ public AsyncFetch(String accessKey, String secretKey, Configuration configuratio this.auth = Auth.create(accessKey, secretKey); this.client = new Client(configuration.clone()); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); -// this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); +// this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); // CloudApiUtils.checkQiniu(bucketManager, bucket); set(configuration, protocol, domain, urlIndex, addPrefix, rmPrefix); } @@ -53,7 +53,7 @@ public AsyncFetch(String accessKey, String secretKey, Configuration configuratio this.auth = Auth.create(accessKey, secretKey); this.client = new Client(configuration.clone()); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); -// this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); +// this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); // CloudApiUtils.checkQiniu(bucketManager, bucket); set(configuration, protocol, domain, urlIndex, addPrefix, rmPrefix); } @@ -102,7 +102,7 @@ public AsyncFetch clone() throws CloneNotSupportedException { AsyncFetch asyncFetch = (AsyncFetch)super.clone(); asyncFetch.auth = Auth.create(accessId, secretKey); asyncFetch.client = new Client(configuration.clone()); -// asyncFetch.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); +// asyncFetch.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); return asyncFetch; } diff --git a/src/main/java/com/qiniu/process/qiniu/CdnHelper.java b/src/main/java/com/qiniu/process/qiniu/CdnHelper.java index b2524331..8179b473 100644 --- a/src/main/java/com/qiniu/process/qiniu/CdnHelper.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnHelper.java @@ -20,14 +20,9 @@ public class CdnHelper { private static final String refreshQueryUrl = "http://fusion.qiniuapi.com/v2/tune/refresh/list"; private static final String prefetchQueryUrl = "http://fusion.qiniuapi.com/v2/tune/prefetch/list"; - public CdnHelper(Auth auth) { - this.auth = auth; - this.client = new Client(); - } - public CdnHelper(Auth auth, Configuration configuration) { this.auth = auth; - this.client = new Client(configuration); + this.client = configuration == null ? new Client() : new Client(configuration); } public Response refresh(String[] urls, String[] dirs) throws QiniuException { diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java index 87875393..f907a030 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java @@ -24,29 +24,18 @@ public class CdnUrlProcess extends Base> { // private boolean prefetch; private List batches; private List> lines; +// private Configuration configuration; private ICdnApplier cdnApplier; public CdnUrlProcess(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, String urlIndex, boolean isDir, boolean prefetch) throws IOException { super(prefetch ? "cdnprefetch" : "cdnrefresh", accessKey, secretKey, null); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration.clone()); + CdnHelper cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::prefetch : isDir ? dirs -> cdnHelper.refresh(null, dirs) : urls -> cdnHelper.refresh(urls, null); CloudApiUtils.checkQiniu(auth); - if (domain == null || "".equals(domain)) { - if (urlIndex == null || "".equals(urlIndex)) { - throw new IOException("please set one of domain and url-index."); - } else { - this.urlIndex = urlIndex; - } - } else { - this.protocol = protocol == null || !protocol.matches("(http|https)") ? "http" : protocol; - RequestUtils.lookUpFirstIpFromHost(domain); - this.domain = domain; - this.urlIndex = "url"; - } - this.isDir = isDir; + set(configuration, protocol, domain, urlIndex, isDir); // this.prefetch = prefetch; } @@ -57,10 +46,17 @@ public CdnUrlProcess(String accessKey, String secretKey, Configuration configura this.batches = new ArrayList<>(30); this.lines = new ArrayList<>(); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration.clone()); + CdnHelper cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::prefetch : isDir ? dirs -> cdnHelper.refresh(null, dirs) : urls -> cdnHelper.refresh(urls, null); CloudApiUtils.checkQiniu(auth); + set(configuration, protocol, domain, urlIndex, isDir); +// this.prefetch = prefetch; + this.fileSaveMapper.preAddWriter("invalid"); + } + + public void set(Configuration configuration, String protocol, String domain, String urlIndex, boolean isDir) throws IOException { +// this.configuration = configuration; if (domain == null || "".equals(domain)) { if (urlIndex == null || "".equals(urlIndex)) { throw new IOException("please set one of domain and url-index."); @@ -74,8 +70,6 @@ public CdnUrlProcess(String accessKey, String secretKey, Configuration configura this.urlIndex = "url"; } this.isDir = isDir; -// this.prefetch = prefetch; - this.fileSaveMapper.preAddWriter("invalid"); } public CdnUrlProcess(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, @@ -87,7 +81,7 @@ public CdnUrlProcess(String accessKey, String secretKey, Configuration configura public CdnUrlProcess clone() throws CloneNotSupportedException { // CdnUrlProcess cdnUrlProcess = (CdnUrlProcess)super.clone(); // cdnUrlProcess.lines = new ArrayList<>(); -// CdnHelper cdnHelper = new CdnHelper(Auth.create(accessId, secretKey)); +// CdnHelper cdnHelper = new CdnHelper(Auth.create(accessId, secretKey), configuration); // cdnUrlProcess.cdnApplier = prefetch ? cdnHelper::prefetch : isDir ? // dirs -> cdnHelper.refresh(null, dirs) : urls -> cdnHelper.refresh(urls, null); // if (cdnUrlProcess.fileSaveMapper != null) { diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java index 372194bd..b1795cde 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java @@ -24,27 +24,16 @@ public class CdnUrlQuery extends Base> { private List batches; private List> lines; private ICdnApplier cdnApplier; + private Configuration configuration; public CdnUrlQuery(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, String urlIndex, boolean prefetch) throws IOException { super(prefetch ? "prefetchquery" : "refreshquery", accessKey, secretKey, null); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration.clone()); + CdnHelper cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; CloudApiUtils.checkQiniu(auth); - if (domain == null || "".equals(domain)) { - if (urlIndex == null || "".equals(urlIndex)) { - throw new IOException("please set one of domain and url-index."); - } else { - this.urlIndex = urlIndex; - } - } else { - this.protocol = protocol == null || !protocol.matches("(http|https)") ? "http" : protocol; - RequestUtils.lookUpFirstIpFromHost(domain); - this.domain = domain; - this.urlIndex = "url"; - } - this.prefetch = prefetch; + set(configuration, protocol, domain, urlIndex, prefetch); } public CdnUrlQuery(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, @@ -54,9 +43,20 @@ public CdnUrlQuery(String accessKey, String secretKey, Configuration configurati this.batches = new ArrayList<>(30); this.lines = new ArrayList<>(); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration.clone()); + CdnHelper cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; CloudApiUtils.checkQiniu(auth); + set(configuration, protocol, domain, urlIndex, prefetch); + this.fileSaveMapper.preAddWriter("processing"); + } + + public CdnUrlQuery(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, + String urlIndex, boolean prefetch, String savePath) throws IOException { + this(accessKey, secretKey, configuration, protocol, domain, urlIndex, prefetch, savePath, 0); + } + + public void set(Configuration configuration, String protocol, String domain, String urlIndex, boolean prefetch) throws IOException { + this.configuration = configuration; if (domain == null || "".equals(domain)) { if (urlIndex == null || "".equals(urlIndex)) { throw new IOException("please set one of domain and url-index."); @@ -70,19 +70,13 @@ public CdnUrlQuery(String accessKey, String secretKey, Configuration configurati this.urlIndex = "url"; } this.prefetch = prefetch; - this.fileSaveMapper.preAddWriter("processing"); - } - - public CdnUrlQuery(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, - String urlIndex, boolean prefetch, String savePath) throws IOException { - this(accessKey, secretKey, configuration, protocol, domain, urlIndex, prefetch, savePath, 0); } @Override public CdnUrlQuery clone() throws CloneNotSupportedException { CdnUrlQuery cdnUrlQuery = (CdnUrlQuery)super.clone(); cdnUrlQuery.lines = new ArrayList<>(); - CdnHelper cdnHelper = new CdnHelper(Auth.create(accessId, secretKey)); + CdnHelper cdnHelper = new CdnHelper(Auth.create(accessId, secretKey), configuration); cdnUrlQuery.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; if (cdnUrlQuery.fileSaveMapper != null) { cdnUrlQuery.fileSaveMapper.preAddWriter("processing"); diff --git a/src/main/java/com/qiniu/process/qiniu/CensorManager.java b/src/main/java/com/qiniu/process/qiniu/CensorManager.java index 615bfd89..ecb62692 100755 --- a/src/main/java/com/qiniu/process/qiniu/CensorManager.java +++ b/src/main/java/com/qiniu/process/qiniu/CensorManager.java @@ -40,7 +40,7 @@ public static JsonArray getScenes(String[] scenes) throws IOException { public CensorManager(Auth auth, Configuration configuration) { this.auth = auth; - this.client = configuration == null ? new Client() : new Client(configuration); + this.client = configuration == null ? new Client() : new Client(configuration.clone()); this.bodyJson = new JsonObject(); this.paramsJson = new JsonObject(); this.uriJson = new JsonObject(); diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java b/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java index 273d5b6d..315b763f 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java @@ -28,7 +28,7 @@ public ChangeLifecycle(String accessKey, String secretKey, Configuration configu super("lifecycle", accessKey, secretKey, bucket); this.days = days; this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -40,7 +40,7 @@ public ChangeLifecycle(String accessKey, String secretKey, Configuration configu this.batchOperations = new BatchOperations(); this.lines = new ArrayList<>(); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -52,7 +52,7 @@ public ChangeLifecycle(String accessKey, String secretKey, Configuration configu @Override public ChangeLifecycle clone() throws CloneNotSupportedException { ChangeLifecycle changeLifecycle = (ChangeLifecycle)super.clone(); - changeLifecycle.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + changeLifecycle.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); changeLifecycle.batchOperations = new BatchOperations(); changeLifecycle.lines = new ArrayList<>(); return changeLifecycle; diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java b/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java index e6196f5d..b5d6e7a7 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java @@ -28,7 +28,7 @@ public ChangeStatus(String accessKey, String secretKey, Configuration configurat super("status", accessKey, secretKey, bucket); this.status = status; this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -40,7 +40,7 @@ public ChangeStatus(String accessKey, String secretKey, Configuration configurat this.batchOperations = new BatchOperations(); this.lines = new ArrayList<>(); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -52,7 +52,7 @@ public ChangeStatus(String accessKey, String secretKey, Configuration configurat @Override public ChangeStatus clone() throws CloneNotSupportedException { ChangeStatus changeStatus = (ChangeStatus)super.clone(); - changeStatus.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + changeStatus.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); changeStatus.batchOperations = new BatchOperations(); changeStatus.lines = new ArrayList<>(); return changeStatus; diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeType.java b/src/main/java/com/qiniu/process/qiniu/ChangeType.java index cb9c6371..49e8ae35 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeType.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeType.java @@ -36,7 +36,7 @@ public ChangeType(String accessKey, String secretKey, Configuration configuratio storageType = type == 0 ? StorageType.COMMON : StorageType.INFREQUENCY; this.auth = Auth.create(accessKey, secretKey); this.configuration = configuration; - this.bucketManager = new BucketManager(auth, configuration.clone()); + this.bucketManager = new BucketManager(auth, configuration); // BucketManager 中已经做了 configuration.clone() CloudApiUtils.checkQiniu(bucketManager, bucket); caller = list -> null; } @@ -50,7 +50,7 @@ public ChangeType(String accessKey, String secretKey, Configuration configuratio this.lines = new ArrayList<>(); this.configuration = configuration; this.auth = Auth.create(accessKey, secretKey); - this.bucketManager = new BucketManager(auth, configuration.clone()); + this.bucketManager = new BucketManager(auth, configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); caller = list -> null; } @@ -72,7 +72,7 @@ public void enableRestoreArchive(int days, String condition) { public ChangeType clone() throws CloneNotSupportedException { ChangeType changeType = (ChangeType)super.clone(); changeType.auth = Auth.create(accessId, secretKey); - changeType.bucketManager = new BucketManager(changeType.auth, configuration.clone()); + changeType.bucketManager = new BucketManager(changeType.auth, configuration); changeType.batchOperations = new BatchOperations(); changeType.lines = new ArrayList<>(); if (restoreArchive != null) { diff --git a/src/main/java/com/qiniu/process/qiniu/CopyFile.java b/src/main/java/com/qiniu/process/qiniu/CopyFile.java index 050ed061..e3740438 100755 --- a/src/main/java/com/qiniu/process/qiniu/CopyFile.java +++ b/src/main/java/com/qiniu/process/qiniu/CopyFile.java @@ -31,7 +31,7 @@ public class CopyFile extends Base> { public CopyFile(String accessKey, String secretKey, Configuration configuration, String bucket, String toBucket, String toKeyIndex, String addPrefix, String rmPrefix) throws IOException { super("copy", accessKey, secretKey, bucket); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); CloudApiUtils.checkQiniu(bucketManager, toBucket); set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix); @@ -43,7 +43,7 @@ public CopyFile(String accessKey, String secretKey, Configuration configuration, this.batchSize = 1000; this.batchOperations = new BatchOperations(); this.lines = new ArrayList<>(); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); CloudApiUtils.checkQiniu(bucketManager, toBucket); set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix); @@ -73,7 +73,7 @@ private void set(Configuration configuration, String toBucket, String toKeyIndex @Override public CopyFile clone() throws CloneNotSupportedException { CopyFile copyFile = (CopyFile)super.clone(); - copyFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + copyFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); copyFile.batchOperations = new BatchOperations(); copyFile.lines = new ArrayList<>(); return copyFile; diff --git a/src/main/java/com/qiniu/process/qiniu/DeleteFile.java b/src/main/java/com/qiniu/process/qiniu/DeleteFile.java index e56af4ff..a873f861 100755 --- a/src/main/java/com/qiniu/process/qiniu/DeleteFile.java +++ b/src/main/java/com/qiniu/process/qiniu/DeleteFile.java @@ -25,7 +25,7 @@ public class DeleteFile extends Base> { public DeleteFile(String accessKey, String secretKey, Configuration configuration, String bucket) throws IOException { super("delete", accessKey, secretKey, bucket); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -36,7 +36,7 @@ public DeleteFile(String accessKey, String secretKey, Configuration configuratio this.batchOperations = new BatchOperations(); this.lines = new ArrayList<>(); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -48,7 +48,7 @@ public DeleteFile(String accessKey, String secretKey, Configuration configuratio @Override public DeleteFile clone() throws CloneNotSupportedException { DeleteFile deleteFile = (DeleteFile)super.clone(); - deleteFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + deleteFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); deleteFile.batchOperations = new BatchOperations(); deleteFile.lines = new ArrayList<>(); return deleteFile; diff --git a/src/main/java/com/qiniu/process/qiniu/DomainsOfBucket.java b/src/main/java/com/qiniu/process/qiniu/DomainsOfBucket.java index cd0d28dc..9420ad4c 100755 --- a/src/main/java/com/qiniu/process/qiniu/DomainsOfBucket.java +++ b/src/main/java/com/qiniu/process/qiniu/DomainsOfBucket.java @@ -18,7 +18,7 @@ public DomainsOfBucket(String accessKey, String secretKey, Configuration configu super("domainsofbucket", accessKey, secretKey, null); this.configuration = configuration; Auth auth = Auth.create(accessKey, secretKey); - this.bucketManager = new BucketManager(auth, configuration.clone()); + this.bucketManager = new BucketManager(auth, configuration); CloudApiUtils.checkQiniu(auth); } @@ -26,7 +26,7 @@ public DomainsOfBucket(String accessKey, String secretKey, Configuration configu super("domainsofbucket", accessKey, secretKey, null, savePath, saveIndex); this.configuration = configuration; Auth auth = Auth.create(accessKey, secretKey); - this.bucketManager = new BucketManager(auth, configuration.clone()); + this.bucketManager = new BucketManager(auth, configuration); CloudApiUtils.checkQiniu(auth); } @@ -38,7 +38,7 @@ public DomainsOfBucket(String accessKey, String secretKey, Configuration configu @Override public DomainsOfBucket clone() throws CloneNotSupportedException { DomainsOfBucket mirrorFile = (DomainsOfBucket) super.clone(); - mirrorFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + mirrorFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); return mirrorFile; } diff --git a/src/main/java/com/qiniu/process/qiniu/FetchFile.java b/src/main/java/com/qiniu/process/qiniu/FetchFile.java index 3249afaf..0214abc3 100755 --- a/src/main/java/com/qiniu/process/qiniu/FetchFile.java +++ b/src/main/java/com/qiniu/process/qiniu/FetchFile.java @@ -23,7 +23,7 @@ public FetchFile(String accessKey, String secretKey, Configuration configuration String domain, String urlIndex, String addPrefix, String rmPrefix) throws IOException { super("fetch", accessKey, secretKey, bucket); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); set(configuration, protocol, domain, urlIndex, addPrefix, rmPrefix); } @@ -33,7 +33,7 @@ public FetchFile(String accessKey, String secretKey, Configuration configuration throws IOException { super("fetch", accessKey, secretKey, bucket, savePath, saveIndex); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); set(configuration, protocol, domain, urlIndex, addPrefix, rmPrefix); } @@ -65,7 +65,7 @@ private void set(Configuration configuration, String protocol, String domain, St @Override public FetchFile clone() throws CloneNotSupportedException { FetchFile fetchFile = (FetchFile) super.clone(); - fetchFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + fetchFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); return fetchFile; } diff --git a/src/main/java/com/qiniu/process/qiniu/FileChecker.java b/src/main/java/com/qiniu/process/qiniu/FileChecker.java index e5a8d67f..66636162 100755 --- a/src/main/java/com/qiniu/process/qiniu/FileChecker.java +++ b/src/main/java/com/qiniu/process/qiniu/FileChecker.java @@ -32,7 +32,7 @@ public FileChecker(String protocol, String algorithm) { } public FileChecker(Configuration configuration, String protocol, String algorithm) { - this.client = new Client(configuration); + this.client = configuration == null ? new Client() : new Client(configuration.clone()); this.protocol = "https".equals(protocol)? "https" : "http"; this.algorithm = algorithms.contains(algorithm) ? algorithm : "md5"; } diff --git a/src/main/java/com/qiniu/process/qiniu/ImageCensor.java b/src/main/java/com/qiniu/process/qiniu/ImageCensor.java index 5f1472d3..a327d4e0 100755 --- a/src/main/java/com/qiniu/process/qiniu/ImageCensor.java +++ b/src/main/java/com/qiniu/process/qiniu/ImageCensor.java @@ -26,7 +26,7 @@ public ImageCensor(String accessKey, String secretKey, Configuration configurati set(configuration, protocol, domain, urlIndex, suffixOrQuery, scenes); Auth auth = Auth.create(accessKey, secretKey); CloudApiUtils.checkQiniu(auth); - censorManager = new CensorManager(auth, configuration.clone()); + censorManager = new CensorManager(auth, configuration); } public ImageCensor(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, @@ -35,7 +35,7 @@ public ImageCensor(String accessKey, String secretKey, Configuration configurati set(configuration, protocol, domain, urlIndex, suffixOrQuery, scenes); Auth auth = Auth.create(accessKey, secretKey); CloudApiUtils.checkQiniu(auth); - censorManager = new CensorManager(auth, configuration.clone()); + censorManager = new CensorManager(auth, configuration); } public ImageCensor(String accesskey, String secretKey, Configuration configuration, String protocol, String domain, @@ -66,7 +66,7 @@ private void set(Configuration configuration, String protocol, String domain, St @Override public ImageCensor clone() throws CloneNotSupportedException { ImageCensor videoCensor = (ImageCensor)super.clone(); - videoCensor.censorManager = new CensorManager(Auth.create(accessId, secretKey), configuration.clone()); + videoCensor.censorManager = new CensorManager(Auth.create(accessId, secretKey), configuration); return videoCensor; } diff --git a/src/main/java/com/qiniu/process/qiniu/MediaManager.java b/src/main/java/com/qiniu/process/qiniu/MediaManager.java index 785d9a6e..8829b903 100755 --- a/src/main/java/com/qiniu/process/qiniu/MediaManager.java +++ b/src/main/java/com/qiniu/process/qiniu/MediaManager.java @@ -25,7 +25,7 @@ public MediaManager(String protocol) { } public MediaManager(Configuration configuration) { - this.client = new Client(configuration); + this.client = configuration == null ? new Client() : new Client(configuration.clone()); this.protocol = "http"; } diff --git a/src/main/java/com/qiniu/process/qiniu/MirrorFile.java b/src/main/java/com/qiniu/process/qiniu/MirrorFile.java index f7be6fc3..27e79cca 100755 --- a/src/main/java/com/qiniu/process/qiniu/MirrorFile.java +++ b/src/main/java/com/qiniu/process/qiniu/MirrorFile.java @@ -19,7 +19,7 @@ public class MirrorFile extends Base> { public MirrorFile(String accessKey, String secretKey, Configuration configuration, String bucket) throws IOException { super("mirror", accessKey, secretKey, bucket); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -27,7 +27,7 @@ public MirrorFile(String accessKey, String secretKey, Configuration configuratio int saveIndex) throws IOException { super("mirror", accessKey, secretKey, bucket, savePath, saveIndex); this.configuration = configuration; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -39,7 +39,7 @@ public MirrorFile(String accessKey, String secretKey, Configuration configuratio @Override public MirrorFile clone() throws CloneNotSupportedException { MirrorFile mirrorFile = (MirrorFile) super.clone(); - mirrorFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + mirrorFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); return mirrorFile; } diff --git a/src/main/java/com/qiniu/process/qiniu/MoveFile.java b/src/main/java/com/qiniu/process/qiniu/MoveFile.java index 6be8c340..621056b3 100755 --- a/src/main/java/com/qiniu/process/qiniu/MoveFile.java +++ b/src/main/java/com/qiniu/process/qiniu/MoveFile.java @@ -34,7 +34,7 @@ public MoveFile(String accessKey, String secretKey, Configuration configuration, // 目标 bucket 为空时规定为 rename 操作 super(toBucket == null || "".equals(toBucket) ? "rename" : "move", accessKey, secretKey, bucket); if ("rename".equals(processName)) isRename = true; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); CloudApiUtils.checkQiniu(bucketManager, toBucket); set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, forceIfOnlyPrefix); @@ -49,7 +49,7 @@ public MoveFile(String accessKey, String secretKey, Configuration configuration, this.batchSize = 1000; this.batchOperations = new BatchOperations(); this.lines = new ArrayList<>(); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); CloudApiUtils.checkQiniu(bucketManager, toBucket); set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, forceIfOnlyPrefix); @@ -91,7 +91,7 @@ private void set(Configuration configuration, String toBucket, String toKeyIndex @Override public MoveFile clone() throws CloneNotSupportedException { MoveFile moveFile = (MoveFile)super.clone(); - moveFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + moveFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); moveFile.batchOperations = new BatchOperations(); moveFile.lines = new ArrayList<>(); return moveFile; diff --git a/src/main/java/com/qiniu/process/qiniu/PfopCommand.java b/src/main/java/com/qiniu/process/qiniu/PfopCommand.java index 335605d4..e5878279 100755 --- a/src/main/java/com/qiniu/process/qiniu/PfopCommand.java +++ b/src/main/java/com/qiniu/process/qiniu/PfopCommand.java @@ -31,14 +31,14 @@ public PfopCommand(Configuration configuration, String avinfoIndex, boolean hasD String pfopJsonPath, List pfopConfigs) throws IOException { super("pfopcmd", "", "", null); set(configuration, avinfoIndex, hasDuration, hasSize, combine, pfopJsonPath, pfopConfigs); - this.mediaManager = new MediaManager(configuration.clone()); + this.mediaManager = new MediaManager(configuration); } public PfopCommand(Configuration configuration, String avinfoIndex, boolean hasDuration, boolean hasSize, boolean combine, String pfopJsonPath, List pfopConfigs, String savePath, int saveIndex) throws IOException { super("pfopcmd", "", "", null, savePath, saveIndex); set(configuration, avinfoIndex, hasDuration, hasSize, combine, pfopJsonPath, pfopConfigs); - this.mediaManager = new MediaManager(configuration.clone()); + this.mediaManager = new MediaManager(configuration); } public PfopCommand(Configuration configuration, String avinfoIndex, boolean hasDuration, boolean hasSize, boolean combine, @@ -72,7 +72,7 @@ private void set(Configuration configuration, String avinfoIndex, boolean hasDur @Override public PfopCommand clone() throws CloneNotSupportedException { PfopCommand pfopCommand = (PfopCommand)super.clone(); - pfopCommand.mediaManager = new MediaManager(configuration.clone()); + pfopCommand.mediaManager = new MediaManager(configuration); return pfopCommand; } diff --git a/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java b/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java index bb529679..1a221200 100755 --- a/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java +++ b/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java @@ -27,7 +27,7 @@ public QiniuPfop(String accessKey, String secretKey, Configuration configuration String fopsIndex) throws IOException { super("pfop", accessKey, secretKey, bucket); set(configuration, pipeline, notifyURL, force, pfopJsonPath, pfopConfigs, fopsIndex); - this.operationManager = new OperationManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.operationManager = new OperationManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); } @@ -36,7 +36,7 @@ public QiniuPfop(String accessKey, String secretKey, Configuration configuration String fopsIndex, String savePath, int saveIndex) throws IOException { super("pfop", accessKey, secretKey, bucket, savePath, saveIndex); set(configuration, pipeline, notifyURL, force, pfopJsonPath, pfopConfigs, fopsIndex); - this.operationManager = new OperationManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.operationManager = new OperationManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); } @@ -75,7 +75,7 @@ private void set(Configuration configuration, String pipeline, String notifyURL, @Override public QiniuPfop clone() throws CloneNotSupportedException { QiniuPfop qiniuPfop = (QiniuPfop)super.clone(); - qiniuPfop.operationManager = new OperationManager(Auth.create(accessId, secretKey), configuration.clone()); + qiniuPfop.operationManager = new OperationManager(Auth.create(accessId, secretKey), configuration); return qiniuPfop; } diff --git a/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java b/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java index 194535d1..b14120fd 100755 --- a/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java +++ b/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java @@ -18,14 +18,14 @@ public class QueryAvinfo extends Base> { public QueryAvinfo(Configuration configuration, String protocol, String domain, String urlIndex) throws IOException { super("avinfo", "", "", null); set(configuration, protocol, domain, urlIndex); - this.mediaManager = new MediaManager(configuration.clone(), protocol); + this.mediaManager = new MediaManager(configuration, protocol); } public QueryAvinfo(Configuration configuration, String protocol, String domain, String urlIndex, String savePath, int saveIndex) throws IOException { super("avinfo", "", "", null, savePath, saveIndex); set(configuration, protocol, domain, urlIndex); - this.mediaManager = new MediaManager(configuration.clone(), protocol); + this.mediaManager = new MediaManager(configuration, protocol); } public QueryAvinfo(Configuration configuration, String protocol, String domain, String urlIndex, String savePath) @@ -52,7 +52,7 @@ private void set(Configuration configuration, String protocol, String domain, St @Override public QueryAvinfo clone() throws CloneNotSupportedException { QueryAvinfo queryAvinfo = (QueryAvinfo)super.clone(); - queryAvinfo.mediaManager = new MediaManager(configuration.clone(), protocol); + queryAvinfo.mediaManager = new MediaManager(configuration, protocol); return queryAvinfo; } diff --git a/src/main/java/com/qiniu/process/qiniu/QueryCensorResult.java b/src/main/java/com/qiniu/process/qiniu/QueryCensorResult.java index b5aa6f87..1186096a 100755 --- a/src/main/java/com/qiniu/process/qiniu/QueryCensorResult.java +++ b/src/main/java/com/qiniu/process/qiniu/QueryCensorResult.java @@ -22,7 +22,7 @@ public QueryCensorResult(String accessKey, String secretKey, Configuration confi else this.jobIdIndex = jobIdIndex; Auth auth = Auth.create(accessKey, secretKey); CloudApiUtils.checkQiniu(auth); - censorManager = new CensorManager(auth, configuration.clone()); + censorManager = new CensorManager(auth, configuration); } public QueryCensorResult(String accessKey, String secretKey, Configuration configuration, String jobIdIndex, String savePath, @@ -33,7 +33,7 @@ public QueryCensorResult(String accessKey, String secretKey, Configuration confi else this.jobIdIndex = jobIdIndex; Auth auth = Auth.create(accessKey, secretKey); CloudApiUtils.checkQiniu(auth); - censorManager = new CensorManager(auth, configuration.clone()); + censorManager = new CensorManager(auth, configuration); fileSaveMapper.preAddWriter("waiting"); } @@ -45,7 +45,7 @@ public QueryCensorResult(String accessKey, String secretKey, Configuration confi @Override public QueryCensorResult clone() throws CloneNotSupportedException { QueryCensorResult censorResult = (QueryCensorResult)super.clone(); - censorResult.censorManager = new CensorManager(Auth.create(accessId, secretKey), configuration.clone()); + censorResult.censorManager = new CensorManager(Auth.create(accessId, secretKey), configuration); if (censorResult.fileSaveMapper != null) { censorResult.fileSaveMapper.preAddWriter("waiting"); } diff --git a/src/main/java/com/qiniu/process/qiniu/QueryHash.java b/src/main/java/com/qiniu/process/qiniu/QueryHash.java index d294a940..1afa6647 100755 --- a/src/main/java/com/qiniu/process/qiniu/QueryHash.java +++ b/src/main/java/com/qiniu/process/qiniu/QueryHash.java @@ -20,14 +20,14 @@ public QueryHash(Configuration configuration, String protocol, String domain, St throws IOException { super("qhash", "", "", null); set(configuration, protocol, domain, urlIndex, algorithm); - this.fileChecker = new FileChecker(configuration.clone(), protocol, algorithm); + this.fileChecker = new FileChecker(configuration, protocol, algorithm); } public QueryHash(Configuration configuration, String protocol, String domain, String urlIndex, String algorithm, String savePath, int saveIndex) throws IOException { super("qhash", "", "", null, savePath, saveIndex); set(configuration, protocol, domain, urlIndex, algorithm); - this.fileChecker = new FileChecker(configuration.clone(), protocol, algorithm); + this.fileChecker = new FileChecker(configuration, protocol, algorithm); } public QueryHash(Configuration configuration, String protocol, String domain, String urlIndex, String algorithm, @@ -56,7 +56,7 @@ private void set(Configuration configuration, String protocol, String domain, St @Override public QueryHash clone() throws CloneNotSupportedException { QueryHash queryHash = (QueryHash)super.clone(); - queryHash.fileChecker = new FileChecker(configuration.clone(), algorithm, protocol); + queryHash.fileChecker = new FileChecker(configuration, algorithm, protocol); return queryHash; } diff --git a/src/main/java/com/qiniu/process/qiniu/QueryPfopResult.java b/src/main/java/com/qiniu/process/qiniu/QueryPfopResult.java index 44a0014a..4df92297 100755 --- a/src/main/java/com/qiniu/process/qiniu/QueryPfopResult.java +++ b/src/main/java/com/qiniu/process/qiniu/QueryPfopResult.java @@ -19,14 +19,14 @@ public class QueryPfopResult extends Base> { public QueryPfopResult(Configuration configuration, String protocol, String pIdIndex) throws IOException { super("pfopresult", "", "", null); set(configuration, protocol, pIdIndex); - this.mediaManager = new MediaManager(configuration.clone(), protocol); + this.mediaManager = new MediaManager(configuration, protocol); } public QueryPfopResult(Configuration configuration, String protocol, String pIdIndex, String savePath, int saveIndex) throws IOException { super("pfopresult", "", "", null, savePath, saveIndex); set(configuration, protocol, pIdIndex); - this.mediaManager = new MediaManager(configuration.clone(), protocol); + this.mediaManager = new MediaManager(configuration, protocol); this.fileSaveMapper.preAddWriter("waiting"); this.fileSaveMapper.preAddWriter("notify_failed"); } @@ -45,7 +45,7 @@ private void set(Configuration configuration, String protocol, String pidIndex) @Override public QueryPfopResult clone() throws CloneNotSupportedException { QueryPfopResult pfopResult = (QueryPfopResult)super.clone(); - pfopResult.mediaManager = new MediaManager(configuration.clone(), protocol); + pfopResult.mediaManager = new MediaManager(configuration, protocol); if (pfopResult.fileSaveMapper != null) { pfopResult.fileSaveMapper.preAddWriter("waiting"); pfopResult.fileSaveMapper.preAddWriter("notify_failed"); diff --git a/src/main/java/com/qiniu/process/qiniu/StatFile.java b/src/main/java/com/qiniu/process/qiniu/StatFile.java index 3695a698..2bef6564 100755 --- a/src/main/java/com/qiniu/process/qiniu/StatFile.java +++ b/src/main/java/com/qiniu/process/qiniu/StatFile.java @@ -32,7 +32,7 @@ public StatFile(String accessKey, String secretKey, Configuration configuration, String separator, List rmFields) throws IOException { super("stat", accessKey, secretKey, bucket); set(configuration, rmFields, format, separator); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -43,7 +43,7 @@ public StatFile(String accessKey, String secretKey, Configuration configuration, this.batchSize = 100; this.batchOperations = new BatchOperations(); this.lines = new ArrayList<>(); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration.clone()); + this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -78,7 +78,7 @@ private IStringFormat getNewStatJsonFormatter(List rmFields) @Override public StatFile clone() throws CloneNotSupportedException { StatFile statFile = (StatFile)super.clone(); - statFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration.clone()); + statFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); statFile.batchOperations = new BatchOperations(); statFile.lines = new ArrayList<>(); statFile.stringFormatter = getNewStatJsonFormatter(rmFields); diff --git a/src/main/java/com/qiniu/process/qiniu/SyncUpload.java b/src/main/java/com/qiniu/process/qiniu/SyncUpload.java index 8824e299..cb99c9d9 100755 --- a/src/main/java/com/qiniu/process/qiniu/SyncUpload.java +++ b/src/main/java/com/qiniu/process/qiniu/SyncUpload.java @@ -36,9 +36,8 @@ public SyncUpload(String accessKey, String secretKey, Configuration configuratio super("syncupload", accessKey, secretKey, bucket); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); auth = Auth.create(accessKey, secretKey); - Configuration config = configuration.clone(); - uploadManager = new UploadManager(config); - downloader = new HttpDownloader(config); + uploadManager = new UploadManager(configuration); + downloader = new HttpDownloader(configuration); set(configuration, protocol, domain, urlIndex, host, addPrefix, rmPrefix, expires, policy, params); } @@ -48,9 +47,8 @@ public SyncUpload(String accessKey, String secretKey, Configuration configuratio super("syncupload", accessKey, secretKey, bucket, savePath, saveIndex); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); auth = Auth.create(accessKey, secretKey); - Configuration config = configuration.clone(); - uploadManager = new UploadManager(config); - downloader = new HttpDownloader(config); + uploadManager = new UploadManager(configuration); + downloader = new HttpDownloader(configuration); set(configuration, protocol, domain, urlIndex, host, addPrefix, rmPrefix, expires, policy, params); } @@ -93,9 +91,8 @@ private void set(Configuration configuration, String protocol, String domain, St public SyncUpload clone() throws CloneNotSupportedException { SyncUpload syncUpload = (SyncUpload)super.clone(); syncUpload.auth = Auth.create(accessId, secretKey); - Configuration config = configuration.clone(); - syncUpload.uploadManager = new UploadManager(config); - syncUpload.downloader = new HttpDownloader(config); + syncUpload.uploadManager = new UploadManager(configuration); + syncUpload.downloader = new HttpDownloader(configuration); return syncUpload; } diff --git a/src/main/java/com/qiniu/process/qiniu/UploadFile.java b/src/main/java/com/qiniu/process/qiniu/UploadFile.java index 73fc466d..fdd344f5 100755 --- a/src/main/java/com/qiniu/process/qiniu/UploadFile.java +++ b/src/main/java/com/qiniu/process/qiniu/UploadFile.java @@ -35,9 +35,9 @@ public UploadFile(String accessKey, String secretKey, Configuration configuratio auth = Auth.create(accessKey, secretKey); if (record) { recorder = String.join(FileUtils.pathSeparator, FileUtils.userHome, ".qsuits.record"); - uploadManager = new UploadManager(configuration.clone(), new FileRecorder(recorder)); + uploadManager = new UploadManager(configuration, new FileRecorder(recorder)); } else { - uploadManager = new UploadManager(configuration.clone()); + uploadManager = new UploadManager(configuration); } set(configuration, pathIndex, parentPath, keepPath, addPrefix, rmPrefix, expires, policy, params, checkCrc); } @@ -50,9 +50,9 @@ public UploadFile(String accessKey, String secretKey, Configuration configuratio auth = Auth.create(accessKey, secretKey); if (record) { recorder = String.join(FileUtils.pathSeparator, savePath, ".record"); - uploadManager = new UploadManager(configuration.clone(), new FileRecorder(recorder)); + uploadManager = new UploadManager(configuration, new FileRecorder(recorder)); } else { - uploadManager = new UploadManager(configuration.clone()); + uploadManager = new UploadManager(configuration); } set(configuration, pathIndex, parentPath, keepPath, addPrefix, rmPrefix, expires, policy, params, checkCrc); } @@ -87,8 +87,8 @@ public UploadFile clone() throws CloneNotSupportedException { UploadFile uploadFile = (UploadFile)super.clone(); uploadFile.auth = Auth.create(accessId, secretKey); try { - uploadFile.uploadManager = recorder == null ? new UploadManager(configuration.clone()) : - new UploadManager(configuration.clone(), new FileRecorder(recorder)); + uploadFile.uploadManager = recorder == null ? new UploadManager(configuration) : + new UploadManager(configuration, new FileRecorder(recorder)); } catch (IOException e) { throw new CloneNotSupportedException(e.getMessage() + ", init writer failed."); } @@ -139,14 +139,15 @@ protected String singleResult(Map line) throws Exception { // key = filepath.substring(filepath.lastIndexOf(FileUtils.pathSeparator) + 1); // } // } - if (keepPath && key == null) { - key = filepath; - line.put("key", key); - } else if (key == null) { - key = filepath.substring(filepath.lastIndexOf(FileUtils.pathSeparator) + 1); - line.put("key", key); + if (keepPath) { + if (key == null) { + key = filepath; + line.put("key", key); + } } else { - key = key.substring(key.lastIndexOf(FileUtils.pathSeparator) + 1); + if (key == null) key = filepath.substring(filepath.lastIndexOf(FileUtils.pathSeparator) + 1); + else key = key.substring(key.lastIndexOf(FileUtils.pathSeparator) + 1); + line.put("key", key); } if (parentPath != null) { if (filepath.startsWith(FileUtils.pathSeparator)) { diff --git a/src/main/java/com/qiniu/process/qiniu/VideoCensor.java b/src/main/java/com/qiniu/process/qiniu/VideoCensor.java index 32043c19..46c7111f 100755 --- a/src/main/java/com/qiniu/process/qiniu/VideoCensor.java +++ b/src/main/java/com/qiniu/process/qiniu/VideoCensor.java @@ -27,7 +27,7 @@ public VideoCensor(String accessKey, String secretKey, Configuration configurati set(configuration, protocol, domain, urlIndex, scenes, interval, saverBucket, saverPrefix, hookUrl); Auth auth = Auth.create(accessKey, secretKey); CloudApiUtils.checkQiniu(auth); - censorManager = new CensorManager(auth, configuration.clone()); + censorManager = new CensorManager(auth, configuration); } public VideoCensor(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, @@ -37,7 +37,7 @@ public VideoCensor(String accessKey, String secretKey, Configuration configurati set(configuration, protocol, domain, urlIndex, scenes, interval, saverBucket, saverPrefix, hookUrl); Auth auth = Auth.create(accessKey, secretKey); CloudApiUtils.checkQiniu(auth); - censorManager = new CensorManager(auth, configuration.clone()); + censorManager = new CensorManager(auth, configuration); } public VideoCensor(String accesskey, String secretKey, Configuration configuration, String protocol, String domain, @@ -84,7 +84,7 @@ private void set(Configuration configuration, String protocol, String domain, St @Override public VideoCensor clone() throws CloneNotSupportedException { VideoCensor videoCensor = (VideoCensor)super.clone(); - videoCensor.censorManager = new CensorManager(Auth.create(accessId, secretKey), configuration.clone()); + videoCensor.censorManager = new CensorManager(Auth.create(accessId, secretKey), configuration); return videoCensor; } From 425756c9e8108d9ab7dfa81c5dca2302f239349b Mon Sep 17 00:00:00 2001 From: wubingheng Date: Wed, 19 Feb 2020 17:21:49 +0800 Subject: [PATCH 05/12] add restorear process entry with restorear.md. --- README.md | 3 +- docs/lifecycle.md | 2 +- docs/metadata.md | 2 +- docs/restorear.md | 32 ++++++++++++++ docs/type.md | 6 +++ .../java/com/qiniu/entry/QSuitsEntry.java | 42 ++++++++++++++++++- 6 files changed, 82 insertions(+), 5 deletions(-) create mode 100644 docs/restorear.md diff --git a/README.md b/README.md index d660fa63..af90c099 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,8 @@ qiniu-suits-java 是一个多线程的云存储 api tools (base-qiniu)，通过 - [x] 移动资源到指定空间 [move 配置](docs/move.md) - [x] 对空间资源进行重命名 [rename 配置](docs/rename.md) - [x] 查询空间资源的元信息 [stat 配置](docs/stat.md) -- [x] 修改空间资源的存储类型（低频/标准）[type 配置](docs/type.md) +- [x] 修改空间资源的存储类型（低频/标准/归档）[type 配置](docs/type.md) +- [x] 对空间的归档文件进行解冻 [restorar 配置](docs/restorear.md) - [x] 修改空间资源的状态（启用/禁用）[status 配置](docs/status.md) - [x] 修改空间资源的生命周期 [lifecycle 配置](docs/lifecycle.md) - [x] 对设置了镜像源的空间资源进行镜像更新或拉取 [mirror 配置](docs/mirror.md) diff --git a/docs/lifecycle.md b/docs/lifecycle.md index fe983762..a64337f3 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -9,7 +9,7 @@ ## 配置 ``` -process= +process=lifecycle ak= sk= bucket= diff --git a/docs/metadata.md b/docs/metadata.md index 1dca0dc1..0faa770b 100644 --- a/docs/metadata.md +++ b/docs/metadata.md @@ -22,7 +22,7 @@ cond.[]= |ak、sk|长度40的字符串|七牛账号的ak、sk，通过七牛控制台个人中心获取| |bucket| 字符串| 资源所在空间名称| |meta.[]| 字符串| metadata 的设置，全局设置，应用在所有文件上进行修改，支持任意自定义字段，也可以是常用的标准字段，如 meta.key1=value1, meta.Cache-Control=public, max-age=36000| -|cond.[]| 字符串| 可以设置一些修改操作时的条件 condition，cond 当前支持设置 hash、mime、fsize、putTime 条件，只有条件匹配才会执行修改操作，如 cond.mime=text/plain| +|cond.[]| 字符串| 可以设置一些修改操作时的 condition，cond 当前支持设置 hash、mime、fsize、putTime 条件，只有条件匹配才会执行修改操作，如 cond.mime=text/plain| ### 命令行参数方式 ``` diff --git a/docs/restorear.md b/docs/restorear.md new file mode 100644 index 00000000..60f62383 --- /dev/null +++ b/docs/restorear.md @@ -0,0 +1,32 @@ +# 解冻归档文件 + +## 简介 +对空间中的归档文件进行解冻。参考：[七牛归档文件解冻接口](https://developer.qiniu.com/kodo/api/6380/restore-archive) +1. **操作通常需要指定数据源，默认表示从七牛空间列举文件执行操作，如非默认或需更多条件，请先[配置数据源](datasource.md)** +2. 支持通过 `-a=`/`-d` 使用已设置的账号，则不需要再直接设置密钥，参考：[账号设置](../README.md#账号设置) +3. 单次修改一个文件请参考[ single 操作](single.md) +4. 交互式操作随时输入 key 进行修改请参考[ interactive 操作](interactive.md) +5. 如果要解冻的同时去转换为低频或者标准，可参考：[ type 操作](type.md) + +## 配置 +``` +process=restorear +ak= +sk= +bucket= +days= +cond.[]= +``` +|参数名|参数值及类型 | 含义| +|-----|-------|-----| +|process=lifecycle| 更新资源生命周期时设置为lifecycle| 表示更新生命周期操作| +|ak、sk|长度40的字符串|七牛账号的ak、sk，通过七牛控制台个人中心获取，当数据源为 qiniu 时无需再设置| +|bucket| 字符串| 操作的资源原空间，当数据源为 qiniu 时无需再设置| +|indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| +|days| 整型数字| 设置解冻的有效期为 days（单位天数），范围 1-7 天| +|cond.[]| 字符串| 可以设置一些操作时的 condition，cond 当前支持设置 hash、mime、fsize、putTime 条件，只有条件匹配才会执行修改操作，如 cond.mime=text/plain| + +### 命令行方式 +``` +-process=restorear -ak= -sk= -bucket= -days= +``` diff --git a/docs/type.md b/docs/type.md index 5d5b9918..0a05750d 100644 --- a/docs/type.md +++ b/docs/type.md @@ -15,6 +15,7 @@ sk= bucket= indexes= type= +archive= ``` |参数名|参数值及类型 | 含义| |-----|-------|-----| @@ -23,6 +24,11 @@ type= |bucket| 字符串| 操作的资源原空间，当数据源为 qiniu 时无需再设置| |indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| |type| 0/1/2| 设置资源的存储类型为 type，0 表示标准存储，1 表示低频存储，2 表示归档存储| +|archive| true/false| 为归档存储设计的选项，如果是要从归档文件转成标准或低频，请设置为 true，否则默认为 false| + +### 关于 archive +因为归档文件要求先解冻才能转换成其他类型，设置 `archive=true` 可以在执行转换之前触发解冻行为，进行归档文件解冻时还有两个选项：`days` 和 `cond.[]`， +用来设置进行解冻的有效期天数和触发解冻的条件，具体可参考[ restorear 操作](restorear.md) ### 命令行方式 ``` diff --git a/src/main/java/com/qiniu/entry/QSuitsEntry.java b/src/main/java/com/qiniu/entry/QSuitsEntry.java index 7cc66710..575041ef 100755 --- a/src/main/java/com/qiniu/entry/QSuitsEntry.java +++ b/src/main/java/com/qiniu/entry/QSuitsEntry.java @@ -516,6 +516,7 @@ public ILineProcess> whichNextProcessor(boolean single) thro switch (process) { case "status": processor = getChangeStatus(single); break; case "type": processor = getChangeType(single); break; + case "restorear": processor = getRestoreArchive(single); break; case "lifecycle": processor = getChangeLifecycle(single); break; case "copy": processor = getCopyFile(indexes, single); break; case "move": @@ -598,8 +599,45 @@ private ILineProcess> getChangeStatus(boolean single) throws private ILineProcess> getChangeType(boolean single) throws IOException { String type = entryParam.getValue("type").trim(); ParamsUtils.checked(type, "type", "\\d"); - return single ? new ChangeType(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, Integer.parseInt(type)) - : new ChangeType(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, Integer.parseInt(type), savePath); + ChangeType changeType = single ? new ChangeType(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, + Integer.parseInt(type)) : new ChangeType(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, + Integer.parseInt(type), savePath); + String archive = entryParam.getValue("archive", "false").trim(); + ParamsUtils.checked(archive, "archive", "(true|false)"); + if (Boolean.parseBoolean(archive)) { + String days = entryParam.getValue("days", "1").trim(); + ParamsUtils.checked(days, "days", "[1-7]"); + StringBuilder condition = new StringBuilder(); + for (Map.Entry entry : entryParam.getParamsMap().entrySet()) { + if (entry.getKey().startsWith("cond.")) { + if (condition.length() > 0) { + condition.append(entry.getKey().substring(5)).append("=").append(entry.getValue().trim()).append("&"); + } else { + condition.append(entry.getKey().substring(5)).append("=").append(entry.getValue().trim()); + } + } + } + changeType.enableRestoreArchive(Integer.parseInt(days), condition.toString()); + } + return changeType; + } + + private ILineProcess> getRestoreArchive(boolean single) throws IOException { + String days = entryParam.getValue("days", "1").trim(); + ParamsUtils.checked(days, "days", "[1-7]"); + StringBuilder condition = new StringBuilder(); + for (Map.Entry entry : entryParam.getParamsMap().entrySet()) { + if (entry.getKey().startsWith("cond.")) { + if (condition.length() > 0) { + condition.append(entry.getKey().substring(5)).append("=").append(entry.getValue().trim()).append("&"); + } else { + condition.append(entry.getKey().substring(5)).append("=").append(entry.getValue().trim()); + } + } + } + return single ? new RestoreArchive(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, Integer.parseInt(days), + condition.toString()) : new RestoreArchive(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, + Integer.parseInt(days), condition.toString(), savePath); } private ILineProcess> getChangeLifecycle(boolean single) throws IOException { From 61c7139ecb7b09acb52e088f4427dc5a6c62076a Mon Sep 17 00:00:00 2001 From: wubingheng Date: Wed, 19 Feb 2020 21:29:01 +0800 Subject: [PATCH 06/12] mainly update ChangeType to support type 2 and remove archive option. --- docs/restorear.md | 4 +- docs/type.md | 6 - .../java/com/qiniu/entry/QSuitsEntry.java | 20 +-- .../com/qiniu/interfaces/IFileCaller.java | 9 -- .../com/qiniu/process/qiniu/CdnUrlQuery.java | 5 +- .../qiniu/process/qiniu/ChangeMetadata.java | 18 +-- .../com/qiniu/process/qiniu/ChangeType.java | 119 ++++++++++-------- .../qiniu/process/qiniu/RestoreArchive.java | 22 +--- .../java/com/qiniu/util/CloudApiUtils.java | 2 + 9 files changed, 91 insertions(+), 114 deletions(-) delete mode 100644 src/main/java/com/qiniu/interfaces/IFileCaller.java diff --git a/docs/restorear.md b/docs/restorear.md index 60f62383..eaaeb48f 100644 --- a/docs/restorear.md +++ b/docs/restorear.md @@ -6,7 +6,7 @@ 2. 支持通过 `-a=`/`-d` 使用已设置的账号，则不需要再直接设置密钥，参考：[账号设置](../README.md#账号设置) 3. 单次修改一个文件请参考[ single 操作](single.md) 4. 交互式操作随时输入 key 进行修改请参考[ interactive 操作](interactive.md) -5. 如果要解冻的同时去转换为低频或者标准，可参考：[ type 操作](type.md) +5. 由于归档解冻是异步行为需要 1-5 分钟完成，不能立即调用转换为低频或标准的接口，如果要转换，需要单独使用[ type 操作](type.md) ## 配置 ``` @@ -19,7 +19,7 @@ cond.[]= ``` |参数名|参数值及类型 | 含义| |-----|-------|-----| -|process=lifecycle| 更新资源生命周期时设置为lifecycle| 表示更新生命周期操作| +|process=restorear| 对归档文件进行解冻时设置为 restorear| 表示归档文件解冻操作| |ak、sk|长度40的字符串|七牛账号的ak、sk，通过七牛控制台个人中心获取，当数据源为 qiniu 时无需再设置| |bucket| 字符串| 操作的资源原空间，当数据源为 qiniu 时无需再设置| |indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| diff --git a/docs/type.md b/docs/type.md index 0a05750d..5d5b9918 100644 --- a/docs/type.md +++ b/docs/type.md @@ -15,7 +15,6 @@ sk= bucket= indexes= type= -archive= ``` |参数名|参数值及类型 | 含义| |-----|-------|-----| @@ -24,11 +23,6 @@ archive= |bucket| 字符串| 操作的资源原空间，当数据源为 qiniu 时无需再设置| |indexes|字符串| 设置输入行中 key 字段的下标（有默认值），参考[数据源 indexes 设置](datasource.md#1-公共参数)| |type| 0/1/2| 设置资源的存储类型为 type，0 表示标准存储，1 表示低频存储，2 表示归档存储| -|archive| true/false| 为归档存储设计的选项，如果是要从归档文件转成标准或低频，请设置为 true，否则默认为 false| - -### 关于 archive -因为归档文件要求先解冻才能转换成其他类型，设置 `archive=true` 可以在执行转换之前触发解冻行为，进行归档文件解冻时还有两个选项：`days` 和 `cond.[]`， -用来设置进行解冻的有效期天数和触发解冻的条件，具体可参考[ restorear 操作](restorear.md) ### 命令行方式 ``` diff --git a/src/main/java/com/qiniu/entry/QSuitsEntry.java b/src/main/java/com/qiniu/entry/QSuitsEntry.java index 575041ef..e25999fd 100755 --- a/src/main/java/com/qiniu/entry/QSuitsEntry.java +++ b/src/main/java/com/qiniu/entry/QSuitsEntry.java @@ -599,27 +599,9 @@ private ILineProcess> getChangeStatus(boolean single) throws private ILineProcess> getChangeType(boolean single) throws IOException { String type = entryParam.getValue("type").trim(); ParamsUtils.checked(type, "type", "\\d"); - ChangeType changeType = single ? new ChangeType(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, + return single ? new ChangeType(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, Integer.parseInt(type)) : new ChangeType(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, Integer.parseInt(type), savePath); - String archive = entryParam.getValue("archive", "false").trim(); - ParamsUtils.checked(archive, "archive", "(true|false)"); - if (Boolean.parseBoolean(archive)) { - String days = entryParam.getValue("days", "1").trim(); - ParamsUtils.checked(days, "days", "[1-7]"); - StringBuilder condition = new StringBuilder(); - for (Map.Entry entry : entryParam.getParamsMap().entrySet()) { - if (entry.getKey().startsWith("cond.")) { - if (condition.length() > 0) { - condition.append(entry.getKey().substring(5)).append("=").append(entry.getValue().trim()).append("&"); - } else { - condition.append(entry.getKey().substring(5)).append("=").append(entry.getValue().trim()); - } - } - } - changeType.enableRestoreArchive(Integer.parseInt(days), condition.toString()); - } - return changeType; } private ILineProcess> getRestoreArchive(boolean single) throws IOException { diff --git a/src/main/java/com/qiniu/interfaces/IFileCaller.java b/src/main/java/com/qiniu/interfaces/IFileCaller.java deleted file mode 100644 index 9de7de12..00000000 --- a/src/main/java/com/qiniu/interfaces/IFileCaller.java +++ /dev/null @@ -1,9 +0,0 @@ -package com.qiniu.interfaces; - -import java.io.IOException; -import java.util.List; - -public interface IFileCaller { - - String call(List list) throws IOException; -} diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java index b1795cde..7ef15cfe 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java @@ -146,8 +146,9 @@ protected List> parseBatchResult(List> p } } } else { - fileSaveMapper.writeError(String.join("\n", processList.stream().map(this::resultInfo) - .collect(Collectors.toList())) + "\t" + code + refreshResult.get("error").getAsString(), false); + fileSaveMapper.writeError(String.join("\t", processList.stream().map(this::resultInfo) + .collect(Collectors.joining("\n")), String.valueOf(code), + refreshResult.get("error").getAsString()), false); } refreshResult = null; return null; diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java b/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java index c1dd90b3..4e4330cd 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java @@ -21,7 +21,6 @@ public class ChangeMetadata extends Base> { private Auth auth; private Configuration configuration; private Client client; - private static final String URL = "http://rs.qiniu.com/batch"; public ChangeMetadata(String accessKey, String secretKey, Configuration configuration, String bucket, Map metadata, String condition) throws IOException { @@ -57,12 +56,14 @@ public ChangeMetadata(String accessKey, String secretKey, Configuration configur @Override public ChangeMetadata clone() throws CloneNotSupportedException { - ChangeMetadata changeType = (ChangeMetadata)super.clone(); - changeType.ops = new ArrayList<>(); - changeType.lines = new ArrayList<>(); - changeType.auth = Auth.create(accessId, secretKey); - changeType.client = new Client(configuration.clone()); - return changeType; + ChangeMetadata changeMetadata = (ChangeMetadata)super.clone(); + if (fileSaveMapper == null) { + changeMetadata.ops = new ArrayList<>(); + changeMetadata.lines = new ArrayList<>(); + } + changeMetadata.auth = Auth.create(accessId, secretKey); + changeMetadata.client = new Client(configuration.clone()); + return changeMetadata; } @Override @@ -103,7 +104,8 @@ protected List> putBatchOperations(List> @Override protected String batchResult(List> lineList) throws IOException { byte[] body = StringUtils.utf8Bytes(StringUtils.join(ops, "&op=", "op=")); - return HttpRespUtils.getResult(client.post(URL, body, auth.authorization(URL, body, Client.FormMime), Client.FormMime)); + return HttpRespUtils.getResult(client.post(CloudApiUtils.QINIU_RS_BATCH_URL, body, + auth.authorization(CloudApiUtils.QINIU_RS_BATCH_URL, body, Client.FormMime), Client.FormMime)); } @Override diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeType.java b/src/main/java/com/qiniu/process/qiniu/ChangeType.java index 49e8ae35..19c80679 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeType.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeType.java @@ -1,16 +1,12 @@ package com.qiniu.process.qiniu; import com.qiniu.common.QiniuException; +import com.qiniu.http.Client; import com.qiniu.http.Response; -import com.qiniu.interfaces.IFileCaller; import com.qiniu.process.Base; -import com.qiniu.storage.BucketManager; -import com.qiniu.storage.BucketManager.*; import com.qiniu.storage.Configuration; -import com.qiniu.storage.model.StorageType; -import com.qiniu.util.Auth; -import com.qiniu.util.HttpRespUtils; -import com.qiniu.util.CloudApiUtils; +//import com.qiniu.storage.model.StorageType; +import com.qiniu.util.*; import java.io.IOException; import java.util.ArrayList; @@ -19,40 +15,42 @@ public class ChangeType extends Base> { - private StorageType storageType; - private BatchOperations batchOperations; +// private StorageType storageType; + private int type; + private ArrayList ops; private List> lines; private Auth auth; private Configuration configuration; - private BucketManager bucketManager; - private int days; - private String condition; - private RestoreArchive restoreArchive; - private IFileCaller> caller; + private Client client; +// private BucketManager bucketManager; - public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type) - throws IOException { + public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type) throws IOException { super("type", accessKey, secretKey, bucket); - storageType = type == 0 ? StorageType.COMMON : StorageType.INFREQUENCY; - this.auth = Auth.create(accessKey, secretKey); +// storageType = type == 0 ? StorageType.COMMON : StorageType.INFREQUENCY; + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); + this.type = type; this.configuration = configuration; - this.bucketManager = new BucketManager(auth, configuration); // BucketManager 中已经做了 configuration.clone() - CloudApiUtils.checkQiniu(bucketManager, bucket); - caller = list -> null; + this.auth = Auth.create(accessKey, secretKey); + this.client = new Client(configuration.clone()); +// this.bucketManager = new BucketManager(auth, configuration); +// CloudApiUtils.checkQiniu(bucketManager, bucket); } - public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type, - String savePath, int saveIndex) throws IOException { + public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type, String savePath, + int saveIndex) throws IOException { super("type", accessKey, secretKey, bucket, savePath, saveIndex); - storageType = type == 0 ? StorageType.COMMON : StorageType.INFREQUENCY; +// storageType = type == 0 ? StorageType.COMMON : StorageType.INFREQUENCY; + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); + this.type = type; this.batchSize = 1000; - this.batchOperations = new BatchOperations(); + this.ops = new ArrayList<>(); this.lines = new ArrayList<>(); this.configuration = configuration; this.auth = Auth.create(accessKey, secretKey); - this.bucketManager = new BucketManager(auth, configuration); - CloudApiUtils.checkQiniu(bucketManager, bucket); - caller = list -> null; + this.client = new Client(configuration.clone()); +// this.batchOperations = new BatchOperations(); +// this.bucketManager = new BucketManager(auth, configuration); +// CloudApiUtils.checkQiniu(bucketManager, bucket); } public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type, @@ -60,25 +58,17 @@ public ChangeType(String accessKey, String secretKey, Configuration configuratio this(accessKey, secretKey, configuration, bucket, type, savePath, 0); } - public void enableRestoreArchive(int days, String condition) { - this.days = days; - this.condition = condition; - restoreArchive = new RestoreArchive(auth, secretKey, configuration, bucket, days, condition); - caller = restoreArchive::batchResult; - if (batchSize > 100) batchSize = 100; // 因为 restore archive 操作一次只能 100 个 entry - } - @Override public ChangeType clone() throws CloneNotSupportedException { ChangeType changeType = (ChangeType)super.clone(); - changeType.auth = Auth.create(accessId, secretKey); - changeType.bucketManager = new BucketManager(changeType.auth, configuration); - changeType.batchOperations = new BatchOperations(); - changeType.lines = new ArrayList<>(); - if (restoreArchive != null) { - changeType.restoreArchive = new RestoreArchive(changeType.auth, secretKey, configuration, bucket, days, condition); - changeType.caller = restoreArchive::batchResult; +// changeType.bucketManager = new BucketManager(changeType.auth, configuration); +// changeType.batchOperations = new BatchOperations(); + if (fileSaveMapper != null) { + changeType.ops = new ArrayList<>(); + changeType.lines = new ArrayList<>(); } + changeType.auth = Auth.create(accessId, secretKey); + changeType.client = new Client(configuration.clone()); return changeType; } @@ -89,15 +79,33 @@ protected String resultInfo(Map line) { @Override protected List> putBatchOperations(List> processList) throws IOException { - batchOperations.clearOps(); +// batchOperations.clearOps(); +// lines.clear(); +// String key; +// for (Map map : processList) { +// key = map.get("key"); +// if (key != null) { +// lines.add(map); +// batchOperations.addChangeTypeOps(bucket, storageType, key); +// } else { +// fileSaveMapper.writeError("key is not exists or empty in " + map, false); +// } +// } +// return lines; + ops.clear(); lines.clear(); String key; - caller.call(processList); +// String encodedMetaValue; +// String path; + StringBuilder pathBuilder; for (Map map : processList) { key = map.get("key"); if (key != null) { lines.add(map); - batchOperations.addChangeTypeOps(bucket, storageType, key); + pathBuilder = new StringBuilder("/chtype/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))) + .append("/type/").append(type); + ops.add(pathBuilder.toString()); } else { fileSaveMapper.writeError("key is not exists or empty in " + map, false); } @@ -107,15 +115,21 @@ protected List> putBatchOperations(List> @Override protected String batchResult(List> lineList) throws IOException { - return HttpRespUtils.getResult(bucketManager.batch(batchOperations)); + byte[] body = StringUtils.utf8Bytes(StringUtils.join(ops, "&op=", "op=")); + return HttpRespUtils.getResult(client.post(CloudApiUtils.QINIU_RS_BATCH_URL, body, + auth.authorization(CloudApiUtils.QINIU_RS_BATCH_URL, body, Client.FormMime), Client.FormMime)); } @Override protected String singleResult(Map line) throws IOException { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); - if (restoreArchive != null) restoreArchive.singleResult(line); - Response response = bucketManager.changeType(bucket, key, storageType); +// String path = String.format("/chgm/%s", BucketManager.encodedEntry(bucket, key)); + StringBuilder urlBuilder = new StringBuilder("http://rs.qiniu.com/chtype/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))); + urlBuilder.append("/type/").append(type); + StringMap headers = auth.authorization(urlBuilder.toString(), null, Client.FormMime); + Response response = client.post(urlBuilder.toString(), null, headers, Client.FormMime); if (response.statusCode != 200) throw new QiniuException(response); response.close(); return String.join("\t", key, "200"); @@ -124,11 +138,12 @@ protected String singleResult(Map line) throws IOException { @Override public void closeResource() { super.closeResource(); - batchOperations = null; + if (ops != null) ops.clear(); + ops = null; + if (lines != null) lines.clear(); lines = null; auth = null; configuration = null; - bucketManager = null; - caller = null; + client = null; } } diff --git a/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java b/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java index 6302f0a0..632564eb 100755 --- a/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java +++ b/src/main/java/com/qiniu/process/qiniu/RestoreArchive.java @@ -17,7 +17,7 @@ public class RestoreArchive extends Base> { private int days; - private String condition; + private String encodedCondition; private Configuration configuration; private Auth auth; private Client client; @@ -27,7 +27,7 @@ public RestoreArchive(String accessKey, String secretKey, Configuration configur String condition) throws IOException { super("restorear", accessKey, secretKey, bucket); this.days = days; - this.condition = condition; + if (condition != null && !condition.isEmpty()) encodedCondition = UrlSafeBase64.encodeToString(condition); this.configuration = configuration; this.auth = Auth.create(accessKey, secretKey); this.client = new Client(configuration.clone()); @@ -39,7 +39,7 @@ public RestoreArchive(String accessKey, String secretKey, Configuration configur String condition, String savePath, int saveIndex) throws IOException { super("restorear", accessKey, secretKey, bucket, savePath, saveIndex); this.days = days; - this.condition = condition; + if (condition != null && !condition.isEmpty()) encodedCondition = UrlSafeBase64.encodeToString(condition); this.configuration = configuration; this.auth = Auth.create(accessKey, secretKey); this.client = new Client(configuration.clone()); @@ -52,16 +52,6 @@ public RestoreArchive(String accessKey, String secretKey, Configuration configur this(accessKey, secretKey, configuration, bucket, days, condition, savePath, 0); } - RestoreArchive(Auth auth, String secretKey, Configuration configuration, String bucket, int days, String condition) { - super("restorear", auth.accessKey, secretKey, bucket); - this.auth = auth; - this.days = days; - this.condition = condition; - this.configuration = configuration; - this.client = new Client(configuration.clone()); - requestUrl = configuration.useHttpsDomains ? "https://rs.qbox.me/restoreAr" : "http://rs.qbox.me/restoreAr"; - } - @Override public RestoreArchive clone() throws CloneNotSupportedException { RestoreArchive restoreArchive = (RestoreArchive)super.clone(); @@ -84,7 +74,7 @@ protected String batchResult(List> lineList) throws IOExcept for (Map line : lineList) { entry.addProperty("key", line.get("key")); entry.addProperty("freeze_after_days", days); - entry.addProperty("cond", condition); + entry.addProperty("cond", encodedCondition); entries.add(entry); } bodyJson.add("entries", entries); @@ -117,7 +107,7 @@ protected String singleResult(Map line) throws IOException { String key = line.get("key"); entry.addProperty("key", key); entry.addProperty("freeze_after_days", days); - entry.addProperty("cond", condition); + entry.addProperty("cond", encodedCondition); entries.add(entry); bodyJson.add("entries", entries); byte[] body = bodyJson.toString().getBytes(); @@ -135,7 +125,7 @@ protected String singleResult(Map line) throws IOException { @Override public void closeResource() { super.closeResource(); - condition = null; + encodedCondition = null; configuration = null; auth = null; client = null; diff --git a/src/main/java/com/qiniu/util/CloudApiUtils.java b/src/main/java/com/qiniu/util/CloudApiUtils.java index e60b293e..c4835b4d 100755 --- a/src/main/java/com/qiniu/util/CloudApiUtils.java +++ b/src/main/java/com/qiniu/util/CloudApiUtils.java @@ -73,6 +73,8 @@ public final class CloudApiUtils { put(LOCAL, TYPE_File); }}; + public static final String QINIU_RS_BATCH_URL = "http://rs.qiniu.com/batch"; + public static String getSourceType(String source) { return datasourceMap.get(source); } From 1d05fed7f31b23087826a6349dd5edbeeccc3e60 Mon Sep 17 00:00:00 2001 From: wubingheng Date: Thu, 20 Feb 2020 10:34:56 +0800 Subject: [PATCH 07/12] update CdnUrlQuery. --- .../java/com/qiniu/entry/QSuitsEntry.java | 6 ++ .../com/qiniu/process/qiniu/CdnHelper.java | 24 ++++++- .../qiniu/process/qiniu/CdnUrlProcess.java | 23 +++--- .../com/qiniu/process/qiniu/CdnUrlQuery.java | 70 ++++++++++++++----- .../qiniu/process/qiniu/CdnHelperTest.java | 14 ++-- 5 files changed, 106 insertions(+), 31 deletions(-) diff --git a/src/main/java/com/qiniu/entry/QSuitsEntry.java b/src/main/java/com/qiniu/entry/QSuitsEntry.java index e25999fd..118a090c 100755 --- a/src/main/java/com/qiniu/entry/QSuitsEntry.java +++ b/src/main/java/com/qiniu/entry/QSuitsEntry.java @@ -1127,6 +1127,12 @@ private ILineProcess> getRefreshQuery(Map in return single ? new CdnUrlQuery(qiniuAccessKey, qiniuSecretKey, getNewQiniuConfig(), protocol, domain, urlIndex, false) : new CdnUrlQuery(qiniuAccessKey, qiniuSecretKey, getNewQiniuConfig(), protocol, domain, urlIndex, false, savePath); +// String pageNo = entryParam.getValue("pageNo", "").trim(); +// String pageSize = entryParam.getValue("pageSize", "").trim(); +// String startTime = entryParam.getValue("startTime", "").trim(); +// ParamsUtils.checked(startTime, "startTime", "\\d{4}-[01]\\d-[0-3]\\d"); +// String endTime = entryParam.getValue("endTime", "").trim(); +// ParamsUtils.checked(endTime, "endTime", "\\d{4}-[01]\\d-[0-3]\\d"); } private ILineProcess> getPrefetchQuery(Map indexMap, boolean single) throws IOException { diff --git a/src/main/java/com/qiniu/process/qiniu/CdnHelper.java b/src/main/java/com/qiniu/process/qiniu/CdnHelper.java index 8179b473..ac0e5ce9 100644 --- a/src/main/java/com/qiniu/process/qiniu/CdnHelper.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnHelper.java @@ -1,5 +1,7 @@ package com.qiniu.process.qiniu; +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; import com.qiniu.common.Constants; import com.qiniu.common.QiniuException; import com.qiniu.http.Client; @@ -38,6 +40,16 @@ public Response queryRefresh(String[] urls) throws QiniuException { return UOperationForUrls(refreshQueryUrl, req); } + public Response queryRefresh(JsonArray urls, int pageNo, int pageSize, String startTime, String endTime) throws QiniuException { + JsonObject req = new JsonObject(); + req.add("urls", urls); + req.addProperty("pageNo", pageNo); + req.addProperty("pageSize", pageSize); + req.addProperty("startTime", startTime); + req.addProperty("endTime", endTime); + return UOperationForUrls(refreshQueryUrl, req); + } + public Response prefetch(String[] urls) throws QiniuException { Map req = new HashMap<>(); req.put("urls", urls); @@ -50,7 +62,17 @@ public Response queryPrefetch(String[] urls) throws QiniuException { return UOperationForUrls(prefetchQueryUrl, req); } - private Response UOperationForUrls(String apiUrl, Map req) throws QiniuException { + public Response queryPrefetch(JsonArray urls, int pageNo, int pageSize, String startTime, String endTime) throws QiniuException { + JsonObject req = new JsonObject(); + req.add("urls", urls); + req.addProperty("pageNo", pageNo); + req.addProperty("pageSize", pageSize); + req.addProperty("startTime", startTime); + req.addProperty("endTime", endTime); + return UOperationForUrls(prefetchQueryUrl, req); + } + + private Response UOperationForUrls(String apiUrl, Object req) throws QiniuException { byte[] body = Json.encode(req).getBytes(Constants.UTF_8); return client.post(apiUrl, body, auth.authorizationV2(apiUrl, "POST", body, Client.JsonMime), Client.JsonMime); } diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java index f907a030..399fd8d3 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java @@ -11,6 +11,7 @@ import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.List; import java.util.Map; import java.util.stream.Collectors; @@ -22,7 +23,7 @@ public class CdnUrlProcess extends Base> { private String urlIndex; private boolean isDir; // private boolean prefetch; - private List batches; + private String[] batches; private List> lines; // private Configuration configuration; private ICdnApplier cdnApplier; @@ -43,7 +44,7 @@ public CdnUrlProcess(String accessKey, String secretKey, Configuration configura String urlIndex, boolean isDir, boolean prefetch, String savePath, int saveIndex) throws IOException { super(prefetch ? "cdnprefetch" : "cdnrefresh", accessKey, secretKey, null, savePath, saveIndex); this.batchSize = 30; - this.batches = new ArrayList<>(30); + this.batches = new String[30]; this.lines = new ArrayList<>(); Auth auth = Auth.create(accessKey, secretKey); CdnHelper cdnHelper = new CdnHelper(auth, configuration); @@ -99,22 +100,26 @@ protected String resultInfo(Map line) { @Override protected synchronized List> putBatchOperations(List> processList) throws IOException { - batches.clear(); + Arrays.fill(batches, null); lines.clear(); + Map line; if (domain == null) { - for (Map line : processList) { + for (int i = 0; i < processList.size(); i++) { + line = processList.get(i); lines.add(line); - batches.add(line.get(urlIndex)); + batches[i] = line.get(urlIndex); } } else { String key; - for (Map line : processList) { + for (int i = 0; i < processList.size(); i++) { + line = processList.get(i); key = line.get("key"); if (key == null) { fileSaveMapper.writeError("key and url are not exist or empty in " + line, false); } else { lines.add(line); - batches.add(String.join("", protocol, "://", domain, "/", key.replace("\\?", "%3f"))); + batches[i] = String.join("", protocol, "://", domain, "/", + key.replace("\\?", "%3f")); } } } @@ -123,8 +128,7 @@ protected synchronized List> putBatchOperations(List> lineList) throws IOException { - String[] urls = new String[batches.size()]; - return HttpRespUtils.getResult(cdnApplier.apply(batches.toArray(urls))); + return HttpRespUtils.getResult(cdnApplier.apply(batches)); } @Override @@ -179,7 +183,6 @@ public void closeResource() { batches = null; lines = null; cdnApplier = null; - saveIndex = null; } else { saveIndex.decrementAndGet(); } diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java index 7ef15cfe..416cc119 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java @@ -11,6 +11,7 @@ import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.List; import java.util.Map; import java.util.stream.Collectors; @@ -21,16 +22,22 @@ public class CdnUrlQuery extends Base> { private String domain; private String urlIndex; private boolean prefetch; - private List batches; + private String[] batches; + private boolean hasOption; + private int pageNo; + private int pageSize; + private String startTime; + private String endTime; + private Configuration configuration; private List> lines; + private CdnHelper cdnHelper; private ICdnApplier cdnApplier; - private Configuration configuration; public CdnUrlQuery(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, String urlIndex, boolean prefetch) throws IOException { super(prefetch ? "prefetchquery" : "refreshquery", accessKey, secretKey, null); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration); + cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; CloudApiUtils.checkQiniu(auth); set(configuration, protocol, domain, urlIndex, prefetch); @@ -40,10 +47,10 @@ public CdnUrlQuery(String accessKey, String secretKey, Configuration configurati String urlIndex, boolean prefetch, String savePath, int saveIndex) throws IOException { super(prefetch ? "prefetchquery" : "refreshquery", accessKey, secretKey, null, savePath, saveIndex); this.batchSize = 30; - this.batches = new ArrayList<>(30); + this.batches = new String[30]; this.lines = new ArrayList<>(); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration); + cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; CloudApiUtils.checkQiniu(auth); set(configuration, protocol, domain, urlIndex, prefetch); @@ -72,18 +79,45 @@ public void set(Configuration configuration, String protocol, String domain, Str this.prefetch = prefetch; } + public void setQueryOptions(int pageNo, int pageSize, String startTime, String endTime) { + this.pageNo = pageNo; + this.pageSize = pageSize; + this.startTime = startTime; + this.endTime = endTime; + this.hasOption = true; + this.cdnApplier = prefetch ? urls -> { + JsonArray urlArray = new JsonArray(); + for (String url : urls) urlArray.add(url); + return cdnHelper.queryPrefetch(urlArray, pageNo, pageSize, startTime, endTime); + } : urls -> { + JsonArray urlArray = new JsonArray(); + for (String url : urls) urlArray.add(url); + return cdnHelper.queryRefresh(urlArray, pageNo, pageSize, startTime, endTime); + }; + } + @Override public CdnUrlQuery clone() throws CloneNotSupportedException { CdnUrlQuery cdnUrlQuery = (CdnUrlQuery)super.clone(); cdnUrlQuery.lines = new ArrayList<>(); - CdnHelper cdnHelper = new CdnHelper(Auth.create(accessId, secretKey), configuration); - cdnUrlQuery.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; + cdnUrlQuery.cdnHelper = new CdnHelper(Auth.create(accessId, secretKey), configuration); + if (hasOption) { + cdnUrlQuery.cdnApplier = prefetch ? urls -> { + JsonArray urlArray = new JsonArray(); + for (String url : urls) urlArray.add(url); + return cdnUrlQuery.cdnHelper.queryPrefetch(urlArray, pageNo, pageSize, startTime, endTime); + } : urls -> { + JsonArray urlArray = new JsonArray(); + for (String url : urls) urlArray.add(url); + return cdnUrlQuery.cdnHelper.queryRefresh(urlArray, pageNo, pageSize, startTime, endTime); + }; + } else { + cdnUrlQuery.cdnApplier = prefetch ? cdnUrlQuery.cdnHelper::queryPrefetch : cdnUrlQuery.cdnHelper::queryRefresh; + } if (cdnUrlQuery.fileSaveMapper != null) { cdnUrlQuery.fileSaveMapper.preAddWriter("processing"); } return cdnUrlQuery; -// if (!autoIncrease) saveIndex.addAndGet(1); -// return this; } @Override @@ -93,22 +127,26 @@ protected String resultInfo(Map line) { @Override protected synchronized List> putBatchOperations(List> processList) throws IOException { - batches.clear(); + Arrays.fill(batches, null); lines.clear(); + Map line; if (domain == null) { - for (Map line : processList) { + for (int i = 0; i < processList.size(); i++) { + line = processList.get(i); lines.add(line); - batches.add(line.get(urlIndex)); + batches[i] = line.get(urlIndex); } } else { String key; - for (Map line : processList) { + for (int i = 0; i < processList.size(); i++) { + line = processList.get(i); key = line.get("key"); if (key == null) { fileSaveMapper.writeError("key and url are not exist or empty in " + line, false); } else { lines.add(line); - batches.add(String.join("", protocol, "://", domain, "/", key.replace("\\?", "%3f"))); + batches[i] = String.join("", protocol, "://", domain, "/", + key.replace("\\?", "%3f")); } } } @@ -117,8 +155,7 @@ protected synchronized List> putBatchOperations(List> lineList) throws IOException { - String[] urls = new String[batches.size()]; - return HttpRespUtils.getResult(cdnApplier.apply(batches.toArray(urls))); + return HttpRespUtils.getResult(cdnApplier.apply(batches)); } @Override @@ -178,6 +215,7 @@ public void closeResource() { urlIndex = null; batches = null; lines = null; + cdnHelper = null; cdnApplier = null; saveIndex = null; } diff --git a/src/test/java/com/qiniu/process/qiniu/CdnHelperTest.java b/src/test/java/com/qiniu/process/qiniu/CdnHelperTest.java index edf753f8..6d1a3045 100644 --- a/src/test/java/com/qiniu/process/qiniu/CdnHelperTest.java +++ b/src/test/java/com/qiniu/process/qiniu/CdnHelperTest.java @@ -1,5 +1,6 @@ package com.qiniu.process.qiniu; +import com.google.gson.JsonArray; import com.qiniu.config.PropertiesFile; import com.qiniu.http.Response; import com.qiniu.storage.Configuration; @@ -20,11 +21,16 @@ public void queryRefresh() throws IOException { String secretKey = propertiesFile.getValue("sk"); Auth auth = Auth.create(accessKey, secretKey); CdnHelper cdnHelper = new CdnHelper(auth, new Configuration()); - Response response = cdnHelper.queryRefresh(new String[]{ - "http://xxx.com/2.mp3", +// Response response = cdnHelper.queryRefresh(new String[]{ +// "http://xxx.com/2.mp3", // "http://xxx.com/181224-观点峰会-1.mp3", - "http://xxx.com/VoiceLibrary_MTQ2Njg2Mw" - }); +// "http://xxx.com/VoiceLibrary_MTQ2Njg2Mw" +// }); + JsonArray urls = new JsonArray(); + urls.add("http://xxx.com/2.mp3"); + urls.add("http://xxx.com/181224-观点峰会-1.mp3"); + urls.add("http://xxx.com/VoiceLibrary_MTQ2Njg2Mw"); + Response response = cdnHelper.queryRefresh(urls, 1, 1, "2020-02-15", "2020-02-19"); System.out.println(response.bodyString()); } } \ No newline at end of file From 9a237f947436fb8e507048c55614d19d9f4ef800 Mon Sep 17 00:00:00 2001 From: wubingheng Date: Thu, 20 Feb 2020 12:45:57 +0800 Subject: [PATCH 08/12] optimize CopyFile and MoveFile and support file force rewrite option. --- docs/copy.md | 8 +- docs/interactive.md | 2 +- docs/move.md | 8 +- docs/rename.md | 12 +- docs/single.md | 2 +- .../java/com/qiniu/entry/QSuitsEntry.java | 10 +- .../qiniu/process/qiniu/ChangeMetadata.java | 8 +- .../com/qiniu/process/qiniu/ChangeMime.java | 17 +- .../com/qiniu/process/qiniu/ChangeType.java | 17 +- .../com/qiniu/process/qiniu/CopyFile.java | 150 ++++++++++----- .../com/qiniu/process/qiniu/MoveFile.java | 177 ++++++++++-------- 11 files changed, 240 insertions(+), 171 deletions(-) diff --git a/docs/copy.md b/docs/copy.md index 7fa8546e..f758482c 100644 --- a/docs/copy.md +++ b/docs/copy.md @@ -18,6 +18,7 @@ to-bucket= toKey-index= add-prefix= rm-prefix= +force= ``` |参数名|参数值及类型 | 含义| |-----|-------|-----| @@ -29,12 +30,11 @@ rm-prefix= |toKey-index| 字符串| copy 操作可选择设置的目标文件名索引（下标），需要手动指定才会解析| |add-prefix| 字符串| 表示为保存的文件名添加指定前缀| |rm-prefix| 字符串| 表示将原文件名去除存在的指定前缀后作为 copy 之后保存的文件名| +|force| true/false| 目标文件名如果目标空间中已存在是否进行强制覆盖，默认为 false| ### 关于 toKey-index -指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，但未设置且 add-prefix 不为空 -时需要强制指定 prefix-force=true，表明该次重命名操作只添加文件名前缀。由于 rename 操作既需要原始文件名字段也需要新文件名字段，因此 toKey 下 -标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) -及[关于 parse 和索引](datasource.md#关于-parse)。 +指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，toKey 下标和 key 字段下标不可 +相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)及[关于 parse 和索引](datasource.md#关于-parse)。 **注意**：七牛存储空间不支持文件名以 `../`, `./` 开头或者包含 `/../`, `/./` 这种情况，会造成无法访问，因此设置文件名时请注意。 ### 命令行方式 diff --git a/docs/interactive.md b/docs/interactive.md index df6872b1..ddcafb49 100644 --- a/docs/interactive.md +++ b/docs/interactive.md @@ -37,7 +37,7 @@ please input line data to process: ``` ##### 4 对指定空间的资源进行重命名 [rename](rename.md) ``` -➜ ~ qsuits -i -d -process=rename -bucket=temp -add-prefix=1 -prefix-force=true +➜ ~ qsuits -i -d -process=rename -bucket=temp -add-prefix=1 please input line data to process: 10.mp4 10.mp4 10.mp4 diff --git a/docs/move.md b/docs/move.md index 3fe802d9..84d78994 100644 --- a/docs/move.md +++ b/docs/move.md @@ -17,6 +17,7 @@ to-bucket= toKey-index= add-prefix= rm-prefix= +force= ``` |参数名|参数值及类型 | 含义| |-----|-------|-----| @@ -27,12 +28,11 @@ rm-prefix= |toKey-index| 字符串| move 操作所需要设置的目标文件名索引（下标），需要手动指定才会解析| |add-prefix| 字符串| 表示为保存的文件名添加指定前缀| |rm-prefix| 字符串| 表示将原文件名去除存在的指定前缀后作为 move 之后保存的文件名| +|force| true/false| 目标文件名如果目标空间中已存在是否进行强制覆盖，默认为 false| ### 关于 toKey-index -指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，但未设置且 add-prefix 不为空 -时需要强制指定 prefix-force=true，表明该次重命名操作只添加文件名前缀。由于 rename 操作既需要原始文件名字段也需要新文件名字段，因此 toKey 下 -标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) -及[关于 parse 和索引](datasource.md#关于-parse)。 +指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，toKey 下标和 key 字段下标不可 +相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)及[关于 parse 和索引](datasource.md#关于-parse)。 **注意**：七牛存储空间不支持文件名以 `../`, `./` 开头或者包含 `/../`, `/./` 这种情况，会造成无法访问，因此设置文件名时请注意。 ### 命令行方式 diff --git a/docs/rename.md b/docs/rename.md index 9c495f93..3a498a67 100644 --- a/docs/rename.md +++ b/docs/rename.md @@ -17,7 +17,7 @@ indexes= toKey-index= add-prefix= rm-prefix= -prefix-force= +force= ``` |参数名|参数值及类型 | 含义| |-----|-------|-----| @@ -28,16 +28,14 @@ prefix-force= |toKey-index| 字符串| rename 操作所需要设置的目标文件名索引（下标），需要手动指定才会解析| |add-prefix| 字符串| 表示为保存的文件名添加指定前缀| |rm-prefix| 字符串| 表示将原文件名去除存在的指定前缀后作为 rename 之后保存的文件名| -|prefix-force| 字符串| 设置了 add-prefix/rm-prefix 但未设置 toKey-index 的情况下需要保证该参数为 true| +|force| true/false| 修改后的文件名如果空间中已存在是否进行强制覆盖，默认为 false| ### 关于 toKey-index -指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，但未设置且 add-prefix 不为空 -时需要强制指定 prefix-force=true，表明该次重命名操作只添加文件名前缀。由于 rename 操作既需要原始文件名字段也需要新文件名字段，因此 toKey 下 -标和 key 字段下标不可相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引) -及[关于 parse 和索引](datasource.md#关于-parse)。 +指定输入行中对应修改之后的文件名字段下标，不设置则无法进行解析，当使用 file 源且 parse=tab/csv 时下标必须为整数，toKey 下标和 key 字段下标不可 +相同，key 下标用 indexes 参数设置，默认会根据 parse 类型设置为 0 或 "key"，参见[ indexes 索引](datasource.md#关于-indexes-索引)及[关于 parse 和索引](datasource.md#关于-parse)。 **注意**：七牛存储空间不支持文件名以 `../`, `./` 开头或者包含 `/../`, `/./` 这种情况，会造成无法访问，因此设置文件名时请注意。 ### 命令行方式 ``` --process=rename -ak= -sk= -bucket= -toKey-index= -add-prefix= -rm-prefix= -prefix-force= +-process=rename -ak= -sk= -bucket= -toKey-index= -add-prefix= -rm-prefix= ``` diff --git a/docs/single.md b/docs/single.md index 1bbb2775..e5c2622d 100644 --- a/docs/single.md +++ b/docs/single.md @@ -30,7 +30,7 @@ process 提供的参数用法，具体可参见 process 的文档）包括要处 ``` ##### 4 对指定空间的资源进行重命名 [rename](rename.md) ``` -➜ ~ qsuits -s -d -process=rename -bucket=temp -add-prefix=1 -prefix-force=true -key=10.mp4 +➜ ~ qsuits -s -d -process=rename -bucket=temp -add-prefix=1 -key=10.mp4 10.mp4 10.mp4 ``` ##### 5 查询空间资源的元信息 [stat](stat.md) diff --git a/src/main/java/com/qiniu/entry/QSuitsEntry.java b/src/main/java/com/qiniu/entry/QSuitsEntry.java index 118a090c..0ea69272 100755 --- a/src/main/java/com/qiniu/entry/QSuitsEntry.java +++ b/src/main/java/com/qiniu/entry/QSuitsEntry.java @@ -634,10 +634,12 @@ private ILineProcess> getCopyFile(Map indexM String toKeyIndex = indexMap.containsValue("toKey") ? "toKey" : null; String addPrefix = entryParam.getValue("add-prefix", null); String rmPrefix = entryParam.getValue("rm-prefix", null); + String force = entryParam.getValue("force", "false").trim(); + ParamsUtils.checked(force, "force", "(true|false)"); return single ? new CopyFile(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, toBucket, toKeyIndex, addPrefix, - rmPrefix) + rmPrefix, Boolean.parseBoolean(force)) : new CopyFile(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, toBucket, toKeyIndex, addPrefix, - rmPrefix, savePath); + rmPrefix, Boolean.parseBoolean(force), savePath); } private ILineProcess> getMoveFile(Map indexMap, boolean single) throws IOException { @@ -646,8 +648,8 @@ private ILineProcess> getMoveFile(Map indexM String toKeyIndex = indexMap.containsValue("toKey") ? "toKey" : null; String addPrefix = entryParam.getValue("add-prefix", null); String rmPrefix = entryParam.getValue("rm-prefix", null); - String force = entryParam.getValue("prefix-force", "false").trim(); - ParamsUtils.checked(force, "prefix-force", "(true|false)"); + String force = entryParam.getValue("force", "false").trim(); + ParamsUtils.checked(force, "force", "(true|false)"); return single ? new MoveFile(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, toBucket, toKeyIndex, addPrefix, rmPrefix, Boolean.parseBoolean(force)) : new MoveFile(qiniuAccessKey, qiniuSecretKey, getQiniuConfig(), bucket, toBucket, toKeyIndex, addPrefix, diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java b/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java index 4e4330cd..2111af04 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeMetadata.java @@ -41,8 +41,8 @@ public ChangeMetadata(String accessKey, String secretKey, Configuration configur this.metadata = metadata; if (condition != null && !condition.isEmpty()) encodedCondition = UrlSafeBase64.encodeToString(condition); this.batchSize = 1000; - this.ops = new ArrayList<>(); - this.lines = new ArrayList<>(); + this.ops = new ArrayList<>(1000); + this.lines = new ArrayList<>(1000); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); this.auth = Auth.create(accessKey, secretKey); this.configuration = configuration; @@ -58,8 +58,8 @@ public ChangeMetadata(String accessKey, String secretKey, Configuration configur public ChangeMetadata clone() throws CloneNotSupportedException { ChangeMetadata changeMetadata = (ChangeMetadata)super.clone(); if (fileSaveMapper == null) { - changeMetadata.ops = new ArrayList<>(); - changeMetadata.lines = new ArrayList<>(); + changeMetadata.ops = new ArrayList<>(batchSize); + changeMetadata.lines = new ArrayList<>(batchSize); } changeMetadata.auth = Auth.create(accessId, secretKey); changeMetadata.client = new Client(configuration.clone()); diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeMime.java b/src/main/java/com/qiniu/process/qiniu/ChangeMime.java index e7c41736..4bd9d347 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeMime.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeMime.java @@ -47,8 +47,8 @@ public ChangeMime(String accessKey, String secretKey, Configuration configuratio this.mimeIndex = mimeIndex == null ? "mime" : mimeIndex; if (condition != null && !condition.isEmpty()) encodedCondition = UrlSafeBase64.encodeToString(condition); this.batchSize = 1000; - this.ops = new ArrayList<>(); - this.lines = new ArrayList<>(); + this.ops = new ArrayList<>(1000); + this.lines = new ArrayList<>(1000); CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); this.auth = Auth.create(accessKey, secretKey); this.configuration = configuration; @@ -65,8 +65,10 @@ public ChangeMime clone() throws CloneNotSupportedException { ChangeMime changeType = (ChangeMime)super.clone(); changeType.auth = Auth.create(accessId, secretKey); changeType.client = new Client(configuration.clone()); - changeType.ops = new ArrayList<>(); - changeType.lines = new ArrayList<>(); + if (fileSaveMapper != null) { + changeType.ops = new ArrayList<>(batchSize); + changeType.lines = new ArrayList<>(batchSize); + } return changeType; } @@ -97,7 +99,6 @@ protected List> putBatchOperations(List> .append("/mime/").append(UrlSafeBase64.encodeToString(mime)); if (encodedCondition != null) pathBuilder.append("/cond/").append(encodedCondition); ops.add(pathBuilder.toString()); -// batchOperations.addChgmOp(bucket, mime, key); } else { fileSaveMapper.writeError("key or mime is not exists or empty in " + map, false); } @@ -112,7 +113,6 @@ protected List> putBatchOperations(List> .append("/mime/").append(encodedMime); if (encodedCondition != null) pathBuilder.append("/cond/").append(encodedCondition); ops.add(pathBuilder.toString()); -// batchOperations.addChgmOp(bucket, mimeType, key); } else { fileSaveMapper.writeError("key is not exists or empty in " + map, false); } @@ -125,7 +125,6 @@ protected List> putBatchOperations(List> protected String batchResult(List> lineList) throws IOException { byte[] body = StringUtils.utf8Bytes(StringUtils.join(ops, "&op=", "op=")); return HttpRespUtils.getResult(client.post(URL, body, auth.authorization(URL, body, Client.FormMime), Client.FormMime)); -// return HttpRespUtils.getResult(bucketManager.batch(batchOperations)); } @Override @@ -164,13 +163,11 @@ public void closeResource() { mimeType = null; mimeIndex = null; encodedCondition = null; -// batchOperations = null; if (ops != null) ops.clear(); ops = null; if (lines != null) lines.clear(); auth = null; - configuration = null; client = null; -// bucketManager = null; + configuration = null; } } diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeType.java b/src/main/java/com/qiniu/process/qiniu/ChangeType.java index 19c80679..5991c28a 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeType.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeType.java @@ -19,9 +19,9 @@ public class ChangeType extends Base> { private int type; private ArrayList ops; private List> lines; - private Auth auth; private Configuration configuration; private Client client; + private Auth auth; // private BucketManager bucketManager; public ChangeType(String accessKey, String secretKey, Configuration configuration, String bucket, int type) throws IOException { @@ -43,8 +43,8 @@ public ChangeType(String accessKey, String secretKey, Configuration configuratio CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); this.type = type; this.batchSize = 1000; - this.ops = new ArrayList<>(); - this.lines = new ArrayList<>(); + this.ops = new ArrayList<>(1000); + this.lines = new ArrayList<>(1000); this.configuration = configuration; this.auth = Auth.create(accessKey, secretKey); this.client = new Client(configuration.clone()); @@ -64,8 +64,8 @@ public ChangeType clone() throws CloneNotSupportedException { // changeType.bucketManager = new BucketManager(changeType.auth, configuration); // changeType.batchOperations = new BatchOperations(); if (fileSaveMapper != null) { - changeType.ops = new ArrayList<>(); - changeType.lines = new ArrayList<>(); + changeType.ops = new ArrayList<>(batchSize); + changeType.lines = new ArrayList<>(batchSize); } changeType.auth = Auth.create(accessId, secretKey); changeType.client = new Client(configuration.clone()); @@ -124,10 +124,9 @@ protected String batchResult(List> lineList) throws IOExcept protected String singleResult(Map line) throws IOException { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); -// String path = String.format("/chgm/%s", BucketManager.encodedEntry(bucket, key)); StringBuilder urlBuilder = new StringBuilder("http://rs.qiniu.com/chtype/") - .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))); - urlBuilder.append("/type/").append(type); + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))) + .append("/type/").append(type); StringMap headers = auth.authorization(urlBuilder.toString(), null, Client.FormMime); Response response = client.post(urlBuilder.toString(), null, headers, Client.FormMime); if (response.statusCode != 200) throw new QiniuException(response); @@ -143,7 +142,7 @@ public void closeResource() { if (lines != null) lines.clear(); lines = null; auth = null; - configuration = null; client = null; + configuration = null; } } diff --git a/src/main/java/com/qiniu/process/qiniu/CopyFile.java b/src/main/java/com/qiniu/process/qiniu/CopyFile.java index e3740438..8b4f2e32 100755 --- a/src/main/java/com/qiniu/process/qiniu/CopyFile.java +++ b/src/main/java/com/qiniu/process/qiniu/CopyFile.java @@ -1,15 +1,13 @@ package com.qiniu.process.qiniu; import com.qiniu.common.QiniuException; +import com.qiniu.http.Client; import com.qiniu.http.Response; import com.qiniu.process.Base; -import com.qiniu.storage.BucketManager; -import com.qiniu.storage.BucketManager.*; +//import com.qiniu.storage.BucketManager; +//import com.qiniu.storage.BucketManager.*; import com.qiniu.storage.Configuration; -import com.qiniu.util.Auth; -import com.qiniu.util.FileUtils; -import com.qiniu.util.HttpRespUtils; -import com.qiniu.util.CloudApiUtils; +import com.qiniu.util.*; import java.io.IOException; import java.util.ArrayList; @@ -22,40 +20,54 @@ public class CopyFile extends Base> { private String toKeyIndex; private String addPrefix; private String rmPrefix; - private boolean defaultToKey = false; - private BatchOperations batchOperations; - private List> lines; + private String forceOption; + private boolean defaultToKey; + private Auth auth; + private Client client; + private ArrayList ops; private Configuration configuration; - private BucketManager bucketManager; + private List> lines; +// private BucketManager bucketManager; +// private BatchOperations batchOperations; public CopyFile(String accessKey, String secretKey, Configuration configuration, String bucket, String toBucket, - String toKeyIndex, String addPrefix, String rmPrefix) throws IOException { + String toKeyIndex, String addPrefix, String rmPrefix, boolean force) throws IOException { super("copy", accessKey, secretKey, bucket); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); - CloudApiUtils.checkQiniu(bucketManager, bucket); - CloudApiUtils.checkQiniu(bucketManager, toBucket); - set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix); +// this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); +// CloudApiUtils.checkQiniu(bucketManager, bucket); +// CloudApiUtils.checkQiniu(bucketManager, toBucket); + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); + this.auth = Auth.create(accessKey, secretKey); + this.client = new Client(configuration.clone()); + set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, force); } public CopyFile(String accessKey, String secretKey, Configuration configuration, String bucket, String toBucket, - String toKeyIndex, String addPrefix, String rmPrefix, String savePath, int saveIndex) throws IOException { + String toKeyIndex, String addPrefix, String rmPrefix, boolean force, String savePath, int saveIndex) + throws IOException { super("copy", accessKey, secretKey, bucket, savePath, saveIndex); this.batchSize = 1000; - this.batchOperations = new BatchOperations(); - this.lines = new ArrayList<>(); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); - CloudApiUtils.checkQiniu(bucketManager, bucket); - CloudApiUtils.checkQiniu(bucketManager, toBucket); - set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix); + this.lines = new ArrayList<>(1000); +// this.batchOperations = new BatchOperations(); +// this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); +// CloudApiUtils.checkQiniu(bucketManager, bucket); +// CloudApiUtils.checkQiniu(bucketManager, toBucket); + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); + this.batchSize = 1000; + this.ops = new ArrayList<>(1000); + this.lines = new ArrayList<>(1000); + this.auth = Auth.create(accessKey, secretKey); + this.client = new Client(configuration.clone()); + set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, force); } public CopyFile(String accessKey, String secretKey, Configuration configuration, String bucket, String toBucket, - String toKeyIndex, String keyPrefix, String rmPrefix, String savePath) throws IOException { - this(accessKey, secretKey, configuration, bucket, toBucket, toKeyIndex, keyPrefix, rmPrefix, savePath, 0); + String toKeyIndex, String keyPrefix, String rmPrefix, boolean force, String savePath) throws IOException { + this(accessKey, secretKey, configuration, bucket, toBucket, toKeyIndex, keyPrefix, rmPrefix, force, savePath, 0); } - private void set(Configuration configuration, String toBucket, String toKeyIndex, String addPrefix, String rmPrefix) - throws IOException { + private void set(Configuration configuration, String toBucket, String toKeyIndex, String addPrefix, String rmPrefix, + boolean force) throws IOException { this.configuration = configuration; this.toBucket = toBucket; this.toKeyIndex = toKeyIndex; @@ -68,14 +80,21 @@ private void set(Configuration configuration, String toBucket, String toKeyIndex throw new IOException("toBucket is same as bucket, but no toKeyIndex and no valid addPrefix or rmPrefix."); } } + if (force) forceOption = "/force/true"; + else forceOption = ""; } @Override public CopyFile clone() throws CloneNotSupportedException { CopyFile copyFile = (CopyFile)super.clone(); - copyFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); - copyFile.batchOperations = new BatchOperations(); - copyFile.lines = new ArrayList<>(); +// copyFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); +// copyFile.batchOperations = new BatchOperations(); + if (fileSaveMapper != null) { + copyFile.ops = new ArrayList<>(batchSize); + copyFile.lines = new ArrayList<>(batchSize); + } + copyFile.auth = Auth.create(accessId, secretKey); + copyFile.client = new Client(configuration.clone()); return copyFile; } @@ -86,27 +105,51 @@ protected String resultInfo(Map line) { @Override protected List> putBatchOperations(List> processList) throws IOException { - batchOperations.clearOps(); +// batchOperations.clearOps(); + ops.clear(); lines.clear(); String key; String toKey; - for (Map map : processList) { - key = map.get("key"); - if (key != null) { - try { - if (defaultToKey) { + StringBuilder pathBuilder; + if (defaultToKey) { + for (Map map : processList) { + key = map.get("key"); + if (key != null) { + try { toKey = String.join("", addPrefix, FileUtils.rmPrefix(rmPrefix, key)); - } else { + map.put(toKeyIndex, toKey); + lines.add(map); + pathBuilder = new StringBuilder("/copy/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))).append("/") + .append(UrlSafeBase64.encodeToString(String.join(":", toBucket, toKey))) + .append(forceOption); + ops.add(pathBuilder.toString()); + } catch (IOException e) { + fileSaveMapper.writeError("no " + toKeyIndex + " in " + map, false); + } + } else { + fileSaveMapper.writeError("key is not exists or empty in " + map, false); + } + } + } else { + for (Map map : processList) { + key = map.get("key"); + if (key != null) { + try { toKey = String.join("", addPrefix, FileUtils.rmPrefix(rmPrefix, map.get(toKeyIndex))); + map.put(toKeyIndex, toKey); + lines.add(map); + pathBuilder = new StringBuilder("/copy/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))).append("/") + .append(UrlSafeBase64.encodeToString(String.join(":", toBucket, toKey))) + .append(forceOption); + ops.add(pathBuilder.toString()); + } catch (IOException e) { + fileSaveMapper.writeError("no " + toKeyIndex + " in " + map, false); } - map.put(toKeyIndex, toKey); - lines.add(map); - batchOperations.addCopyOp(bucket, key, toBucket, toKey); - } catch (IOException e) { - fileSaveMapper.writeError("no " + toKeyIndex + " in " + map, false); + } else { + fileSaveMapper.writeError("key is not exists or empty in " + map, false); } - } else { - fileSaveMapper.writeError("key is not exists or empty in " + map, false); } } return lines; @@ -114,7 +157,10 @@ protected List> putBatchOperations(List> @Override protected String batchResult(List> lineList) throws IOException { - return HttpRespUtils.getResult(bucketManager.batch(batchOperations)); +// return HttpRespUtils.getResult(bucketManager.batch(batchOperations)); + byte[] body = StringUtils.utf8Bytes(StringUtils.join(ops, "&op=", "op=")); + return HttpRespUtils.getResult(client.post(CloudApiUtils.QINIU_RS_BATCH_URL, body, + auth.authorization(CloudApiUtils.QINIU_RS_BATCH_URL, body, Client.FormMime), Client.FormMime)); } @Override @@ -122,7 +168,12 @@ protected String singleResult(Map line) throws IOException { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); String toKey = addPrefix + FileUtils.rmPrefix(rmPrefix, defaultToKey ? key : line.get(toKeyIndex)); - Response response = bucketManager.copy(bucket, key, toBucket, toKey, false); +// Response response = bucketManager.copy(bucket, key, toBucket, toKey, false); + StringBuilder urlBuilder = new StringBuilder("http://rs.qiniu.com/copy/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))).append("/") + .append(UrlSafeBase64.encodeToString(String.join(":", toBucket, toKey))).append(forceOption); + StringMap headers = auth.authorization(urlBuilder.toString(), null, Client.FormMime); + Response response = client.post(urlBuilder.toString(), null, headers, Client.FormMime); if (response.statusCode != 200) throw new QiniuException(response); response.close(); return String.join("\t", key, toKey, "200"); @@ -135,9 +186,14 @@ public void closeResource() { toKeyIndex = null; addPrefix = null; rmPrefix = null; - batchOperations = null; + if (ops != null) ops.clear(); + ops = null; + if (lines != null) lines.clear(); lines = null; + auth = null; + client = null; configuration = null; - bucketManager = null; +// bucketManager = null; +// batchOperations = null; } } diff --git a/src/main/java/com/qiniu/process/qiniu/MoveFile.java b/src/main/java/com/qiniu/process/qiniu/MoveFile.java index 621056b3..ab4afc90 100755 --- a/src/main/java/com/qiniu/process/qiniu/MoveFile.java +++ b/src/main/java/com/qiniu/process/qiniu/MoveFile.java @@ -1,15 +1,11 @@ package com.qiniu.process.qiniu; import com.qiniu.common.QiniuException; +import com.qiniu.http.Client; import com.qiniu.http.Response; import com.qiniu.process.Base; -import com.qiniu.storage.BucketManager; -import com.qiniu.storage.BucketManager.*; import com.qiniu.storage.Configuration; -import com.qiniu.util.Auth; -import com.qiniu.util.FileUtils; -import com.qiniu.util.HttpRespUtils; -import com.qiniu.util.CloudApiUtils; +import com.qiniu.util.*; import java.io.IOException; import java.util.ArrayList; @@ -18,82 +14,82 @@ public class MoveFile extends Base> { - private boolean isRename = false; private String toBucket; private String toKeyIndex; private String addPrefix; private String rmPrefix; - private boolean defaultToKey = false; - private BatchOperations batchOperations; - private List> lines; + private String forceOption; + private boolean defaultToKey; + private Auth auth; + private Client client; + private ArrayList ops; private Configuration configuration; - private BucketManager bucketManager; + private List> lines; public MoveFile(String accessKey, String secretKey, Configuration configuration, String bucket, String toBucket, - String toKeyIndex, String addPrefix, String rmPrefix, boolean forceIfOnlyPrefix) throws IOException { + String toKeyIndex, String addPrefix, String rmPrefix, boolean force) throws IOException { // 目标 bucket 为空时规定为 rename 操作 super(toBucket == null || "".equals(toBucket) ? "rename" : "move", accessKey, secretKey, bucket); - if ("rename".equals(processName)) isRename = true; - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); - CloudApiUtils.checkQiniu(bucketManager, bucket); - CloudApiUtils.checkQiniu(bucketManager, toBucket); - set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, forceIfOnlyPrefix); + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); + this.auth = Auth.create(accessKey, secretKey); + this.client = new Client(configuration.clone()); + set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, force); } public MoveFile(String accessKey, String secretKey, Configuration configuration, String bucket, String toBucket, - String toKeyIndex, String addPrefix, String rmPrefix, boolean forceIfOnlyPrefix, String savePath, - int saveIndex) throws IOException { + String toKeyIndex, String addPrefix, String rmPrefix, boolean force, String savePath, int saveIndex) + throws IOException { // 目标 bucket 为空时规定为 rename 操作 super(toBucket == null || "".equals(toBucket) ? "rename" : "move", accessKey, secretKey, bucket, savePath, saveIndex); - if ("rename".equals(processName)) isRename = true; + CloudApiUtils.checkQiniu(accessKey, secretKey, configuration, bucket); this.batchSize = 1000; - this.batchOperations = new BatchOperations(); - this.lines = new ArrayList<>(); - this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); - CloudApiUtils.checkQiniu(bucketManager, bucket); - CloudApiUtils.checkQiniu(bucketManager, toBucket); - set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, forceIfOnlyPrefix); + this.ops = new ArrayList<>(1000); + this.lines = new ArrayList<>(1000); + this.auth = Auth.create(accessKey, secretKey); + this.client = new Client(configuration.clone()); + set(configuration, toBucket, toKeyIndex, addPrefix, rmPrefix, force); } public MoveFile(String accessKey, String secretKey, Configuration configuration, String bucket, String toBucket, - String toKeyIndex, String addPrefix, String rmPrefix, boolean forceIfOnlyPrefix, String savePath) - throws IOException { - this(accessKey, secretKey, configuration, bucket, toBucket, toKeyIndex, addPrefix, rmPrefix, forceIfOnlyPrefix, - savePath, 0); + String toKeyIndex, String addPrefix, String rmPrefix, boolean force, String savePath) throws IOException { + this(accessKey, secretKey, configuration, bucket, toBucket, toKeyIndex, addPrefix, rmPrefix, force, savePath, 0); } - private void set(Configuration configuration, String toBucket, String toKeyIndex, String addPrefix, - String rmPrefix, boolean forceIfOnlyPrefix) throws IOException { + private void set(Configuration configuration, String toBucket, String toKeyIndex, String addPrefix, String rmPrefix, + boolean force) throws IOException { this.configuration = configuration; - this.toBucket = toBucket; + if (toBucket == null || "".equals(toBucket)) { + this.toBucket = bucket; // rename 的情况 + } else { + this.toBucket = toBucket; + CloudApiUtils.checkQiniu(accessId, secretKey, configuration, toBucket); + } this.toKeyIndex = toKeyIndex; this.addPrefix = addPrefix == null ? "" : addPrefix; this.rmPrefix = rmPrefix; if (toKeyIndex == null || "".equals(toKeyIndex)) { this.toKeyIndex = "toKey"; // 没有传入的 toKeyIndex 参数的话直接设置为默认的 "toKey" defaultToKey = true; - if (isRename) { // rename 操作时未设置 new-key 的条件判断 - if (forceIfOnlyPrefix) { - if ((addPrefix == null || "".equals(addPrefix)) && (rmPrefix == null || "".equals(rmPrefix))) - throw new IOException("although prefix-force is true, but there no add/rm prefix for target key."); - } else { - throw new IOException("there is no to-key index, if you only want to add/rm prefix for rename, " + - "please set the \"prefix-force\" as true."); - } - } else { - if (bucket.equals(toBucket)) { - throw new IOException("bucket can not be same as toBucket if process is move."); - } + if ("rename".equals(processName)) { // rename 操作时未设置 new-key 的条件判断 + if ((addPrefix == null || "".equals(addPrefix)) && (rmPrefix == null || "".equals(rmPrefix))) + throw new IOException("toKey index and add/rm prefix must have one for target key."); + } else if (bucket.equals(toBucket)) { + throw new IOException("bucket can not be same as toBucket if process is move."); } } + if (force) forceOption = "/force/true"; + else forceOption = ""; } @Override public MoveFile clone() throws CloneNotSupportedException { MoveFile moveFile = (MoveFile)super.clone(); - moveFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); - moveFile.batchOperations = new BatchOperations(); - moveFile.lines = new ArrayList<>(); + if (fileSaveMapper != null) { + moveFile.ops = new ArrayList<>(batchSize); + moveFile.lines = new ArrayList<>(batchSize); + } + moveFile.auth = Auth.create(accessId, secretKey); + moveFile.client = new Client(configuration.clone()); return moveFile; } @@ -104,31 +100,50 @@ protected String resultInfo(Map line) { @Override protected List> putBatchOperations(List> processList) throws IOException { - batchOperations.clearOps(); + ops.clear(); lines.clear(); String key; String toKey; - for (Map map : processList) { - key = map.get("key"); - if (key != null) { - try { - if (defaultToKey) { + StringBuilder pathBuilder; + if (defaultToKey) { + for (Map map : processList) { + key = map.get("key"); + if (key != null) { + try { toKey = String.join("", addPrefix, FileUtils.rmPrefix(rmPrefix, key)); - } else { - toKey = String.join("", addPrefix, FileUtils.rmPrefix(rmPrefix, map.get(toKeyIndex))); + map.put(toKeyIndex, toKey); + lines.add(map); + pathBuilder = new StringBuilder("/move/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))).append("/") + .append(UrlSafeBase64.encodeToString(String.join(":", toBucket, toKey))) + .append(forceOption); + ops.add(pathBuilder.toString()); + } catch (IOException e) { + fileSaveMapper.writeError("no " + toKeyIndex + " in " + map, false); } - map.put(toKeyIndex, toKey); - lines.add(map); - if (isRename) { - batchOperations.addRenameOp(bucket, key, toKey); - } else { - batchOperations.addMoveOp(bucket, key, toBucket, toKey); + } else { + fileSaveMapper.writeError("key is not exists or empty in " + map, false); + } + } + } else { + for (Map map : processList) { + key = map.get("key"); + if (key != null) { + try { + toKey = String.join("", addPrefix, FileUtils.rmPrefix(rmPrefix, map.get(toKeyIndex))); + map.put(toKeyIndex, toKey); + lines.add(map); + pathBuilder = new StringBuilder("/move/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))).append("/") + .append(UrlSafeBase64.encodeToString(String.join(":", toBucket, toKey))) + .append(forceOption); + ops.add(pathBuilder.toString()); + } catch (IOException e) { + fileSaveMapper.writeError("no " + toKeyIndex + " in " + map, false); } - } catch (IOException e) { - fileSaveMapper.writeError("no " + toKeyIndex + " in " + map, false); + } else { + fileSaveMapper.writeError("key is not exists or empty in " + map, false); } - } else { - fileSaveMapper.writeError("key is not exists or empty in " + map, false); } } return lines; @@ -136,7 +151,9 @@ protected List> putBatchOperations(List> @Override protected String batchResult(List> lineList) throws IOException { - return HttpRespUtils.getResult(bucketManager.batch(batchOperations)); + byte[] body = StringUtils.utf8Bytes(StringUtils.join(ops, "&op=", "op=")); + return HttpRespUtils.getResult(client.post(CloudApiUtils.QINIU_RS_BATCH_URL, body, + auth.authorization(CloudApiUtils.QINIU_RS_BATCH_URL, body, Client.FormMime), Client.FormMime)); } @Override @@ -144,17 +161,14 @@ protected String singleResult(Map line) throws IOException { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); String toKey = addPrefix + FileUtils.rmPrefix(rmPrefix, defaultToKey ? key : line.get(toKeyIndex)); - if (isRename) { - Response response = bucketManager.rename(bucket, key, toKey); - if (response.statusCode != 200) throw new QiniuException(response); - response.close(); - return String.join("\t", key, toKey, "200"); - } else { - Response response = bucketManager.move(bucket, key, toBucket, toKey); - if (response.statusCode != 200) throw new QiniuException(response); - response.close(); - return String.join("\t", key, toKey, "200"); - } + StringBuilder urlBuilder = new StringBuilder("http://rs.qiniu.com/move/") + .append(UrlSafeBase64.encodeToString(String.join(":", bucket, key))).append("/") + .append(UrlSafeBase64.encodeToString(String.join(":", toBucket, toKey))).append(forceOption); + StringMap headers = auth.authorization(urlBuilder.toString(), null, Client.FormMime); + Response response = client.post(urlBuilder.toString(), null, headers, Client.FormMime); + if (response.statusCode != 200) throw new QiniuException(response); + response.close(); + return String.join("\t", key, toKey, "200"); } @Override @@ -164,9 +178,12 @@ public void closeResource() { toKeyIndex = null; addPrefix = null; rmPrefix = null; - batchOperations = null; + if (ops != null) ops.clear(); + ops = null; + if (lines != null) lines.clear(); lines = null; + auth = null; + client = null; configuration = null; - bucketManager = null; } } From 6b0a05d5aebdff647f3e8338ac066e87a7def7fd Mon Sep 17 00:00:00 2001 From: wubingheng Date: Thu, 20 Feb 2020 15:26:46 +0800 Subject: [PATCH 09/12] fix PfopCommand and CdnUrlQuery's clone(), and optimize some ArrayList's init. --- src/test/java/com/qiniu/model/qdora/PfopResultTest.java | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 src/test/java/com/qiniu/model/qdora/PfopResultTest.java diff --git a/src/test/java/com/qiniu/model/qdora/PfopResultTest.java b/src/test/java/com/qiniu/model/qdora/PfopResultTest.java new file mode 100644 index 00000000..4d3467ea --- /dev/null +++ b/src/test/java/com/qiniu/model/qdora/PfopResultTest.java @@ -0,0 +1,7 @@ +package com.qiniu.model.qdora; + +import static org.junit.Assert.*; + +public class PfopResultTest { + +} \ No newline at end of file From e3692bef13c93f88bfb9a3403c7ba010c0728faa Mon Sep 17 00:00:00 2001 From: wubingheng Date: Thu, 20 Feb 2020 15:28:16 +0800 Subject: [PATCH 10/12] fix PfopCommand and CdnUrlQuery's clone(), and optimize some ArrayList's init. --- .../java/com/qiniu/convert/Converter.java | 21 +++++----- .../datasource/CloudStorageContainer.java | 2 +- .../datasource/DefaultFileContainer.java | 2 +- .../com/qiniu/datasource/FileContainer.java | 4 +- .../com/qiniu/datasource/FileInfoLister.java | 6 +-- .../com/qiniu/datasource/QiniuLister.java | 2 +- .../qiniu/datasource/TextFileContainer.java | 2 +- .../java/com/qiniu/datasource/UpLister.java | 2 +- .../com/qiniu/datasource/UpYosContainer.java | 2 +- .../java/com/qiniu/entry/CommonParams.java | 4 +- .../com/qiniu/model/qdora/PfopResult.java | 3 +- src/main/java/com/qiniu/process/Base.java | 2 +- .../qiniu/process/qiniu/CdnUrlProcess.java | 8 ++-- .../com/qiniu/process/qiniu/CdnUrlQuery.java | 13 +++--- .../qiniu/process/qiniu/ChangeLifecycle.java | 8 ++-- .../com/qiniu/process/qiniu/ChangeStatus.java | 8 ++-- .../com/qiniu/process/qiniu/DeleteFile.java | 8 ++-- .../com/qiniu/process/qiniu/PfopCommand.java | 40 +++++++++++-------- .../com/qiniu/process/qiniu/QiniuPfop.java | 2 +- .../com/qiniu/process/qiniu/QueryAvinfo.java | 3 +- .../com/qiniu/process/qiniu/StatFile.java | 10 +++-- .../com/qiniu/model/qdora/PfopResultTest.java | 11 +++++ .../com/qiniu/process/qiniu/CopyFileTest.java | 2 +- urls.txt-result-result/pfopcmd_success_1.txt | 3 ++ 24 files changed, 100 insertions(+), 68 deletions(-) create mode 100644 urls.txt-result-result/pfopcmd_success_1.txt diff --git a/src/main/java/com/qiniu/convert/Converter.java b/src/main/java/com/qiniu/convert/Converter.java index 040237c2..bdcd3f09 100755 --- a/src/main/java/com/qiniu/convert/Converter.java +++ b/src/main/java/com/qiniu/convert/Converter.java @@ -15,17 +15,16 @@ public abstract class Converter implements ITypeConvert { @Override public List convertToVList(List lineList) { - List mapList = new ArrayList<>(); - if (lineList != null && lineList.size() > 0) { - for (E line : lineList) { - try { - mapList.add(convertToV(line)); - } catch (Exception e) { - if (line instanceof String) { - errorList.add(String.join("\t", String.valueOf(line), "convert error", e.getMessage())); - } else { - errorList.add(String.join("\t", JsonUtils.toJson(line), "convert error", e.getMessage())); - } + if (lineList == null) return new ArrayList<>(); + List mapList = new ArrayList<>(lineList.size()); + for (E line : lineList) { + try { + mapList.add(convertToV(line)); + } catch (Exception e) { + if (line instanceof String) { + errorList.add(String.join("\t", String.valueOf(line), "convert error", e.getMessage())); + } else { + errorList.add(String.join("\t", JsonUtils.toJson(line), "convert error", e.getMessage())); } } } diff --git a/src/main/java/com/qiniu/datasource/CloudStorageContainer.java b/src/main/java/com/qiniu/datasource/CloudStorageContainer.java index 3f1e7a1a..729baeb8 100755 --- a/src/main/java/com/qiniu/datasource/CloudStorageContainer.java +++ b/src/main/java/com/qiniu/datasource/CloudStorageContainer.java @@ -495,7 +495,7 @@ private List checkListerInPool(List> listerList, int c rootLogger.info("prefix: {}, nextMarker: {}, endMap: {}", prefix, nextMarker, endMap); // 如果 truncate 时的 nextMarker 已经为空说明已经列举完成了 if (nextMarker == null || nextMarker.isEmpty()) continue; - if (extremePrefixes == null) extremePrefixes = new ArrayList<>(); + if (extremePrefixes == null) extremePrefixes = new ArrayList<>(1); extremePrefixes.add(prefix); prefixMap.put("marker", nextMarker); prefixesMap.put(prefix, prefixMap); diff --git a/src/main/java/com/qiniu/datasource/DefaultFileContainer.java b/src/main/java/com/qiniu/datasource/DefaultFileContainer.java index a3dff3c3..3ac593a9 100755 --- a/src/main/java/com/qiniu/datasource/DefaultFileContainer.java +++ b/src/main/java/com/qiniu/datasource/DefaultFileContainer.java @@ -84,7 +84,7 @@ protected IFileLister getLister(String singleFilePath) throws IO fileInfo.mime = e.getMessage().replace("\n", ","); } } if (indexMap.containsKey("parent")) fileInfo = fileInfo.withParent(); - List list = new ArrayList<>(); + List list = new ArrayList<>(1); list.add(fileInfo); return getLister(singleFilePath, list, null, null, unitLen); } diff --git a/src/main/java/com/qiniu/datasource/FileContainer.java b/src/main/java/com/qiniu/datasource/FileContainer.java index 80478200..67377ac9 100755 --- a/src/main/java/com/qiniu/datasource/FileContainer.java +++ b/src/main/java/com/qiniu/datasource/FileContainer.java @@ -109,7 +109,7 @@ private void setDirectoriesAndMap(Map> directoriesMa String end = value == null ? null : value.get("end"); File tempFile = new File(temp); if (!tempFile.exists()) tempFile = new File(realPath, temp); - directories = new ArrayList<>(); + directories = new ArrayList<>(size); if (tempFile.isDirectory()) directories.add(tempFile); else throw new IOException(temp + " is not valid directory."); String forCheckPath = tempFile.getCanonicalPath() + FileUtils.pathSeparator; @@ -367,7 +367,7 @@ private List loopForFutures(List>> futures) th private AtomicInteger integer = new AtomicInteger(0); private List listForNextIteratively(List directories) throws Exception { - List>> futures = new ArrayList<>(); + List>> futures = new ArrayList<>(directories.size() * 2 / 3 + 1); List nextDirectories = new ArrayList<>(); Future> future; List tempDirectories; diff --git a/src/main/java/com/qiniu/datasource/FileInfoLister.java b/src/main/java/com/qiniu/datasource/FileInfoLister.java index a84c3b93..bcec101a 100755 --- a/src/main/java/com/qiniu/datasource/FileInfoLister.java +++ b/src/main/java/com/qiniu/datasource/FileInfoLister.java @@ -44,7 +44,7 @@ public FileInfoLister(File file, boolean keepDir, boolean withEtag, boolean with } File[] fs = fileFilter == null ? file.listFiles() : file.listFiles(fileFilter); if (fs == null) throw new IOException(file.getPath() + " is not valid directory path."); - int initSize = fs.length > 10 ? (int)(fs.length * 0.7) : fs.length; + int initSize = fs.length > 10 ? fs.length * 2 / 3 + 1 : fs.length; fileInfoList = new ArrayList<>(initSize); if (keepDir) { FileInfo fileInfo = new FileInfo(file, transferPath, leftTrimSize); @@ -72,7 +72,7 @@ public FileInfoLister(File file, boolean keepDir, boolean withEtag, boolean with fileInfo.mime = e.getMessage().replace("\n", ","); }}); } if (withParent) fileInfoList.forEach(FileInfo::withParent); - currents = new ArrayList<>(); + currents = count < limit ? new ArrayList<>((int)count) : new ArrayList<>(limit); fileInfoList.sort(Comparator.comparing(fileInfo -> fileInfo.filepath)); iterator = fileInfoList.iterator(); count = fileInfoList.size(); @@ -107,7 +107,7 @@ public FileInfoLister(String name, List fileInfoList, String startPref fileInfo.filepath.compareTo(startPrefix) > 0 && fileInfo.filepath.compareTo(endPrefix) <= 0); } this.fileInfoList = stream.sorted(Comparator.comparing(fileInfo -> fileInfo.filepath)).collect(Collectors.toList()); - currents = new ArrayList<>(); + currents = count < limit ? new ArrayList<>((int)count) : new ArrayList<>(limit); iterator = this.fileInfoList.iterator(); count = this.fileInfoList.size(); // if (iterator.hasNext()) { diff --git a/src/main/java/com/qiniu/datasource/QiniuLister.java b/src/main/java/com/qiniu/datasource/QiniuLister.java index a09b0648..dd2a3fbc 100755 --- a/src/main/java/com/qiniu/datasource/QiniuLister.java +++ b/src/main/java/com/qiniu/datasource/QiniuLister.java @@ -72,7 +72,7 @@ private List getListResult(String prefix, String marker, int limit) th InputStream inputStream = new BufferedInputStream(response.bodyStream()); Reader reader = new InputStreamReader(inputStream); BufferedReader bufferedReader = new BufferedReader(reader); - List fileInfoList = new ArrayList<>(limit > 100 ? (int)(limit * 0.67) : limit); + List fileInfoList = new ArrayList<>(limit > 100 ? limit * 2 / 3 + 1 : limit); String line = bufferedReader.readLine(); try { if (line == null) { diff --git a/src/main/java/com/qiniu/datasource/TextFileContainer.java b/src/main/java/com/qiniu/datasource/TextFileContainer.java index a71cad3b..231b585a 100755 --- a/src/main/java/com/qiniu/datasource/TextFileContainer.java +++ b/src/main/java/com/qiniu/datasource/TextFileContainer.java @@ -74,7 +74,7 @@ private List splitSingleFile(File file) throws IOException { RandomAccessFile[] accessFiles = new RandomAccessFile[threads]; accessFiles[0] = new RandomAccessFile(file, "r"); String endLine; - List readers = new ArrayList<>(); + List readers = new ArrayList<>(threads); int i = 1; for (; i < threads; i++) { RandomAccessFile accessFile = new RandomAccessFile(file, "r"); diff --git a/src/main/java/com/qiniu/datasource/UpLister.java b/src/main/java/com/qiniu/datasource/UpLister.java index 34245cbb..eefedbcc 100755 --- a/src/main/java/com/qiniu/datasource/UpLister.java +++ b/src/main/java/com/qiniu/datasource/UpLister.java @@ -81,7 +81,7 @@ private List getListResult(String prefix, String marker, int limit) th JsonElement jsonElement = returnJson.get("files"); if (jsonElement instanceof JsonArray) { JsonArray files = returnJson.get("files").getAsJsonArray(); - int initSize = files.size() > 100 ? (int)(files.size() * 0.67) : files.size(); + int initSize = files.size() > 100 ? files.size() * 2 / 3 + 1 : files.size(); List fileItems = new ArrayList<>(initSize); if (files.size() > 0) { JsonObject object; diff --git a/src/main/java/com/qiniu/datasource/UpYosContainer.java b/src/main/java/com/qiniu/datasource/UpYosContainer.java index 7872815a..40bc5acd 100755 --- a/src/main/java/com/qiniu/datasource/UpYosContainer.java +++ b/src/main/java/com/qiniu/datasource/UpYosContainer.java @@ -123,7 +123,7 @@ private List directoriesAfterListerRun(String prefix) { private void listForNextIteratively(List prefixes) throws Exception { List tempPrefixes; - List>> futures = new ArrayList<>(); + List>> futures = new ArrayList<>(prefixes.size() * 2 / 3 + 1); for (String prefix : prefixes) { if (atomicLong.get() > threads) { tempPrefixes = directoriesAfterListerRun(prefix); diff --git a/src/main/java/com/qiniu/entry/CommonParams.java b/src/main/java/com/qiniu/entry/CommonParams.java index b8952f5e..e7a39584 100755 --- a/src/main/java/com/qiniu/entry/CommonParams.java +++ b/src/main/java/com/qiniu/entry/CommonParams.java @@ -991,7 +991,7 @@ private void setIndexMap() throws IOException { return; } int fieldsMode = 0; - List keys = new ArrayList<>(); + List keys = new ArrayList<>(9); String indexes = entryParam.getValue("indexes", "").trim(); boolean useDefault = "".equals(indexes); boolean zeroDefault = false; @@ -1323,7 +1323,7 @@ private void setRmFields() throws IOException { rmFields = null; } else { String[] fields = ParamsUtils.escapeSplit(param); - rmFields = new ArrayList<>(); + rmFields = new ArrayList<>(fields.length); Collections.addAll(rmFields, fields); } } diff --git a/src/main/java/com/qiniu/model/qdora/PfopResult.java b/src/main/java/com/qiniu/model/qdora/PfopResult.java index 92ba2e98..92a5424e 100755 --- a/src/main/java/com/qiniu/model/qdora/PfopResult.java +++ b/src/main/java/com/qiniu/model/qdora/PfopResult.java @@ -1,7 +1,6 @@ package com.qiniu.model.qdora; import java.util.ArrayList; -import java.util.List; public class PfopResult { @@ -10,7 +9,7 @@ public class PfopResult { public String id; public String inputBucket; public String inputKey; - public List items = new ArrayList<>(); + public ArrayList items; public String pipeline; public String reqid; } diff --git a/src/main/java/com/qiniu/process/Base.java b/src/main/java/com/qiniu/process/Base.java index ce048ae5..2adb0bb6 100755 --- a/src/main/java/com/qiniu/process/Base.java +++ b/src/main/java/com/qiniu/process/Base.java @@ -153,7 +153,7 @@ protected List parseBatchResult(List processList, String result) throws Ex jsonObject.toString()), false); break; case 0: - if (retryList == null) retryList = new ArrayList<>(); + if (retryList == null) retryList = new ArrayList<>(1); retryList.add(processList.get(j)); // 放回重试列表 break; case -1: diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java index 399fd8d3..ee91a610 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java @@ -43,9 +43,9 @@ public CdnUrlProcess(String accessKey, String secretKey, Configuration configura public CdnUrlProcess(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, String urlIndex, boolean isDir, boolean prefetch, String savePath, int saveIndex) throws IOException { super(prefetch ? "cdnprefetch" : "cdnrefresh", accessKey, secretKey, null, savePath, saveIndex); - this.batchSize = 30; - this.batches = new String[30]; - this.lines = new ArrayList<>(); + this.batchSize = isDir ? 10 : 30; + this.batches = new String[batchSize]; + this.lines = new ArrayList<>(batchSize); Auth auth = Auth.create(accessKey, secretKey); CdnHelper cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::prefetch : isDir ? @@ -81,7 +81,7 @@ public CdnUrlProcess(String accessKey, String secretKey, Configuration configura @Override public CdnUrlProcess clone() throws CloneNotSupportedException { // CdnUrlProcess cdnUrlProcess = (CdnUrlProcess)super.clone(); -// cdnUrlProcess.lines = new ArrayList<>(); +// cdnUrlProcess.lines = new ArrayList<>(batchSize); // CdnHelper cdnHelper = new CdnHelper(Auth.create(accessId, secretKey), configuration); // cdnUrlProcess.cdnApplier = prefetch ? cdnHelper::prefetch : isDir ? // dirs -> cdnHelper.refresh(null, dirs) : urls -> cdnHelper.refresh(urls, null); diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java index 416cc119..245fc661 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java @@ -46,9 +46,9 @@ public CdnUrlQuery(String accessKey, String secretKey, Configuration configurati public CdnUrlQuery(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, String urlIndex, boolean prefetch, String savePath, int saveIndex) throws IOException { super(prefetch ? "prefetchquery" : "refreshquery", accessKey, secretKey, null, savePath, saveIndex); - this.batchSize = 30; - this.batches = new String[30]; - this.lines = new ArrayList<>(); + this.batchSize = 100; + this.batches = new String[100]; + this.lines = new ArrayList<>(100); Auth auth = Auth.create(accessKey, secretKey); cdnHelper = new CdnHelper(auth, configuration); this.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; @@ -99,7 +99,10 @@ public void setQueryOptions(int pageNo, int pageSize, String startTime, String e @Override public CdnUrlQuery clone() throws CloneNotSupportedException { CdnUrlQuery cdnUrlQuery = (CdnUrlQuery)super.clone(); - cdnUrlQuery.lines = new ArrayList<>(); + if (fileSaveMapper != null) { + cdnUrlQuery.batches = new String[batchSize]; + cdnUrlQuery.lines = new ArrayList<>(batchSize); + } cdnUrlQuery.cdnHelper = new CdnHelper(Auth.create(accessId, secretKey), configuration); if (hasOption) { cdnUrlQuery.cdnApplier = prefetch ? urls -> { @@ -126,7 +129,7 @@ protected String resultInfo(Map line) { } @Override - protected synchronized List> putBatchOperations(List> processList) throws IOException { + protected List> putBatchOperations(List> processList) throws IOException { Arrays.fill(batches, null); lines.clear(); Map line; diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java b/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java index 315b763f..b9d60bd3 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeLifecycle.java @@ -38,7 +38,7 @@ public ChangeLifecycle(String accessKey, String secretKey, Configuration configu this.days = days; this.batchSize = 1000; this.batchOperations = new BatchOperations(); - this.lines = new ArrayList<>(); + this.lines = new ArrayList<>(1000); this.configuration = configuration; this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); @@ -53,8 +53,10 @@ public ChangeLifecycle(String accessKey, String secretKey, Configuration configu public ChangeLifecycle clone() throws CloneNotSupportedException { ChangeLifecycle changeLifecycle = (ChangeLifecycle)super.clone(); changeLifecycle.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); - changeLifecycle.batchOperations = new BatchOperations(); - changeLifecycle.lines = new ArrayList<>(); + if (fileSaveMapper != null) { + changeLifecycle.batchOperations = new BatchOperations(); + changeLifecycle.lines = new ArrayList<>(batchSize); + } return changeLifecycle; } diff --git a/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java b/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java index b5d6e7a7..c85f8705 100755 --- a/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java +++ b/src/main/java/com/qiniu/process/qiniu/ChangeStatus.java @@ -38,7 +38,7 @@ public ChangeStatus(String accessKey, String secretKey, Configuration configurat this.status = status; this.batchSize = 1000; this.batchOperations = new BatchOperations(); - this.lines = new ArrayList<>(); + this.lines = new ArrayList<>(1000); this.configuration = configuration; this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); @@ -53,8 +53,10 @@ public ChangeStatus(String accessKey, String secretKey, Configuration configurat public ChangeStatus clone() throws CloneNotSupportedException { ChangeStatus changeStatus = (ChangeStatus)super.clone(); changeStatus.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); - changeStatus.batchOperations = new BatchOperations(); - changeStatus.lines = new ArrayList<>(); + if (fileSaveMapper != null) { + changeStatus.batchOperations = new BatchOperations(); + changeStatus.lines = new ArrayList<>(batchSize); + } return changeStatus; } diff --git a/src/main/java/com/qiniu/process/qiniu/DeleteFile.java b/src/main/java/com/qiniu/process/qiniu/DeleteFile.java index a873f861..574113d3 100755 --- a/src/main/java/com/qiniu/process/qiniu/DeleteFile.java +++ b/src/main/java/com/qiniu/process/qiniu/DeleteFile.java @@ -34,7 +34,7 @@ public DeleteFile(String accessKey, String secretKey, Configuration configuratio super("delete", accessKey, secretKey, bucket, savePath, saveIndex); this.batchSize = 1000; this.batchOperations = new BatchOperations(); - this.lines = new ArrayList<>(); + this.lines = new ArrayList<>(1000); this.configuration = configuration; this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); @@ -49,8 +49,10 @@ public DeleteFile(String accessKey, String secretKey, Configuration configuratio public DeleteFile clone() throws CloneNotSupportedException { DeleteFile deleteFile = (DeleteFile)super.clone(); deleteFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); - deleteFile.batchOperations = new BatchOperations(); - deleteFile.lines = new ArrayList<>(); + if (fileSaveMapper != null) { + deleteFile.batchOperations = new BatchOperations(); + deleteFile.lines = new ArrayList<>(batchSize); + } return deleteFile; } diff --git a/src/main/java/com/qiniu/process/qiniu/PfopCommand.java b/src/main/java/com/qiniu/process/qiniu/PfopCommand.java index e5878279..8abcec32 100755 --- a/src/main/java/com/qiniu/process/qiniu/PfopCommand.java +++ b/src/main/java/com/qiniu/process/qiniu/PfopCommand.java @@ -56,9 +56,9 @@ private void set(Configuration configuration, String avinfoIndex, boolean hasDur if (pfopConfigs != null && pfopConfigs.size() > 0) { this.pfopConfigs = pfopConfigs; } else if (pfopJsonPath != null && !"".equals(pfopJsonPath)) { - this.pfopConfigs = new ArrayList<>(); JsonFile jsonFile = new JsonFile(pfopJsonPath); JsonArray array = jsonFile.getElement("pfopcmd").getAsJsonArray(); + this.pfopConfigs = new ArrayList<>(array.size()); for (JsonElement jsonElement : array) { JsonObject jsonObject = PfopUtils.checkPfopJson(jsonElement.getAsJsonObject(), false); this.pfopConfigs.add(jsonObject); @@ -87,18 +87,18 @@ protected String singleResult(Map line) throws Exception { String info; Avinfo avinfo; String fopCmd; - StringBuilder fops = new StringBuilder(); VideoStream videoStream; List scale; - List resultList = new ArrayList<>(); + key = line.get("key"); + if (key == null) throw new IOException("key is not exists or empty in " + line); + info = line.get(avinfoIndex); + if (info == null || "".equals(info)) throw new IOException("avinfo is empty."); + avinfo = mediaManager.getAvinfoByJson(JsonUtils.fromJson(info, JsonObject.class)); + videoStream = avinfo.getVideoStream(); + if (videoStream == null) throw new IOException("videoStream is null."); + StringBuilder fops = new StringBuilder(); + List resultList = new ArrayList<>(pfopConfigs.size()); for (JsonObject pfopConfig : pfopConfigs) { - key = line.get("key"); - if (key == null) throw new IOException("key is not exists or empty in " + line); - info = line.get(avinfoIndex); - if (info == null || "".equals(info)) throw new IOException("avinfo is empty."); - avinfo = mediaManager.getAvinfoByJson(JsonUtils.fromJson(info, JsonObject.class)); - videoStream = avinfo.getVideoStream(); - if (videoStream == null) throw new IOException("videoStream is null."); if (pfopConfig.get("scale") instanceof JsonArray) { scale = JsonUtils.fromJsonArray(pfopConfig.get("scale").getAsJsonArray(), new TypeToken>() { }); @@ -111,15 +111,23 @@ protected String singleResult(Map line) throws Exception { fopCmd = PfopUtils.generateFopCmd(key, pfopConfig); } if (fopCmd.contains("$(duration)")) fopCmd = fopCmd.replace("$(duration)", avinfo.getFormat().duration); - if (combine) fops.append(fopCmd); - else fops.append(key).append("\t").append(fopCmd); - if (hasDuration) fops.append("\t").append(Double.valueOf(avinfo.getFormat().duration)); - if (hasSize) fops.append("\t").append(Long.parseLong(avinfo.getFormat().size)); + if (combine) { + fops.append(fopCmd); + } else { + fops.append(key).append("\t").append(fopCmd); + if (hasDuration) fops.append("\t").append(Double.valueOf(avinfo.getFormat().duration)); + if (hasSize) fops.append("\t").append(Long.parseLong(avinfo.getFormat().size)); + } resultList.add(fops.toString()); fops.delete(0, fops.length()); } - if (combine) return String.join("\t", key, String.join(";", resultList)); - else return String.join("\n", resultList); + if (combine) { + if (hasDuration) fops.append("\t").append(Double.valueOf(avinfo.getFormat().duration)); + if (hasSize) fops.append("\t").append(Long.parseLong(avinfo.getFormat().size)); + return String.join("\t", key, String.join(";", resultList)) + fops; + } else { + return String.join("\n", resultList); + } } @Override diff --git a/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java b/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java index 1a221200..6cd3858d 100755 --- a/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java +++ b/src/main/java/com/qiniu/process/qiniu/QiniuPfop.java @@ -57,9 +57,9 @@ private void set(Configuration configuration, String pipeline, String notifyURL, if (pfopConfigs != null && pfopConfigs.size() > 0) { this.pfopConfigs = pfopConfigs; } else if (pfopJsonPath != null && !"".equals(pfopJsonPath)) { - this.pfopConfigs = new ArrayList<>(); JsonFile jsonFile = new JsonFile(pfopJsonPath); JsonArray array = jsonFile.getElement("pfop").getAsJsonArray(); + this.pfopConfigs = new ArrayList<>(array.size()); for (JsonElement jsonElement : array) { JsonObject jsonObject = PfopUtils.checkPfopJson(jsonElement.getAsJsonObject(), false); this.pfopConfigs.add(jsonObject); diff --git a/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java b/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java index b14120fd..c28fd726 100755 --- a/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java +++ b/src/main/java/com/qiniu/process/qiniu/QueryAvinfo.java @@ -66,7 +66,8 @@ protected String singleResult(Map line) throws Exception { String url; if (domain == null) { url = line.get(urlIndex); - return String.join("\t", url, JsonUtils.toJson(mediaManager.getAvinfoBody(url))); + // 解析出文件名是为了便于后续可能需要通过 key 和 avinfo 来进行一些其他对应操作 + return String.join("\t", URLUtils.getKey(url), JsonUtils.toJson(mediaManager.getAvinfoBody(url)), url); } else { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); diff --git a/src/main/java/com/qiniu/process/qiniu/StatFile.java b/src/main/java/com/qiniu/process/qiniu/StatFile.java index 2bef6564..ac9094af 100755 --- a/src/main/java/com/qiniu/process/qiniu/StatFile.java +++ b/src/main/java/com/qiniu/process/qiniu/StatFile.java @@ -42,7 +42,7 @@ public StatFile(String accessKey, String secretKey, Configuration configuration, set(configuration, rmFields, format, separator); this.batchSize = 100; this.batchOperations = new BatchOperations(); - this.lines = new ArrayList<>(); + this.lines = new ArrayList<>(1000); this.bucketManager = new BucketManager(Auth.create(accessKey, secretKey), configuration); CloudApiUtils.checkQiniu(bucketManager, bucket); } @@ -79,8 +79,10 @@ private IStringFormat getNewStatJsonFormatter(List rmFields) public StatFile clone() throws CloneNotSupportedException { StatFile statFile = (StatFile)super.clone(); statFile.bucketManager = new BucketManager(Auth.create(accessId, secretKey), configuration); - statFile.batchOperations = new BatchOperations(); - statFile.lines = new ArrayList<>(); + if (fileSaveMapper != null) { + statFile.batchOperations = new BatchOperations(); + statFile.lines = new ArrayList<>(batchSize); + } statFile.stringFormatter = getNewStatJsonFormatter(rmFields); return statFile; } @@ -136,7 +138,7 @@ protected List> parseBatchResult(List> p fileSaveMapper.writeSuccess(stringFormatter.toFormatString(data), false); break; case 0: - if (retryList == null) retryList = new ArrayList<>(); + if (retryList == null) retryList = new ArrayList<>(1); retryList.add(processList.get(j)); // 放回重试列表 break; case -1: diff --git a/src/test/java/com/qiniu/model/qdora/PfopResultTest.java b/src/test/java/com/qiniu/model/qdora/PfopResultTest.java index 4d3467ea..e4ceb58b 100644 --- a/src/test/java/com/qiniu/model/qdora/PfopResultTest.java +++ b/src/test/java/com/qiniu/model/qdora/PfopResultTest.java @@ -1,7 +1,18 @@ package com.qiniu.model.qdora; +import com.qiniu.util.JsonUtils; +import org.junit.Test; + import static org.junit.Assert.*; public class PfopResultTest { + @Test + public void test() { + String result = "{\"code\":200,\"items\":[{\"cmd\":\"avthumb/mp4\"}]}"; + PfopResult pfopResult = JsonUtils.fromJson(result, PfopResult.class); + System.out.println(pfopResult.code); + System.out.println(pfopResult.items.get(0).cmd); + } + } \ No newline at end of file diff --git a/src/test/java/com/qiniu/process/qiniu/CopyFileTest.java b/src/test/java/com/qiniu/process/qiniu/CopyFileTest.java index c82c8eff..b94ff7fe 100755 --- a/src/test/java/com/qiniu/process/qiniu/CopyFileTest.java +++ b/src/test/java/com/qiniu/process/qiniu/CopyFileTest.java @@ -17,7 +17,7 @@ public void testSingle() throws IOException { String secretKey = propertiesFile.getValue("sk"); String bucket = propertiesFile.getValue("bucket"); CopyFile copyFile = new CopyFile(accessKey, secretKey, new Configuration(), bucket, "ts-work", - null, null, null, "../temp"); + null, null, null, false, "../temp"); Map map = new HashMap(){{ put("key", "qiniu_success_1.txt"); put("toKey", "qiniu_success_1-1.txt"); diff --git a/urls.txt-result-result/pfopcmd_success_1.txt b/urls.txt-result-result/pfopcmd_success_1.txt new file mode 100644 index 00000000..66f28d05 --- /dev/null +++ b/urls.txt-result-result/pfopcmd_success_1.txt @@ -0,0 +1,3 @@ +http://p3l1d5mx4.bkt.clouddn.com/CR1904660812.mp4 avthumb/mp4;avsmart|saveas/dGVtcA== +http://p3l1d5mx4.bkt.clouddn.com/CR1904660812.mp4F720.mp4 avthumb/mp4;avsmart|saveas/dGVtcA== +http://p3l1d5mx4.bkt.clouddn.com/CR1904660812F720.mp4 avthumb/mp4;avsmart|saveas/dGVtcA== From 6b4c24ba8bcac8a8124b856b57ff7c9c8d29b42e Mon Sep 17 00:00:00 2001 From: wubingheng Date: Thu, 20 Feb 2020 16:57:49 +0800 Subject: [PATCH 11/12] fix and update cdn processors. --- keys.txt-result.json | 1 + .../com/qiniu/interfaces/ICdnApplier.java | 4 +- .../qiniu/process/qiniu/CdnUrlProcess.java | 56 ++++++++++------- .../com/qiniu/process/qiniu/CdnUrlQuery.java | 60 +++++++++++-------- urls.txt-result-result/pfopcmd_success_1.txt | 3 - 5 files changed, 74 insertions(+), 50 deletions(-) create mode 100644 keys.txt-result.json delete mode 100644 urls.txt-result-result/pfopcmd_success_1.txt diff --git a/keys.txt-result.json b/keys.txt-result.json new file mode 100644 index 00000000..339586af --- /dev/null +++ b/keys.txt-result.json @@ -0,0 +1 @@ +{"/Users/wubingheng/Downloads/keys.txt":{}} diff --git a/src/main/java/com/qiniu/interfaces/ICdnApplier.java b/src/main/java/com/qiniu/interfaces/ICdnApplier.java index 74723798..096fe110 100644 --- a/src/main/java/com/qiniu/interfaces/ICdnApplier.java +++ b/src/main/java/com/qiniu/interfaces/ICdnApplier.java @@ -3,7 +3,9 @@ import com.qiniu.common.QiniuException; import com.qiniu.http.Response; +import java.util.List; + public interface ICdnApplier { - Response apply(String[] urls) throws QiniuException; + Response apply(List urls) throws QiniuException; } diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java index ee91a610..3b91343e 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlProcess.java @@ -11,7 +11,6 @@ import java.io.IOException; import java.util.ArrayList; -import java.util.Arrays; import java.util.List; import java.util.Map; import java.util.stream.Collectors; @@ -23,18 +22,27 @@ public class CdnUrlProcess extends Base> { private String urlIndex; private boolean isDir; // private boolean prefetch; - private String[] batches; + private List batches; private List> lines; // private Configuration configuration; + private CdnHelper cdnHelper; private ICdnApplier cdnApplier; public CdnUrlProcess(String accessKey, String secretKey, Configuration configuration, String protocol, String domain, String urlIndex, boolean isDir, boolean prefetch) throws IOException { super(prefetch ? "cdnprefetch" : "cdnrefresh", accessKey, secretKey, null); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration); - this.cdnApplier = prefetch ? cdnHelper::prefetch : isDir ? - dirs -> cdnHelper.refresh(null, dirs) : urls -> cdnHelper.refresh(urls, null); + cdnHelper = new CdnHelper(auth, configuration); + this.cdnApplier = prefetch ? urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.prefetch(urls.toArray(urlArray)); + } : isDir ? urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.refresh(null, urls.toArray(urlArray)); + } : urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.refresh(urls.toArray(urlArray), null); + }; CloudApiUtils.checkQiniu(auth); set(configuration, protocol, domain, urlIndex, isDir); // this.prefetch = prefetch; @@ -44,12 +52,20 @@ public CdnUrlProcess(String accessKey, String secretKey, Configuration configura String urlIndex, boolean isDir, boolean prefetch, String savePath, int saveIndex) throws IOException { super(prefetch ? "cdnprefetch" : "cdnrefresh", accessKey, secretKey, null, savePath, saveIndex); this.batchSize = isDir ? 10 : 30; - this.batches = new String[batchSize]; + this.batches = new ArrayList<>(batchSize); this.lines = new ArrayList<>(batchSize); Auth auth = Auth.create(accessKey, secretKey); - CdnHelper cdnHelper = new CdnHelper(auth, configuration); - this.cdnApplier = prefetch ? cdnHelper::prefetch : isDir ? - dirs -> cdnHelper.refresh(null, dirs) : urls -> cdnHelper.refresh(urls, null); + cdnHelper = new CdnHelper(auth, configuration); + this.cdnApplier = prefetch ? urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.prefetch(urls.toArray(urlArray)); + } : isDir ? urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.refresh(null, urls.toArray(urlArray)); + } : urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.refresh(urls.toArray(urlArray), null); + }; CloudApiUtils.checkQiniu(auth); set(configuration, protocol, domain, urlIndex, isDir); // this.prefetch = prefetch; @@ -100,26 +116,23 @@ protected String resultInfo(Map line) { @Override protected synchronized List> putBatchOperations(List> processList) throws IOException { - Arrays.fill(batches, null); + batches.clear(); lines.clear(); - Map line; if (domain == null) { - for (int i = 0; i < processList.size(); i++) { - line = processList.get(i); + for (Map line : processList) { lines.add(line); - batches[i] = line.get(urlIndex); + batches.add(line.get(urlIndex)); } } else { String key; - for (int i = 0; i < processList.size(); i++) { - line = processList.get(i); + for (Map line : processList) { key = line.get("key"); if (key == null) { fileSaveMapper.writeError("key and url are not exist or empty in " + line, false); } else { lines.add(line); - batches[i] = String.join("", protocol, "://", domain, "/", - key.replace("\\?", "%3f")); + batches.add(String.join("", protocol, "://", domain, "/", + key.replace("\\?", "%3f"))); } } } @@ -162,14 +175,12 @@ protected String singleResult(Map line) throws IOException { String url; if (domain == null) { url = line.get(urlIndex); - String[] urls = new String[]{url}; - return String.join("\t", url, HttpRespUtils.getResult(cdnApplier.apply(urls))); + return String.join("\t", url, HttpRespUtils.getResult(cdnApplier.apply(new ArrayList(){{ add(url); }}))); } else { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); url = String.join("", protocol, "://", domain, "/", key.replace("\\?", "%3f")); - String[] urls = new String[]{url}; - return String.join("\t", key, HttpRespUtils.getResult(cdnApplier.apply(urls))); + return String.join("\t", key, HttpRespUtils.getResult(cdnApplier.apply(new ArrayList(){{ add(url); }}))); } } @@ -182,6 +193,7 @@ public void closeResource() { urlIndex = null; batches = null; lines = null; + cdnHelper = null; cdnApplier = null; } else { saveIndex.decrementAndGet(); diff --git a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java index 245fc661..e707d5e0 100755 --- a/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java +++ b/src/main/java/com/qiniu/process/qiniu/CdnUrlQuery.java @@ -11,7 +11,6 @@ import java.io.IOException; import java.util.ArrayList; -import java.util.Arrays; import java.util.List; import java.util.Map; import java.util.stream.Collectors; @@ -22,12 +21,12 @@ public class CdnUrlQuery extends Base> { private String domain; private String urlIndex; private boolean prefetch; - private String[] batches; private boolean hasOption; private int pageNo; private int pageSize; private String startTime; private String endTime; + private List batches; private Configuration configuration; private List> lines; private CdnHelper cdnHelper; @@ -38,7 +37,13 @@ public CdnUrlQuery(String accessKey, String secretKey, Configuration configurati super(prefetch ? "prefetchquery" : "refreshquery", accessKey, secretKey, null); Auth auth = Auth.create(accessKey, secretKey); cdnHelper = new CdnHelper(auth, configuration); - this.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; + this.cdnApplier = prefetch ? urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.queryPrefetch(urls.toArray(urlArray)); + } : urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.queryRefresh(urls.toArray(urlArray)); + }; CloudApiUtils.checkQiniu(auth); set(configuration, protocol, domain, urlIndex, prefetch); } @@ -47,11 +52,17 @@ public CdnUrlQuery(String accessKey, String secretKey, Configuration configurati String urlIndex, boolean prefetch, String savePath, int saveIndex) throws IOException { super(prefetch ? "prefetchquery" : "refreshquery", accessKey, secretKey, null, savePath, saveIndex); this.batchSize = 100; - this.batches = new String[100]; + this.batches = new ArrayList<>(batchSize); this.lines = new ArrayList<>(100); Auth auth = Auth.create(accessKey, secretKey); cdnHelper = new CdnHelper(auth, configuration); - this.cdnApplier = prefetch ? cdnHelper::queryPrefetch : cdnHelper::queryRefresh; + this.cdnApplier = prefetch ? urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.queryPrefetch(urls.toArray(urlArray)); + } : urls -> { + String[] urlArray = new String[urls.size()]; + return cdnHelper.queryRefresh(urls.toArray(urlArray)); + }; CloudApiUtils.checkQiniu(auth); set(configuration, protocol, domain, urlIndex, prefetch); this.fileSaveMapper.preAddWriter("processing"); @@ -86,11 +97,11 @@ public void setQueryOptions(int pageNo, int pageSize, String startTime, String e this.endTime = endTime; this.hasOption = true; this.cdnApplier = prefetch ? urls -> { - JsonArray urlArray = new JsonArray(); + JsonArray urlArray = new JsonArray(urls.size()); for (String url : urls) urlArray.add(url); return cdnHelper.queryPrefetch(urlArray, pageNo, pageSize, startTime, endTime); } : urls -> { - JsonArray urlArray = new JsonArray(); + JsonArray urlArray = new JsonArray(urls.size()); for (String url : urls) urlArray.add(url); return cdnHelper.queryRefresh(urlArray, pageNo, pageSize, startTime, endTime); }; @@ -100,22 +111,28 @@ public void setQueryOptions(int pageNo, int pageSize, String startTime, String e public CdnUrlQuery clone() throws CloneNotSupportedException { CdnUrlQuery cdnUrlQuery = (CdnUrlQuery)super.clone(); if (fileSaveMapper != null) { - cdnUrlQuery.batches = new String[batchSize]; + cdnUrlQuery.batches = new ArrayList<>(batchSize); cdnUrlQuery.lines = new ArrayList<>(batchSize); } cdnUrlQuery.cdnHelper = new CdnHelper(Auth.create(accessId, secretKey), configuration); if (hasOption) { cdnUrlQuery.cdnApplier = prefetch ? urls -> { - JsonArray urlArray = new JsonArray(); + JsonArray urlArray = new JsonArray(urls.size()); for (String url : urls) urlArray.add(url); return cdnUrlQuery.cdnHelper.queryPrefetch(urlArray, pageNo, pageSize, startTime, endTime); } : urls -> { - JsonArray urlArray = new JsonArray(); + JsonArray urlArray = new JsonArray(urls.size()); for (String url : urls) urlArray.add(url); return cdnUrlQuery.cdnHelper.queryRefresh(urlArray, pageNo, pageSize, startTime, endTime); }; } else { - cdnUrlQuery.cdnApplier = prefetch ? cdnUrlQuery.cdnHelper::queryPrefetch : cdnUrlQuery.cdnHelper::queryRefresh; + cdnUrlQuery.cdnApplier = prefetch ? urls -> { + String[] urlArray = new String[urls.size()]; + return cdnUrlQuery.cdnHelper.queryPrefetch(urls.toArray(urlArray)); + } : urls -> { + String[] urlArray = new String[urls.size()]; + return cdnUrlQuery.cdnHelper.queryRefresh(urls.toArray(urlArray)); + }; } if (cdnUrlQuery.fileSaveMapper != null) { cdnUrlQuery.fileSaveMapper.preAddWriter("processing"); @@ -130,26 +147,23 @@ protected String resultInfo(Map line) { @Override protected List> putBatchOperations(List> processList) throws IOException { - Arrays.fill(batches, null); + batches.clear(); lines.clear(); - Map line; if (domain == null) { - for (int i = 0; i < processList.size(); i++) { - line = processList.get(i); + for (Map line : processList) { lines.add(line); - batches[i] = line.get(urlIndex); + batches.add(line.get(urlIndex)); } } else { String key; - for (int i = 0; i < processList.size(); i++) { - line = processList.get(i); + for (Map line : processList) { key = line.get("key"); if (key == null) { fileSaveMapper.writeError("key and url are not exist or empty in " + line, false); } else { lines.add(line); - batches[i] = String.join("", protocol, "://", domain, "/", - key.replace("\\?", "%3f")); + batches.add(String.join("", protocol, "://", domain, "/", + key.replace("\\?", "%3f"))); } } } @@ -199,14 +213,12 @@ protected String singleResult(Map line) throws IOException { String url; if (domain == null) { url = line.get(urlIndex); - String[] urls = new String[]{url}; - return String.join("\t", url, HttpRespUtils.getResult(cdnApplier.apply(urls))); + return String.join("\t", url, HttpRespUtils.getResult(cdnApplier.apply(new ArrayList(){{ add(url); }}))); } else { String key = line.get("key"); if (key == null) throw new IOException("key is not exists or empty in " + line); url = String.join("", protocol, "://", domain, "/", key.replace("\\?", "%3f")); - String[] urls = new String[]{url}; - return String.join("\t", key, HttpRespUtils.getResult(cdnApplier.apply(urls))); + return String.join("\t", key, HttpRespUtils.getResult(cdnApplier.apply(new ArrayList(){{ add(url); }}))); } } diff --git a/urls.txt-result-result/pfopcmd_success_1.txt b/urls.txt-result-result/pfopcmd_success_1.txt deleted file mode 100644 index 66f28d05..00000000 --- a/urls.txt-result-result/pfopcmd_success_1.txt +++ /dev/null @@ -1,3 +0,0 @@ -http://p3l1d5mx4.bkt.clouddn.com/CR1904660812.mp4 avthumb/mp4;avsmart|saveas/dGVtcA== -http://p3l1d5mx4.bkt.clouddn.com/CR1904660812.mp4F720.mp4 avthumb/mp4;avsmart|saveas/dGVtcA== -http://p3l1d5mx4.bkt.clouddn.com/CR1904660812F720.mp4 avthumb/mp4;avsmart|saveas/dGVtcA== From 818d9c5526289b1c4466b8dd31ae11bf0fda1a0b Mon Sep 17 00:00:00 2001 From: wubingheng Date: Thu, 20 Feb 2020 16:58:35 +0800 Subject: [PATCH 12/12] fix and update cdn processors. --- keys.txt-result.json | 1 - 1 file changed, 1 deletion(-) delete mode 100644 keys.txt-result.json diff --git a/keys.txt-result.json b/keys.txt-result.json deleted file mode 100644 index 339586af..00000000 --- a/keys.txt-result.json +++ /dev/null @@ -1 +0,0 @@ -{"/Users/wubingheng/Downloads/keys.txt":{}}