Skip to content

Commit 17ca15c

Browse files
author
wubingheng
committed
complete config-https set, and add it for other private url processors, and update docs.
1 parent 8b35df5 commit 17ca15c

13 files changed

+116
-66
lines changed

Diff for: ‎README.md

+4-3
Original file line numberDiff line numberDiff line change
@@ -340,11 +340,12 @@ rm-fields 可选择持久化结果中去除某些字段，未设置的情况下
340340

341341
详细配置说明见 [持久化配置](docs/resultsave.md)。
342342

343-
### 7 超时设置
344-
多数数据源或者操作涉及网络请求，因此提供超时时间设置，默认的超时时间一般能够满足要求，特殊需要的情况下可以修改各超时时间：
343+
### 7 网络设置
344+
多数数据源或者操作涉及网络请求，因此提供超时时间和协议设置，默认设置一般能够满足要求，特殊需要的情况下可以修改各超时时间和协议：
345345
`connect-timeout=60` 网络连接超时时间，程序默认 60s
346346
`read-timeout=120` socket 读取超时时间，程序默认 120s
347347
`request-timeout=60` 网络请求超时时间，程序默认 60s
348+
`config-https=true/false` 对数据源或 process 涉及的公共 api 是否使用 https 来请求，七牛云/华为云数据源或者七牛的 process 均默认使用 https
348349

349350
### 8 错误及异常
350351
1、一般情况下，终端输出异常信息如 socket timeout 超时为正常现象，如：
@@ -353,7 +354,7 @@ list prefix:<prefix> retrying...
353354
...
354355
java.net.SocketTimeoutException: timeout
355356
```
356-
程序会自动重试，如果比较频繁则可以修改[超时配置](#7-超时设置)重新运行程序，超过重试次数或者其他非预期异常发生时程序会退出，可以将异常信息反馈在
357+
程序会自动重试，如果比较频繁则可以修改[超时配置](#7-网络设置)重新运行程序，超过重试次数或者其他非预期异常发生时程序会退出，可以将异常信息反馈在
357358
[ISSUE列表](https://github.com/NigelWu95/qiniu-suits-java/issues) 中。
358359
2、常见错误信息：
359360
（1）java.lang.UnsupportedClassVersionError: Unsupported major.minor version ...

Diff for: ‎docs/datamigration.md

+13
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,20 @@ qiniu-sk=
178178
to-bucket=
179179
# region 可以不设置则自动判断，如选择自行指定则使用 qiniu-region，便于和数据源的 region 区分开
180180
# qiniu-region=z0/z1/qvm-z0/...
181+
```
182+
183+
##### 七牛云私有存储
184+
按需要增加如下参数设置 API 的域名即可：
181185
```
186+
rsf-domain=rsf-z0.qiniu.com
187+
#rsf-domain=rsf-z1.qiniu.com
188+
rs-domain=rs-z0.qiniu.com
189+
#rs-domain=rs-z1.qiniu.com
190+
api-domain=api.qiniu.com
191+
#api-domain=api-z1.qiniu.com
192+
#是否使用 https 对上述设置的域名进行请求
193+
config-https=true/false
194+
```
182195

183196
#### 8. 本地文件列表
184197
```

Diff for: ‎docs/datasource.md

+4-1
Original file line numberDiff line numberDiff line change
@@ -309,14 +309,17 @@ threads=300
309309
prefixes=
310310
#region=
311311
```
312-
**使用七牛数据源除了可以自行指定 region（建议您自行指定 region 较好），也可以单独指定 rsf,rs,api 域名（一般不需要用到），如**：
312+
**使用七牛数据源除了可以自行指定 region（建议您自行指定 region 较好），也可以单独指定 rsf,rs,api 域名（一般不需要用到），如下所示，一般可用在七
313+
牛私有存储的场景中**：
313314
```
314315
rsf-domain=rsf-z0.qiniu.com
315316
#rsf-domain=rsf-z1.qiniu.com
316317
rs-domain=rs-z0.qiniu.com
317318
#rs-domain=rs-z1.qiniu.com
318319
api-domain=api.qiniu.com
319320
#api-domain=api-z1.qiniu.com
321+
#是否使用 https 对上述设置的域名进行请求
322+
config-https=true/false
320323
```
321324

322325
### 2 腾讯云存储

Diff for: ‎docs/syncupload.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ private=
4949
|private| 数据源私有类型|是否是对私有空间资源进行同步访问上传，选择对应的私有类型，参考[私有访问](#资源需要私有签名)|
5050

5151
### 超时时间
52-
timeout 参数可以通过全局的 timeout 来设置，参考：[超时设置](../README.md#7-超时设置)
52+
timeout 参数可以通过全局的 timeout 来设置，参考：[网络设置](../README.md#7-网络设置)
5353

5454
### 关于 url-index
5555
当使用 file 源且 parse=tab/csv 时 [xx-]index(ex) 设置的下标必须为整数。url-index 表示输入行含 url 形式的源文件地址，未设置的情况下则使用

Diff for: ‎docs/uploadfile.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ process=qupload
1313
path=
1414
directories=
1515
directory-config=
16+
anti-prefixes=
1617
ak=
1718
sk=
1819
bucket=
@@ -36,6 +37,7 @@ check=
3637
|path| 本地路径| path 是数据源选项，可以通过设置本地路径来指定要上传的文件，为目录时会遍历目录下（包括内层目录）除隐藏文件外的所有文件|
3738
|directories| 文件目录列表| 如果希望上传 path 下的几个目录中文件，可设置 path 路径下需要读取的目录列表，以 `,` 号分割目录名，不设置默认读取 path 下全部目录进行文件的上传|
3839
|directory-config| 配置文件路径|数据源文件目录及对应已上传的文件名配置，配置中记录已上传的文件在 path 中的位置标识，配置格式为 json，参考[ directory-config 配置文件](#directory-config-配置)，该配置不需要自行编写|
40+
|anti-prefixes| 文件目录列表| 表示上传目录下文件时排除某些名称前缀的子目录，支持以 `,` 分隔的列表，特殊字符同样需要转义符|
3941
|ak、sk|长度40的字符串|七牛账号的ak、sk，通过七牛控制台个人中心获取|
4042
|bucket| 字符串| 上传到的资源原空间名称|
4143
|filepath-index| 文件路径索引| 非必填字端，当直接上传 path 路径中的文件时无需设置，如果是通过读取文本文件每一行中的路径信息则需要设置|
@@ -72,7 +74,7 @@ directory-config 中的目录名可以采取相对 path 路径下的目录名。
7274
|value| 表示数据源中某一行的内容，如 "qiniu_success_1.txt" 表示 "/Users/wubingheng/Projects/Github/test" 目录中可能存在该文件名|
7375

7476
### 上传超时时间
75-
timeout 参数可以通过全局的 timeout 来设置，参考：[超时设置](../README.md#7-超时设置)
77+
timeout 参数可以通过全局的 timeout 来设置，参考：[网络设置](../README.md#7-网络设置)
7678

7779
### 关于上传的文件路径
7880
上传文件的操作可以直接上传通过 path 指定的目录下的文件，也支持传通过文件列表中提供的 filepath 的文件，如：

Diff for: ‎pom.xml

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
<groupId>com.qiniu</groupId>
88
<artifactId>qsuits</artifactId>
9-
<version>8.3.9</version>
9+
<version>8.3.10</version>
1010
<name>qsuits</name>
1111
<description>qiniu-suits is a efficient tools for qiniu api implemented by java8.</description>
1212
<url>https://github.com/NigelWu95/qiniu-suits-java</url>

Diff for: ‎src/main/java/com/qiniu/entry/CommonParams.java

+21-22
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ public class CommonParams {
2424
private int connectTimeout;
2525
private int readTimeout;
2626
private int requestTimeout;
27+
private boolean httpsForConfigEnabled;
2728
private String path;
2829
private String source;
2930
private boolean isStorageSource;
@@ -43,11 +44,9 @@ public class CommonParams {
4344
private String huaweiSecretKey;
4445
private String baiduAccessId;
4546
private String baiduSecretKey;
46-
private boolean httpsForConfigEnabled;
4747
private String bucket;
4848
private String logFilepath;
4949
private Map<String, Map<String, String>> pathConfigMap;
50-
private List<String> antiDirectories;
5150
private List<String> antiPrefixes;
5251
private boolean prefixLeft;
5352
private boolean prefixRight;
@@ -165,6 +164,7 @@ public CommonParams(IEntryParam entryParam) throws Exception {
165164
setTimeout();
166165
path = entryParam.getValue("path", "");
167166
setSource();
167+
setHttpsConfigEnabled();
168168
accountInit();
169169
logFilepath = entryParam.getValue("log", null);
170170
if (isStorageSource) {
@@ -212,6 +212,7 @@ public CommonParams(Map<String, String> paramsMap) throws Exception {
212212
this.entryParam = new ParamsConfig(paramsMap);
213213
setTimeout();
214214
source = "terminal";
215+
setHttpsConfigEnabled();
215216
accountInit();
216217
setParse();
217218
setSeparator();
@@ -333,10 +334,6 @@ private void setTimeout() {
333334
requestTimeout = Integer.parseInt(entryParam.getValue("request-timeout", "60").trim());
334335
}
335336

336-
private void setHttpsConfigEnabled() throws IOException {
337-
String enabled = entryParam.getValue("config-https", "true");
338-
}
339-
340337
private void setSource() throws IOException {
341338
if (entryParam.getValue("interactive", "").trim().equals("true")) {
342339
source = "terminal";
@@ -389,6 +386,16 @@ else if (!source.matches("(local|qiniu|tencent|aliyun|upyun|s3|huawei|baidu)"))
389386
isStorageSource = CloudApiUtils.isStorageSource(source);
390387
}
391388

389+
private void setHttpsConfigEnabled() throws IOException {
390+
String enabled = entryParam.getValue("config-https", "").trim();
391+
if ("".equals(enabled)) {
392+
if ("qiniu".equals(source) || "huawei".equals(source)) httpsForConfigEnabled = true;
393+
} else {
394+
ParamsUtils.checked(enabled, "config-https", "(true|false)");
395+
httpsForConfigEnabled = Boolean.parseBoolean(enabled);
396+
}
397+
}
398+
392399
private void setParse() throws IOException {
393400
parse = entryParam.getValue("parse", "tab").trim();
394401
ParamsUtils.checked(parse, "parse", "(csv|tab|json|object|file)");
@@ -1348,6 +1355,10 @@ public void setRequestTimeout(int requestTimeout) {
13481355
this.requestTimeout = requestTimeout;
13491356
}
13501357

1358+
public void setHttpsForConfigEnabled(boolean httpsForConfigEnabled) {
1359+
this.httpsForConfigEnabled = httpsForConfigEnabled;
1360+
}
1361+
13511362
public void setPath(String path) {
13521363
this.path = path;
13531364
}
@@ -1412,10 +1423,6 @@ public void setBaiduSecretKey(String baiduSecretKey) {
14121423
this.baiduSecretKey = baiduSecretKey;
14131424
}
14141425

1415-
public void setHttpsForConfigEnabled(boolean httpsForConfigEnabled) {
1416-
this.httpsForConfigEnabled = httpsForConfigEnabled;
1417-
}
1418-
14191426
public void setBucket(String bucket) {
14201427
this.bucket = bucket;
14211428
}
@@ -1424,10 +1431,6 @@ public void setPathConfigMap(Map<String, Map<String, String>> pathConfigMap) {
14241431
this.pathConfigMap = pathConfigMap;
14251432
}
14261433

1427-
public void setAntiDirectories(List<String> antiDirectories) {
1428-
this.antiDirectories = antiDirectories;
1429-
}
1430-
14311434
public void setAntiPrefixes(List<String> antiPrefixes) {
14321435
this.antiPrefixes = antiPrefixes;
14331436
}
@@ -1552,6 +1555,10 @@ public int getRequestTimeout() {
15521555
return requestTimeout;
15531556
}
15541557

1558+
public boolean isHttpsForConfigEnabled() {
1559+
return httpsForConfigEnabled;
1560+
}
1561+
15551562
public String getPath() {
15561563
return path;
15571564
}
@@ -1616,10 +1623,6 @@ public String getBaiduSecretKey() {
16161623
return baiduSecretKey;
16171624
}
16181625

1619-
public boolean isHttpsForConfigEnabled() {
1620-
return httpsForConfigEnabled;
1621-
}
1622-
16231626
public String getBucket() {
16241627
return bucket;
16251628
}
@@ -1628,10 +1631,6 @@ public Map<String, Map<String, String>> getPathConfigMap() {
16281631
return pathConfigMap;
16291632
}
16301633

1631-
public List<String> getAntiDirectories() {
1632-
return antiDirectories;
1633-
}
1634-
16351634
public List<String> getAntiPrefixes() {
16361635
return antiPrefixes;
16371636
}

Diff for: ‎src/main/java/com/qiniu/entry/QSuitsEntry.java

+22-14
Original file line numberDiff line numberDiff line change
@@ -328,7 +328,7 @@ public TextFileContainer getTextFileContainer() throws IOException {
328328
public DefaultFileContainer getDefaultFileContainer() throws IOException {
329329
String path = commonParams.getPath();
330330
Map<String, Map<String, String>> directoriesMap = commonParams.getPathConfigMap();
331-
List<String> antiDirectories = commonParams.getAntiDirectories();
331+
List<String> antiDirectories = commonParams.getAntiPrefixes();
332332
boolean keepDir = commonParams.getKeepDir();
333333
DefaultFileContainer defaultFileContainer = new DefaultFileContainer(path, directoriesMap, antiDirectories,
334334
keepDir, indexMap, null, unitLen, threads);
@@ -805,9 +805,9 @@ private com.qiniu.process.tencent.PrivateUrl getTencentPrivateUrl(boolean single
805805
if (region == null || "".equals(region)) region = CloudApiUtils.getTenCosRegion(secretId, secretKey, tenBucket);
806806
String expires = entryParam.getValue("expires", "3600").trim();
807807
ParamsUtils.checked(expires, "expires", "[1-9]\\d*");
808-
return single ? new com.qiniu.process.tencent.PrivateUrl(secretId, secretKey, tenBucket, region,
808+
return single ? new com.qiniu.process.tencent.PrivateUrl(secretId, secretKey, tenBucket, region, httpsConfigEnabled,
809809
1000 * Long.parseLong(expires), getQueriesMap()) : new com.qiniu.process.tencent.PrivateUrl(secretId,
810-
secretKey, tenBucket, region, 1000 * Long.parseLong(expires), getQueriesMap(), savePath);
810+
secretKey, tenBucket, region, httpsConfigEnabled,1000 * Long.parseLong(expires), getQueriesMap(), savePath);
811811
}
812812

813813
private com.qiniu.process.aliyun.PrivateUrl getAliyunPrivateUrl(boolean single) throws IOException {
@@ -821,8 +821,11 @@ private com.qiniu.process.aliyun.PrivateUrl getAliyunPrivateUrl(boolean single)
821821
String endPoint = regionName == null || regionName.isEmpty() ? entryParam.getValue("region", regionName) : regionName;
822822
if (endPoint == null || "".equals(endPoint)) endPoint = CloudApiUtils.getAliOssRegion(accessId, accessSecret, aliBucket);
823823
if (!endPoint.matches("https?://.+")) {
824-
if (endPoint.startsWith("oss-")) endPoint = "http://" + endPoint + ".aliyuncs.com";
825-
else endPoint = "http://oss-" + endPoint + ".aliyuncs.com";
824+
if (endPoint.startsWith("oss-")) {
825+
endPoint = String.join(endPoint, httpsConfigEnabled ? "https://" : "http://", ".aliyuncs.com");
826+
} else {
827+
endPoint = String.join(endPoint, httpsConfigEnabled ? "https://oss-" : "http://oss-", ".aliyuncs.com");
828+
}
826829
}
827830
String expires = entryParam.getValue("expires", "3600").trim();
828831
ParamsUtils.checked(expires, "expires", "[1-9]\\d*");
@@ -845,9 +848,9 @@ private com.qiniu.process.aws.PrivateUrl getAwsS3PrivateUrl(boolean single) thro
845848
region = CloudApiUtils.getS3Region(accessId, secretKey, s3Bucket);
846849
String expires = entryParam.getValue("expires", "3600").trim();
847850
ParamsUtils.checked(expires, "expires", "[1-9]\\d*");
848-
return single ? new com.qiniu.process.aws.PrivateUrl(accessId, secretKey, s3Bucket, endpoint, region,
851+
return single ? new com.qiniu.process.aws.PrivateUrl(accessId, secretKey, s3Bucket, endpoint, region, httpsConfigEnabled,
849852
1000 * Long.parseLong(expires), getQueriesMap()) : new com.qiniu.process.aws.PrivateUrl(accessId,
850-
secretKey, s3Bucket, endpoint, region, 1000 * Long.parseLong(expires), getQueriesMap(), savePath);
853+
secretKey, s3Bucket, endpoint, region, httpsConfigEnabled, 1000 * Long.parseLong(expires), getQueriesMap(), savePath);
851854
}
852855

853856
private com.qiniu.process.huawei.PrivateUrl getHuaweiPrivateUrl(boolean single) throws IOException {
@@ -861,8 +864,11 @@ private com.qiniu.process.huawei.PrivateUrl getHuaweiPrivateUrl(boolean single)
861864
String endPoint = regionName == null || regionName.isEmpty() ? entryParam.getValue("region", regionName) : regionName;
862865
if (endPoint == null || "".equals(endPoint)) endPoint = CloudApiUtils.getHuaweiObsRegion(accessId, secretKey, huaweiBucket);
863866
if (!endPoint.matches("https?://.+")) {
864-
if (endPoint.startsWith("obs.")) endPoint = "http://" + endPoint + ".myhuaweicloud.com";
865-
else endPoint = "http://obs." + endPoint + ".myhuaweicloud.com";
867+
if (endPoint.startsWith("obs.")) {
868+
endPoint = String.join(endPoint, httpsConfigEnabled ? "https://" : "http://", ".myhuaweicloud.com");
869+
} else {
870+
endPoint = String.join(endPoint, httpsConfigEnabled ? "https://obs." : "http://obs.", ".myhuaweicloud.com");
871+
}
866872
}
867873
String expires = entryParam.getValue("expires", "3600").trim();
868874
ParamsUtils.checked(expires, "expires", "[1-9]\\d*");
@@ -883,7 +889,7 @@ private com.qiniu.process.baidu.PrivateUrl getBaiduPrivateUrl(boolean single) th
883889
String endPoint = regionName == null || regionName.isEmpty() ? entryParam.getValue("region", regionName) : regionName;
884890
if (endPoint == null || "".equals(endPoint)) endPoint = CloudApiUtils.getBaiduBosRegion(accessId, secretKey, baiduBucket);
885891
if (!endPoint.matches("https?://.+")) {
886-
endPoint = "http://" + endPoint + ".bcebos.com";
892+
endPoint = String.join(endPoint, httpsConfigEnabled ? "https://" : "http://", ".bcebos.com");
887893
}
888894
String expires = entryParam.getValue("expires", "3600").trim();
889895
ParamsUtils.checked(expires, "expires", "[1-9]\\d*");
@@ -1060,17 +1066,19 @@ private ILineProcess<Map<String, String>> getCdnRefresh(Map<String, String> inde
10601066
String dir = entryParam.getValue("is-dir", "false").trim();
10611067
ParamsUtils.checked(dir, "is-dir", "(true|false)");
10621068
boolean isDir = Boolean.parseBoolean(dir);
1063-
return single ? new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, protocol, domain, urlIndex, isDir, false)
1064-
: new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, protocol, domain, urlIndex, isDir, false, savePath);
1069+
return single ? new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, getNewQiniuConfig(), protocol, domain, urlIndex,
1070+
isDir, false) : new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, getNewQiniuConfig(), protocol,
1071+
domain, urlIndex, isDir, false, savePath);
10651072
}
10661073

10671074
private ILineProcess<Map<String, String>> getCdnPrefetch(Map<String, String> indexMap, boolean single) throws IOException {
10681075
String protocol = entryParam.getValue("protocol", "http").trim();
10691076
ParamsUtils.checked(protocol, "protocol", "https?");
10701077
String domain = entryParam.getValue("domain", "").trim();
10711078
String urlIndex = indexMap.containsValue("url") ? "url" : null;
1072-
return single ? new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, protocol, domain, urlIndex, false, true)
1073-
: new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, protocol, domain, urlIndex, false, true, savePath);
1079+
return single ? new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, getNewQiniuConfig(), protocol, domain, urlIndex,
1080+
false, true) : new CdnUrlProcess(qiniuAccessKey, qiniuSecretKey, getNewQiniuConfig(), protocol,
1081+
domain, urlIndex, false, true, savePath);
10741082
}
10751083

10761084
private ILineProcess<Map<String, String>> getFetch(Map<String, String> indexMap, boolean single) throws IOException {

0 commit comments

Comments
 (0)