Skip to content

Commit d4b7bc4

Browse files
author
markheger
authored
Merge pull request #2 from IBMStreams/develop
1.0.1
2 parents cba1f9e + bfe2497 commit d4b7bc4

File tree

7 files changed

+144
-9
lines changed

7 files changed

+144
-9
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,3 +107,4 @@ venv.bak/
107107

108108
# mypy
109109
.mypy_cache/
110+
/.project

‎README.md‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,11 @@ Documentation is using Sphinx and can be built locally using:
1818
cd package/docs
1919
make html
2020
```
21+
22+
or
23+
24+
ant doc
25+
2126
and viewed using
2227
```
2328
firefox package/docs/build/html/index.html
@@ -28,6 +33,22 @@ The documentation is also setup at `readthedocs.io`.
2833
Documentation links:
2934
* http://streamsxhdfs.readthedocs.io
3035

36+
## Version update
37+
38+
To change the version information of the Python package, edit following files:
39+
40+
- ./package/docs/source/conf.py
41+
- ./package/streamsx/hdfs/\_\_init\_\_.py
42+
43+
When the development status changes, edit the *classifiers* in
44+
45+
- ./package/setup.py
46+
47+
When the documented sample must be changed, change it here:
48+
49+
- ./package/streamsx/hdfs/\_\_init\_\_.py
50+
- ./package/DESC.txt
51+
3152
## Test
3253

3354
Package can be tested with TopologyTester using the [Streaming Analytics](https://www.ibm.com/cloud/streaming-analytics) service and [Analytics Engine](https://www.ibm.com/cloud/analytics-engine) service on IBM Cloud.
@@ -51,13 +72,18 @@ python3 -u -m unittest streamsx.hdfs.tests.test_hdfs.TestParams
5172

5273
This test requires STREAMS_INSTALL set and a running Streams instance.
5374

54-
Required envionment variable for the `com.ibm.streamsx.hdfs` toolkit location: `STREAMS_HDFS_TOOLKIT`
75+
Required envionment variable for the `com.ibm.streamsx.hdfs` toolkit location: `STREAMS_HDFS_TOOLKIT`
5576

5677
```
5778
cd package
5879
python3 -u -m unittest streamsx.hdfs.tests.test_hdfs.TestDistributed
5980
```
6081

82+
or
83+
84+
ant test
85+
86+
6187
### Test with Streaming Analytics Service
6288

6389
This requires Streaming Analytics service and IBM Analytics Engine service in IBM cloud.
@@ -69,4 +95,24 @@ cd package
6995
python3 -u -m unittest streamsx.hdfs.tests.test_hdfs.TestCloud.test_close_on_tuples streamsx.hdfs.tests.test_hdfs.TestCloud.test_hdfs_uri
7096
```
7197

98+
or
99+
100+
ant test-sas
101+
102+
103+
#### Remote build
104+
105+
For using the toolkit from the build service (**force_remote_build**) run the test with:
106+
107+
Run the test with:
108+
109+
ant test-sas-remote
110+
111+
or
112+
113+
```
114+
cd package
115+
python3 -u -m unittest streamsx.hdfs.tests.test_hdfs.TestCloudRemote.test_close_on_tuples streamsx.hdfs.tests.test_hdfs.TestCloudRemote.test_hdfs_uri
116+
```
117+
72118

‎build.xml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
<project name="PypackageTests" default="test" basedir="." xmlns:if="ant:if">
2+
3+
<property name="package" location="package"/>
4+
5+
<target name="clean">
6+
<delete>
7+
<fileset dir="${basedir}" includes="**/*.pyc,**/streamsx*.sab,**/streamsx*.json"/>
8+
</delete>
9+
<delete includeemptydirs="true">
10+
<fileset dir="${basedir}" includes="**/__pycache__/**"/>
11+
</delete>
12+
<delete includeemptydirs="true">
13+
<fileset dir="${basedir}" includes="**/tk*/**"/>
14+
</delete>
15+
<delete includeemptydirs="true">
16+
<fileset dir="${package}" includes="**/build/**,dist/**,streamsx.*.egg-info/**,job*.tar.gz"/>
17+
</delete>
18+
</target>
19+
20+
<target name="doc">
21+
<echo message="Generate html doc"/>
22+
<exec executable="make" failonerror="true" dir="${package}/docs">
23+
<arg value="-f" />
24+
<arg value="Makefile" />
25+
<arg value="html" />
26+
</exec>
27+
</target>
28+
29+
<target name="test" depends="clean">
30+
<exec executable="/bin/sh"
31+
outputproperty="toolkit.test.output" errorproperty="toolkit.test.error" resultproperty="toolkit.test.result"
32+
dir="${package}">
33+
<arg value="-c"/>
34+
<arg value="python3 -u -m unittest streamsx.hdfs.tests.test_hdfs.TestDistributed"/>
35+
</exec>
36+
<echo message="${toolkit.test.output}" if:set="toolkit.test.output"/>
37+
<echo message="${toolkit.test.error}" if:set="toolkit.test.error"/>
38+
<fail message="The test failed - result ${toolkit.test.result}.">
39+
<condition>
40+
<not>
41+
<equals arg1="${toolkit.test.result}" arg2="0"/>
42+
</not>
43+
</condition>
44+
</fail>
45+
</target>
46+
47+
<target name="test-sas" depends="clean">
48+
<exec executable="/bin/sh"
49+
outputproperty="toolkit.test.output" errorproperty="toolkit.test.error" resultproperty="toolkit.test.result"
50+
dir="${package}">
51+
<arg value="-c"/>
52+
<arg value="python3 -u -m unittest streamsx.hdfs.tests.test_hdfs.TestCloud.test_close_on_tuples streamsx.hdfs.tests.test_hdfs.TestCloud.test_hdfs_uri"/>
53+
</exec>
54+
<echo message="${toolkit.test.output}" if:set="toolkit.test.output"/>
55+
<echo message="${toolkit.test.error}" if:set="toolkit.test.error"/>
56+
<fail message="The test failed - result ${toolkit.test.result}.">
57+
<condition>
58+
<not>
59+
<equals arg1="${toolkit.test.result}" arg2="0"/>
60+
</not>
61+
</condition>
62+
</fail>
63+
</target>
64+
65+
<target name="test-sas-remote" depends="clean">
66+
<exec executable="/bin/sh"
67+
outputproperty="toolkit.test.output" errorproperty="toolkit.test.error" resultproperty="toolkit.test.result"
68+
dir="${package}">
69+
<arg value="-c"/>
70+
<arg value="unset STREAMS_INSTALL;python3 -u -m unittest streamsx.hdfs.tests.test_hdfs.TestCloudRemote.test_close_on_tuples streamsx.hdfs.tests.test_hdfs.TestCloudRemote.test_hdfs_uri"/>
71+
</exec>
72+
<echo message="${toolkit.test.output}" if:set="toolkit.test.output"/>
73+
<echo message="${toolkit.test.error}" if:set="toolkit.test.error"/>
74+
<fail message="The test failed - result ${toolkit.test.result}.">
75+
<condition>
76+
<not>
77+
<equals arg1="${toolkit.test.result}" arg2="0"/>
78+
</not>
79+
</condition>
80+
</fail>
81+
</target>
82+
83+
</project>

‎package/DESC.txt‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,12 @@ Overview
22
========
33

44
Provides functions to access files on HDFS. For example, connect to IBM Analytics Engine on IBM Cloud.
5-
This package exposes the `com.ibm.streamsx.hdfs` toolkit as Python methods.
65

7-
* `IBM Streaming Analytics <https://www.ibm.com/cloud/streaming-analytics>`_
6+
This package exposes the `com.ibm.streamsx.hdfs <https://ibmstreams.github.io/streamsx.hdfs/>`_ toolkit as Python methods for use with Streaming Analytics service on
7+
IBM Cloud and IBM Streams including IBM Cloud Private for Data.
8+
9+
* `Streaming Analytics service <https://console.ng.bluemix.net/catalog/services/streaming-analytics>`_
10+
* `IBM Streams developer community <https://developer.ibm.com/streamsdev/>`_
811
* `IBM Analytics Engine <https://www.ibm.com/cloud/analytics-engine>`_
912

1013

@@ -38,6 +41,8 @@ a file to HDFS. Scan for created file on HDFS and read the content::
3841
r.print()
3942

4043
submit('STREAMING_ANALYTICS_SERVICE', topo)
44+
# Use for IBM Streams including IBM Cloud Private for Data
45+
# submit ('DISTRIBUTED', topo)
4146

4247

4348
Documentation

‎package/docs/source/conf.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@
6767
# The short X.Y version.
6868
version = '1.0'
6969
# The full version, including alpha/beta/rc tags.
70-
release = '1.0.0'
70+
release = '1.0.1'
7171

7272
# The language for content autogenerated by Sphinx. Refer to documentation
7373
# for a list of supported languages.

‎package/docs/source/index.rst‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,10 @@ IBM Streams HDFS integration
55
============================
66

77
For details of implementing applications in Python
8-
for IBM Streams and the Streaming Analytics service
8+
for IBM Streams including IBM Cloud Private for Data and the Streaming Analytics service
99
running on IBM Cloud see:
1010

11-
* `streamsx package documentation <http://ibmstreams.github.io/streamsx.topology/doc/releases/latest/pythondoc/index.html>`_
12-
13-
* `Developing IBM Streams Applications with Python <http://ibmstreams.github.io/streamsx.documentation/docs/python/1.6/python-appapi-devguide/>`_
11+
* `streamsx package documentation <https://streamsxtopology.readthedocs.io/en/stable>`_
1412

1513
This package exposes SPL operators in the `com.ibm.streamsx.hdfs <https://ibmstreams.github.io/streamsx.hdfs/>`_ toolkit as Python methods.
1614

‎package/streamsx/hdfs/__init__.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,10 +64,12 @@
6464
r.print()
6565
6666
submit('STREAMING_ANALYTICS_SERVICE', topo)
67+
# Use for IBM Streams including IBM Cloud Private for Data
68+
# submit ('DISTRIBUTED', topo)
6769
6870
"""
6971

70-
__version__='1.0.0'
72+
__version__='1.0.1'
7173

7274
__all__ = ['scan', 'read', 'write']
7375
from streamsx.hdfs._hdfs import scan, read, write

0 commit comments

Comments
 (0)