Skip to content

Commit

Permalink
Merge pull request #73 from jiayuasu/GeoSpark-for-Spark-1.X
Browse files Browse the repository at this point in the history 
GeoSpark version bumps to 0.6.0
  • Loading branch information
@jiayuasu
jiayuasu authored Apr 3, 2017
2 parents baa216b + 4b20f3f commit 7a8f8f6
Show file tree
Hide file tree
Showing 71 changed files with 2,800 additions and 2,994 deletions.
1 change: 1 addition & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ GeoSpark artifacts are hosted in Maven Central: [**Maven Central Coordinates**](

| Version | Summary |
|:----------------: |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|0.6.0| **Major updates:** (1) DistanceJoin is merged into JoinQuery. GeoSpark now supports complete DistanceJoin between Points, Polygons, and LineStrings. (2) Add Refine Phase to Spatial Range and Join Query. Use real polygon coordinates instead of its MBR to filter the final results. **API changes:** All spatial range and join queries now take a parameter called *ConsiderBoundaryIntersection*. This will tell GeoSpark whether returns the objects intersect with windows.|
|0.5.3| **Bug fix:** Fix [Issue #69](https://github.com/DataSystemsLab/GeoSpark/issues/69): Now, if two objects have the same coordinates but different non-spatial attributes (UserData), GeoSpark treats them as different objects.|
|0.5.2| **Bug fix:** Fix [Issue #58](https://github.com/DataSystemsLab/GeoSpark/issues/58) and [Issue #60](https://github.com/DataSystemsLab/GeoSpark/issues/60); **Performance enhancement:**: (1) Deprecate all old Spatial RDD constructors. See the JavaDoc [here](http://www.public.asu.edu/~jiayu2/geospark/javadoc/0.5.2/). (2) Recommend the new SRDD constructors which take an additional RDD storage level and automatically cache rawSpatialRDD to accelerate internal SRDD analyze step|
|0.5.1| **Bug fix:** (1) GeoSpark: Fix inaccurate KNN result when K is large (2) GeoSpark: Replace incompatible Spark API call [Issue #55](https://github.com/DataSystemsLab/GeoSpark/issues/55); (3) Babylon: Remove JPG output format temporarily due to the lack of OpenJDK support|
Expand Down
5 changes: 4 additions & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>org.datasyslab</groupId>
<artifactId>geospark</artifactId>
<version>0.5.3-spark-1.x</version>
<version>0.6.0-spark-1.x</version>

<name>${project.groupId}:${project.artifactId}</name>
<description>Geospatial extension for Apache Spark</description>
Expand Down Expand Up @@ -70,8 +70,11 @@
<groupId>com.vividsolutions</groupId>
<artifactId>jts</artifactId>
</exclusion>

</exclusions>
</dependency>


</dependencies>

<build>
Expand Down
12 changes: 7 additions & 5 deletions src/main/java/org/datasyslab/babylon/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

## Main Features

### Extensible Visualization operator
### Extensible Visualization operator (just like playing LEGO bricks)!

* Support super high resolution image generation: parallel map image rendering
* Visualize Spatial RDD and Spatial Queries (Spatial Range, Spatial K Nearest Neighbors, Spatial Join)
Expand All @@ -22,7 +22,7 @@
### Overlay Operator
Overlay one map layer with many other map layers!

### Various Image filter
### Various Image Filter
* Gaussian Blur
* Box Blur
* Embose
Expand All @@ -34,10 +34,12 @@ You also can buld your new image filter by easily extending the photo filter!

### Various Image Type
* PNG
* JPEG
* JPG
* GIF
* More!

You also can support your desired image type by easily extending image generator! (JPG format is temporarily unavailable due to the lack of OpenJDK support)

You also can support your desired image type by easily extending the photo filter!


### Current Visualization effect
Expand All @@ -50,7 +52,7 @@ You also can support your desired image type by easily extending the photo filte
You also can build your new self-designed effects by easily extending the visualization operator!

### Example
Here is [a runnable single machine exmaple code](https://github.com/jiayuasu/GeoSpark/blob/master/src/main/java/org/datasyslab/babylon/showcase/Example.java). You can clone this repository and directly run it on you local machine!
Here is [a runnable single machine exmaple code](https://github.com/DataSystemsLab/GeoSpark/blob/master/src/main/java/org/datasyslab/babylon/showcase/Example.java). You can clone this repository and directly run it on you local machine!

### Scala and Java API
Please refer to [Babylon Scala and Java API](http://www.public.asu.edu/~jiayu2/geospark/javadoc/latest/).
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/datasyslab/babylon/core/ImageGenerator.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,4 @@ public boolean SaveAsFile(JavaPairRDD<Integer,ImageSerializableWrapper> distribu
* @throws Exception the exception
*/
public abstract boolean SaveAsFile(BufferedImage pixelImage, String outputPath, ImageType imageType) throws Exception;
}
}
11 changes: 4 additions & 7 deletions src/main/java/org/datasyslab/babylon/core/VisualizationOperator.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import org.datasyslab.geospark.spatialRDD.SpatialRDD;

import com.vividsolutions.jts.geom.Envelope;
import com.vividsolutions.jts.geom.Geometry;
import com.vividsolutions.jts.geom.LineString;
import com.vividsolutions.jts.geom.Point;
import com.vividsolutions.jts.geom.Polygon;
Expand Down Expand Up @@ -286,7 +287,7 @@ protected JavaPairRDD<Integer, Long> ApplyPhotoFilter(JavaSparkContext sparkCont
{

@Override
public Iterable<Tuple2<Integer, Long>> call(Iterator<Tuple2<Integer, Long>> currentPartition) throws Exception {
public List<Tuple2<Integer, Long>> call(Iterator<Tuple2<Integer, Long>> currentPartition) throws Exception {
//This function will iterate all tuples within this partition twice. Complexity 2N.
//First round, iterate all tuples to build a hash map. This hash map guarantees constant <key, value> access time.
// Another way is to do a sort on the data and then achieve constant access time in the second round. Complexity is N*log(N).
Expand Down Expand Up @@ -410,7 +411,7 @@ protected boolean RenderImage() throws Exception
{

@Override
public Iterable<Tuple2<Integer, ImageSerializableWrapper>> call(Iterator<Tuple2<Integer, Color>> currentPartition)
public List<Tuple2<Integer, ImageSerializableWrapper>> call(Iterator<Tuple2<Integer, Color>> currentPartition)
throws Exception {
BufferedImage imagePartition = new BufferedImage(partitionIntervalX,partitionIntervalY,BufferedImage.TYPE_INT_ARGB);
Tuple2<Integer,Color> pixelColor=null;
Expand Down Expand Up @@ -551,10 +552,6 @@ public Iterable<Tuple2<Integer, Long>> call(Object spatialObject) throws Excepti
{
return RasterizationUtils.FindPixelCoordinates(resolutionX,resolutionY,datasetBoundary,(Point)spatialObject,reverseSpatialCoordinate);
}
else if(spatialObject instanceof Envelope)
{
return RasterizationUtils.FindPixelCoordinates(resolutionX,resolutionY,datasetBoundary,(Envelope)spatialObject,reverseSpatialCoordinate);
}
else if(spatialObject instanceof Polygon)
{
return RasterizationUtils.FindPixelCoordinates(resolutionX,resolutionY,datasetBoundary,(Polygon)spatialObject,reverseSpatialCoordinate);
Expand All @@ -565,7 +562,7 @@ else if(spatialObject instanceof LineString)
}
else
{
throw new Exception("[VisualizationOperator][Rasterize] Unsupported spatial object types.");
throw new Exception("[VisualizationOperator][Rasterize] Unsupported spatial object types. Babylon only supports Point, Polygon, LineString");
}
}
});
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/datasyslab/babylon/extension/imageGenerator/NativeJavaImageGenerator.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ public boolean SaveAsFile(BufferedImage pixelImage, String outputPath, ImageType
}


}
}
2 changes: 1 addition & 1 deletion src/main/java/org/datasyslab/babylon/extension/imageGenerator/SparkImageGenerator.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ public boolean SaveAsFile(BufferedImage pixelImage, String outputPath, ImageType
}


}
}
4 changes: 2 additions & 2 deletions src/main/java/org/datasyslab/babylon/showcase/Example.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ public static boolean buildChoroplethMap(String outputPath)
spatialRDD.spatialPartitioning(GridType.RTREE);
queryRDD.spatialPartitioning(spatialRDD.grids);
spatialRDD.buildIndex(IndexType.RTREE,true);
JavaPairRDD<Polygon,Long> joinResult = JoinQuery.SpatialJoinQueryCountByKey(spatialRDD,queryRDD,true);
JavaPairRDD<Polygon,Long> joinResult = JoinQuery.SpatialJoinQueryCountByKey(spatialRDD,queryRDD,true,false);

ChoroplethMap visualizationOperator = new ChoroplethMap(1000,600,USMainLandBoundary,false);
visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.RED, true);
Expand Down Expand Up @@ -261,4 +261,4 @@ public static void main(String[] args) throws IOException {
sparkContext.stop();
}

}
}
22 changes: 22 additions & 0 deletions src/main/java/org/datasyslab/geospark/formatMapper/FormatMapper.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,16 @@
package org.datasyslab.geospark.formatMapper;

import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;

import org.datasyslab.geospark.enums.FileDataSplitter;

import com.vividsolutions.jts.geom.Coordinate;
import com.vividsolutions.jts.geom.Geometry;
import com.vividsolutions.jts.geom.GeometryFactory;
import com.vividsolutions.jts.geom.LineString;

// TODO: Auto-generated Javadoc
/**
* The Class FormatMapper.
Expand All @@ -29,6 +36,21 @@ public abstract class FormatMapper implements Serializable{
/** The carry input data. */
public boolean carryInputData = false;

/** The fact. */
public GeometryFactory fact = new GeometryFactory();

/** The line split list. */
public List<String> lineSplitList;

/** The coordinates list. */
public ArrayList<Coordinate> coordinatesList;

/** The coordinates. */
public Coordinate[] coordinates;

/** The spatial object. */
Geometry spatialObject = null;

/**
* Instantiates a new format mapper.
*
Expand Down
7 changes: 2 additions & 5 deletions src/main/java/org/datasyslab/geospark/formatMapper/LineStringFormatMapper.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import com.vividsolutions.jts.geom.GeometryFactory;
import com.vividsolutions.jts.geom.LineString;
import com.vividsolutions.jts.geom.MultiLineString;
import com.vividsolutions.jts.geom.Point;
import com.vividsolutions.jts.io.WKTReader;

// TODO: Auto-generated Javadoc
Expand Down Expand Up @@ -58,13 +59,9 @@ public LineStringFormatMapper(Integer startOffset, Integer endOffset, FileDataSp
/* (non-Javadoc)
* @see org.apache.spark.api.java.function.Function#call(java.lang.Object)
*/
public Iterable<Object> call(String line) throws Exception {
public List<Object> call(String line) throws Exception {
List result= new ArrayList<LineString>();
Geometry spatialObject = null;
MultiLineString multiSpatialObjects = null;
GeometryFactory fact = new GeometryFactory();
List<String> lineSplitList;
ArrayList<Coordinate> coordinatesList;
int actualEndOffset;
switch (splitter) {
case CSV:
Expand Down
8 changes: 2 additions & 6 deletions src/main/java/org/datasyslab/geospark/formatMapper/PointFormatMapper.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@

import com.vividsolutions.jts.geom.Coordinate;
import com.vividsolutions.jts.geom.Geometry;
import com.vividsolutions.jts.geom.GeometryFactory;
import com.vividsolutions.jts.geom.MultiPoint;
import com.vividsolutions.jts.geom.Point;
import com.vividsolutions.jts.io.WKTReader;
Expand Down Expand Up @@ -57,13 +56,10 @@ public PointFormatMapper(Integer startOffset, Integer endOffset, FileDataSplitte
/* (non-Javadoc)
* @see org.apache.spark.api.java.function.Function#call(java.lang.Object)
*/
public Iterable<Object> call(String line) throws Exception {
List result= new ArrayList<Point>();
Geometry spatialObject = null;
public List call(String line) throws Exception {
MultiPoint multiSpatialObjects = null;
GeometryFactory fact = new GeometryFactory();
List<String> lineSplitList;
Coordinate coordinate;
List result= new ArrayList<Point>();
switch (splitter) {
case CSV:
lineSplitList = Arrays.asList(line.split(splitter.getDelimiter()));
Expand Down
10 changes: 3 additions & 7 deletions src/main/java/org/datasyslab/geospark/formatMapper/PolygonFormatMapper.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import com.vividsolutions.jts.geom.GeometryFactory;
import com.vividsolutions.jts.geom.LinearRing;
import com.vividsolutions.jts.geom.MultiPolygon;
import com.vividsolutions.jts.geom.Point;
import com.vividsolutions.jts.geom.Polygon;
import com.vividsolutions.jts.io.WKTReader;

Expand Down Expand Up @@ -58,16 +59,11 @@ public PolygonFormatMapper(Integer startOffset, Integer endOffset, FileDataSplit
/* (non-Javadoc)
* @see org.apache.spark.api.java.function.Function#call(java.lang.Object)
*/
public Iterable<Object> call(String line) throws Exception {
List result= new ArrayList<Polygon>();
Geometry spatialObject = null;
public List call(String line) throws Exception {
MultiPolygon multiSpatialObjects = null;
GeometryFactory fact = new GeometryFactory();
List<String> lineSplitList;
ArrayList<Coordinate> coordinatesList;
Coordinate[] coordinates;
LinearRing linear;
int actualEndOffset;
List result= new ArrayList<Polygon>();
switch (splitter) {
case CSV:
lineSplitList = Arrays.asList(line.split(splitter.getDelimiter()));
Expand Down
Loading

0 comments on commit 7a8f8f6

Please sign in to comment.