Is there a way to use client side point geojson instead of input file in png format? #8

nishadhka · 2022-04-08T11:00:59Z

nishadhka
Apr 8, 2022

Following the heat map input format used in deck.gl https://deck.gl/docs/api-reference/aggregation-layers/heatmap-layer.
Is there a way to use json point data as follows. Thanks!

/**
   * Data format:
   * [
   *   {COORDINATES: [-122.42177834, 37.78346622], u_component:0.1,v_component: -2.3},
   *   ...
   * ]
   */

Answered by ghost

Apr 8, 2022

I haven't tried this, but...

The image is just forwarded as a uniform, so you should be able to pass another Luma texture (such as the heatmap output).
The heatmap layer internally generates a texture: https://github.com/visgl/deck.gl/blob/73e196c678d28e3196022cd1e283e970768f20f6/modules/aggregation-layers/src/heatmap-layer/heatmap-layer.js#L148
So to combine these 2, there are a couple of options:
- From scalar heat to vector U, V (not what you mentioned in first post):
  - Option A: Modify the particle layer to sample multiple neighbour pixels to calculate U and V on the fly (probably a bad idea?).
  - Option B: Differentiate the heatmap layer output (similar to how you'd create a normalmap) i…

View full answer

ghost · 2022-04-08T11:28:52Z

ghost
Apr 8, 2022

I haven't tried this, but...

The image is just forwarded as a uniform, so you should be able to pass another Luma texture (such as the heatmap output).
The heatmap layer internally generates a texture: https://github.com/visgl/deck.gl/blob/73e196c678d28e3196022cd1e283e970768f20f6/modules/aggregation-layers/src/heatmap-layer/heatmap-layer.js#L148
So to combine these 2, there are a couple of options:
- From scalar heat to vector U, V (not what you mentioned in first post):
  - Option A: Modify the particle layer to sample multiple neighbour pixels to calculate U and V on the fly (probably a bad idea?).
  - Option B: Differentiate the heatmap layer output (similar to how you'd create a normalmap) in an extra pass, before passing it to the particle-layer.
- Direct vector U, V (what you mention in first post):
  - Option A: Generate 2 independent U and V heatmaps, then inject some custom shader code in the particle system to sample both.
  - Option B: Modify the heatmap code (or write your own) so it directly stores 2 channels.

You'll certainly end up writing some custom code for this.

0 replies

zakjan · 2022-04-08T12:42:11Z

zakjan
Apr 8, 2022
Maintainer

Hi @nishadhka, thanks for the suggestion!

ParticleLayer doesn't support this, because it's more similar to BitmapLayer, it expects a gridded texture. As @JannikGM pointed out (thanks!), HeatmapLayer generates a texture from arbitrary points internally, so you could be able to re-use HeatmapLayer code for generating such texture, and use it here.

The current target use case of ParticleLayer is to visualize data from data sources which already provide gridded data, such as GFS. I guess your use case is a bit more low-level, to visualize data from isolated meteorological stations?

5 replies

nishadhka Apr 8, 2022
Author

Thank you @JannikGM for suggestion, will work on the directions and report back here.

@zakjan, the data source is MVT layers of U and V vectors based upon https://deck.gl/docs/api-reference/geo-layers/mvt-layer. I am exploring the MVT format of GFS grid points for visualisation.

The point geojson is created after loading the MVT by conversion of MVT layer properties and point lat long by method datainwgs84 of mvt-layer https://deck.gl/docs/api-reference/geo-layers/mvt-layer#datainwgs84.

The GFS data is made into point vectors and converted into MVT following this, due to restrictions on PBF file size and requirement of keeping all the grid points in every zoom, the vector geojson grid points are made into small number of subset points like 506 points, nx=23, ny=22.

tippecanoe \
    -z11 \
    -r1 \
    --no-tile-compression \
    --output-to-directory u_data \
    GFS_u.geojson

tippecanoe \
    -z11 \
    -r1 \
    --no-tile-compression \
    --output-to-directory u_data \
    GFS_u.geojson

Could able to make initial attempt of getting the MVT point visualised as follows,

The deck.gl standalone code used is as follows, we couldn't able to make the heatmap layer with below code, requesting @JannikGM help in this regard. Getting error of GL_INVALID_OPERATION: Vertex buffer is not big enough for the draw call

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Document</title>
</head>

<body>
  <script src="https://unpkg.com/deck.gl@^8.0.0/dist.min.js"></script>
  <!-- or -->
  <script src="https://libs.cartocdn.com/mapbox-gl/v1.13.0/mapbox-gl.js"></script>
  <link href="https://libs.cartocdn.com/mapbox-gl/v1.13.0/mapbox-gl.css" rel="stylesheet" />

  <div id="map" style="width: 100vw; height: 100vh;"></div>
  <script>

    const deckgl = new deck.DeckGL({
      container: 'map',
      mapStyle: 'https://basemaps.cartocdn.com/gl/dark-matter-gl-style/style.json',
      initialViewState: {
        latitude: 16.4,
        longitude: 68.18,
        zoom: 5,
      },
      controller: true,
    });



    var u_data_array = []
    var v_data_array = []
    

    const onViewportLoad = tiles => {
      tiles.forEach(tile => {
        const dataInWGS84 = tile.dataInWGS84;
        for (point in dataInWGS84) {
          if (dataInWGS84) {
            var u_temp = { COORDINATES: dataInWGS84[0]['geometry']['coordinates'], WEIGHT: dataInWGS84[0]['properties'][0] };
            var v_temp = { COORDINATES: dataInWGS84[0]['geometry']['coordinates'], WEIGHT: dataInWGS84[0]['properties'][1] };
            u_data_array.push(u_temp)
            v_data_array.push(v_temp)
          }
        }
      });
    };

   const uv_layer = new deck.MVTLayer({
      data: 'https://acloudstorage/uv/{z}/{x}/{y}.pbf',
      getLineColor: [192, 192, 192],
      lineWidthMinPixels: 1,
      getLineWidth: f => {
      },
      onViewportLoad
    });

    const u_heat_layer = new deck.HeatmapLayer({
      id: 'u_heat_layer',
      data: u_data_array,
      pickable: false,
      opacity: 0.8,
      getPosition: d => d.geometry.COORDINATES,
      getWeight: d => d.properties.WEIGHT,
      radiusPixels: 35,
      intensity: 1,
      threshold: 0.3,
      aggregation: 'SUM'
    });

    const v_heat_layer = new deck.HeatmapLayer({
      id: 'v_heat_layer',
      data: v_data_array,
      pickable: false,
      opacity: 0.8,
      getPosition: d => d.geometry.COORDINATES,
      getWeight: d => d.properties.WEIGHT,
      radiusPixels: 35,
      intensity: 1,
      threshold: 0.3,
      aggregation: 'SUM'
    });


    deckgl.setProps({ layers: [uv_layer,u_heat_layer, v_heat_layer] });
    console.log(u_data_array )
    console.log(v_data_array )
  </script>

</body>

</html>

zakjan Apr 9, 2022
Maintainer

Do you transform source data from GFS to GeoJSON because of performance reasons?

GFS 0.25deg data consists of 1440x720 points per band. Transformed to PNG (see #9) it takes ~0.5MB. This is a manageable size to be delivered as a single file, no urgent need for tiling yet. You can see the performance at the demo.

High-resolution tiled data such as ECMWF is not supported yet, it's on the roadmap.

nishadhka Apr 9, 2022
Author

Yes, due to following reasons

To give the multi hour data like 120 hours, expected MVT reduces the size.
Without tiling, the projection issues are getting raised. While comparing with different visualisation types, points are moved around one or two pixel(25-50km) moved.

Cool, looking forward to high resolution tiled data.

zakjan Apr 9, 2022
Maintainer

I doubt about reduced size, because by converting raster to vector MVT you add overhead lng/lat coordinates of each point. I would recommend to working with raster only.

I'm interested about the projection issues. Could you create a reproducible bug with more details?

nishadhka Apr 9, 2022
Author

Sure, will update here about projection issues.

Is there a way to use client side point geojson instead of input file in png format? #8

Uh oh!

nishadhka Apr 8, 2022

Replies: 2 comments · 5 replies

Uh oh!

Uh oh!

ghost Apr 8, 2022

Uh oh!

zakjan Apr 8, 2022 Maintainer

Uh oh!

Uh oh!

nishadhka Apr 8, 2022 Author

Uh oh!

zakjan Apr 9, 2022 Maintainer

Uh oh!

nishadhka Apr 9, 2022 Author

Uh oh!

Uh oh!

zakjan Apr 9, 2022 Maintainer

Uh oh!

nishadhka Apr 9, 2022 Author

nishadhka
Apr 8, 2022

Replies: 2 comments 5 replies

ghost
Apr 8, 2022

zakjan
Apr 8, 2022
Maintainer

nishadhka Apr 8, 2022
Author

zakjan Apr 9, 2022
Maintainer

nishadhka Apr 9, 2022
Author

zakjan Apr 9, 2022
Maintainer

nishadhka Apr 9, 2022
Author