Expose stop values, not only colors, to `parsePalette`

[jo-chemla]

Hi there,

First of all, thanks for this great utility! When playing with it, I also came across the impressive ~7280 colorramps cpt-city archive, and am using some topo/topobath ramps to display terrain on maplibre via the recent hypso color-relief layer-type.

I've got a few questions relative to parsing the cpt-city CPT strings

• I want to parse the cpt string content into a maplibre paint color-stops like shown below. This is basically an array which starts with interpolation, linear, elevation to declare that the interpolation should happen on the elevation attribute, with linear instead of discrete steps, and then the array declares the stop values and associated colors.
  After fiddling with cpt2js parsePalette(cptString), I can retrieve all the color values array by calling palette.colors() without any args
  But I cannot find the equivalent to retrieve all stops values - tested with domain, out, range, classes etc. This would be useful to convert the resulting cpt2js palette into a maplibre paint style array, and I have the intuition this is just me misinterpreting the options returned by the parsed object. According to the chromajs scale.domain doc, it should be domain, but in my case domain returns only the [min, max] range full range without any stops.

{
  "type": "color-relief",
  "source": "terrainSource",
  "paint": {
    "color-relief-opacity": 1,
    "color-relief-color": [
      "interpolate",
      ["linear"],
      ["elevation"],
      0, "rgb(112, 209, 255)",
      12.88581315, "rgb(113, 211, 247)",
      // ...
    ],
  }
}

• Some colorramps of cpt-city, especially for topobath & topography+bathymetry, have some stops duplicated, where eg stop 0 is often present twice - so stop values are not strictly increasing. Are these correctly parsed and fixed by cpt2js? Testing on the demo it does seem so which is already great.

• finally, would it make sense to include in the cpt2js library - which is made for parsing rather than data - the cpt-city database itself?

  • As said above, there are 7280 colorramps in the whole cpt-city database, which is a pretty large dataset that can be retrieved from pkg Total, it amounts to a pretty large 4.22MB zipped archive, so probably out-of-scope for cpt2js.
  • The copyright information associates next to the data companion COPYING.xml files which describe the type of license (via a pretty free-form field) and whether the ramp could be redistributed via a tag with attribute: <distribute><qgis distribute="yes/no" license="..."/></distribute>.
  • by parsing these, there seem to be 4891 colorramps with license <qgis distribute="yes" .../> and 3806 colorramps with an open-licence (among CC, MIT, Apache, BSD, GPL), so still a pretty large database.
  • It might make more sense to only include the colorramps from cpt-city selection listed on homepage under section Selections from the archive like topo or topobath, which results in only a few 100kBs of data. These can be retrieved by looking into the views directory, which includes an xml representation of the different selections.

[jo-chemla]

This seem to be a limitation of chromajs underlying library, see thread here gka/chroma.js#376 . Indeed, calling scale.domain() only outputs domain [min, max] range when rather than all stop values - the way it works when doing scale.colors() which returns all colors, one for each stop. If so, either the chromajs lib can update its domain function, or the cpt2js lib could make the choice to allow the user to return the domain as well, maybe by passing an additional flag.

cpt2js/src/parse.ts

Line 110 in a65aa16

let palette = chroma.scale(colors as any).domain(domain).mode(mode);

After digging a bit through both chromajs and cpt2js

• I've suggested a change to chroma.js to allow for exposing the domain stops on the palette Scale itself here Retrieving the scale.domain() stop-values, not only min/max gka/chroma.js#376 (comment) , we'll see what the maintainer thinks about it.

• In the meantime, we could update cpt2js to expose the stops domain as well, by editing the functions like shown in below details, returning {palette, domain} from parsePaletteArray, and then returning only the palette in current parsePalette to keep current behavior, and returning both palette and domain in an additional parsePaletteWithStops function:

parsePaletteWithStops suggestion to return {palette, domain}

function parsePaletteArray(paletteArray: PaletteArray, { bounds = [0, 1], mode = DEFAULT_MODE }: ParseOptions & { mode?: InterpolationMode } = {}): {palette: Scale, domain: number[]} {
  const colors: (object | string)[] = [];
  const domain: number[] = [];
  let nodata;
  for (let [value, color] of paletteArray) {
    const parsedValue = parseValue(value, bounds);
    const parsedColor = parseColor(color, mode);

    if (parsedValue != null) {
      colors.push(parsedColor);
      domain.push(parsedValue);
    } else if (parsedValue === null) {
      nodata = parsedColor;
    } else {
      // ignore
    }
  }

  let palette = chroma.scale(colors as any).domain(domain).mode(mode);
  if (typeof nodata !== 'undefined') {
    palette = (palette as any).nodata(nodata);
  }
  return {palette, domain};
}

function parsePaletteText(paletteText: string, { bounds = [0, 1] }: ParseOptions = {}): {palette: Scale, domain: number[]} {
  const { paletteArray, mode } = parsePaletteTextInternal(paletteText);
  return parsePaletteArray(paletteArray, { bounds, mode });
}

export function parsePalette(palette: Palette, { bounds = [0, 1] }: ParseOptions = {}): Scale {
  if (typeof palette === 'string') {
    return parsePaletteText(palette, { bounds }).palette;
  } else if (Array.isArray(palette)) {
    return parsePaletteArray(palette, { bounds }).palette;
  } else {
    throw new Error('Invalid format');
  }
}

export function parsePaletteWithStops(palette: Palette, { bounds = [0, 1] }: ParseOptions = {}): {palette: Scale, domain: number[]} {
  if (typeof palette === 'string') {
    return parsePaletteText(palette, { bounds });
  } else if (Array.isArray(palette)) {
    return parsePaletteArray(palette, { bounds });
  } else {
    throw new Error('Invalid format');
  }
}

EDIT: I also had to trim each line string before processing because in the cpt-city resource there are a lot a CPT strings where lines have heading/trailing \t tab character. I did it around these lines:

Trimming each line for spacing chars

function parseValue(value_: string | number, bounds: [number, number]): number | null | undefined {
  if (typeof value_ === 'string') {
    const value = value_.trim();
    // ... 
  }
}


function isGmt4Text(lines: string[]): boolean {
  return lines.some(line_ => {
    const line = line_.trim();
    // ...
  });
}

function isGmt5Text(lines: string[]): boolean {
  return lines.some(line_ => {
    const line = line_.trim();
    // ...
  });
}

function parsePaletteTextInternal(paletteText: string): { paletteArray: PaletteArray, mode?: InterpolationMode } {
  // ...
  for (let paletteLine_ of paletteLines) {
    const paletteLine = paletteLine_.trim();
    // ...
  }
}

[zakjan]

Hi, it's a good suggestion for improving support with the latest Maplibre. Check the output format of parsePaletteTextInternal. Does it return an array which would be already compatible? If so, I think the easiest would be exposing this function externally.

[jo-chemla]

Thanks for getting back. The second post did include code snippets of the modifications I made - not obvious but the sections can be unfolded. Here is the diff view that can make it more explicit what changes I applied to the original code.

If these do look correct to you, I can make a PR pretty quickly!

Note:

• the first modifications are related to the trimming of CPT files,, since cpt-view had a lot of palettes which include various amount of tabs as first chars that have to be trimmed for the parsing to behave correctly
• the last modification is what makes it possible to export both color list and stop-values = domain, with an edition of the parsePaletteArray to return both palette (original chroma export) and domain, and a new parsePaletteWithStops that exports both to keep the previous cpt2js behavior and avoid breaking changes

Hope this helps!

diff --git original.ts modified.ts

@@ function isGmt4Text(lines: string[]): boolean {
-  return lines.some(line => {
-    if (!isLineComment(line)) {
-      if (line.split(LINE_SEPARATOR_REGEX).length >= 8) {
+  return lines.some(line_ => {
+    const line = line_.trim();
+    if (!isLineComment(line)) {
+      if (line.split(LINE_SEPARATOR_REGEX).length >= 8) {
         return true;
       }
     }
@@ function isGmt5Text(lines: string[]): boolean {
-  return lines.some(line => {
-    if (!isLineComment(line)) {
-      if (line.match(/\d+\-\d+\-\d+/) || line.match(/\d+\/\d+\/\d+/)) {
+  return lines.some(line_ => {
+    const line = line_.trim();
+    if (!isLineComment(line)) {
+      if (line.match(/\d+\-\d+\-\d+/) || line.match(/\d+\/\d+\/\d+/)) {
         return true;
       }
     }
@@ export function parsePaletteTextInternal(paletteText: string) {
-  for (let paletteLine of paletteLines) {
-    const fields = paletteLine.split(LINE_SEPARATOR_REGEX);
+  for (let paletteLine_ of paletteLines) {
+    const paletteLine = paletteLine_.trim();
+    const fields = paletteLine.split(LINE_SEPARATOR_REGEX);
@@ function parseValue(value: string | number, bounds: [number, number]) {
-function parseValue(value: string | number, bounds: [number, number]): number | null | undefined {
-  if (typeof value === 'string') {
-    if (value[value.length - 1] === '%') {
-      const percentage = parseFloat(value) / 100;
+function parseValue(value_: string | number, bounds: [number, number]): number | null | undefined {
+  if (typeof value_ === 'string') {
+    const value = value_.trim();
+    if (value[value.length - 1] === '%') {
+      const percentage = parseFloat(value) / 100;
         if (percentage < 0 || percentage > 1) {
           throw new Error(`Invalid value for a percentage ${value}`);
         }
@@ function parseValue(value: string | number, bounds: [number, number]) {
-  } else if (typeof value === 'number') {
-    return value;
+  } else if (typeof value_ === 'number') {
+    return value_;
@@ function parsePaletteArray(paletteArray: PaletteArray, { bounds = [0, 1], mode = DEFAULT_MODE } = {}) {
-function parsePaletteArray(...): Scale {
+function parsePaletteArray(...): {palette: Scale, domain: number[]} {
@@
-  return palette;
+  return {palette, domain};
@@ function parsePaletteText(paletteText: string, { bounds = [0, 1] }: ParseOptions = {}) {
-function parsePaletteText(...): Scale {
+function parsePaletteText(...): {palette: Scale, domain: number[]} {
@@ export function parsePalette(palette: Palette, { bounds = [0, 1] }: ParseOptions = {}) {
-    return parsePaletteText(palette, { bounds });
+    return parsePaletteText(palette, { bounds }).palette;
@@ export function parsePalette(palette: Palette, { bounds = [0, 1] }: ParseOptions = {}) {
-    return parsePaletteArray(palette, { bounds });
+    return parsePaletteArray(palette, { bounds }).palette;
+
+export function parsePaletteWithStops(palette: Palette, { bounds = [0, 1] }: ParseOptions = {}): {palette: Scale, domain: number[]} {
+  if (typeof palette === 'string') {
+    return parsePaletteText(palette, { bounds });
+  } else if (Array.isArray(palette)) {
+    return parsePaletteArray(palette, { bounds });
+  } else {
+    throw new Error('Invalid format');
+  }
+}

[jo-chemla]

Edit just saw the author of chromajs is open to exposing the stops/domain directly, simply calling scale.domain()
gka/chroma.js#376 (comment)

So it's probably safer to wait for the PR there to be submitted and merged, before editing another PR here.

[jo-chemla]

The PR to chromajs got merged, fixed, and release in 3.2.0!

One can now access the stop values of the colorramp simply by calling the domain function with no arguments passed like so scale.domain()

https://github.com/gka/chroma.js/releases/tag/v3.2.0

I'll draft a PR to cpt2js based on that when I've got some spare time!

[jo-chemla]

The fact that my chromajs PR gka/chroma.js#376 (comment) got merged means we now only need trimming white spaces and tabs from each lines for every cpt-city CPT files to be correctly parsed by cpt2js.

I did a micro PR to cpt2js with the fixes here #6

Below is the code to convert the cpt multiline string to a maplibre colorramp. I'm not sure whether you would also want this to be part of the PR, and where we could put that code - but if it can help anyone, here it is. The fixDomain function is necessary because maplibre expects colorramp stops to be all different, strictly ascending order, so I'm shifting by a tiny amount, 1% of the stop step.

Best,

function fixDomain(domain: number[]) {
  const domainFixed = [...domain];
  for (let i = 1; i < domain.length - 1; i++) {
    if (domain[i] == domain[i - 1]) {
      domainFixed[i] = domain[i - 1] + 0.01 * (domain[i + 1] - domain[i - 1]);
    } 
  }
  return domainFixed;
}

function chromajsScaleToMaplibre(paletteScale: Scale) {
  const colors = paletteScale.colors()
  const domain = paletteScale.domain()
  const domainFixed = fixDomain(domain)
  return [
      "interpolate",
      ["linear"],
      ["elevation"],
      ...domainFixed.flatMap((d: number, i: number) => [d, colors[i]]) 
  ]
}

[zakjan]

Hi, thanks for the changes in chroma-js. I'm having issues in upgrading chroma-js 2->3 though. This package avoids importing the full chroma-js bundle due to a large bundle size increase. Custom bundle cannot be built anymore because src/Color.js is not exported. Light bundle cannot be imported because index-light.js file (referenced here) is missing (renamed here). Thoughts?

[zakjan]

PRs:
gka/chroma.js#382
gka/chroma.js#383

[jo-chemla]

Thanks for the PRs, hope they'll make it to the next chromajs release!