Developing Grafana Worldmap NG

About

Various improvements to the Grafana Worldmap Panel have been made in order to integrate important contributions from the community and by adding some other features from our pen.

What’s ahead

The following posts will give you an idea about how Grafana Map Panel was improved around April/May/June 2019 on behalf of The Hiveeyes Project.

Outlook

Beginning 2020, the development continues on behalf of the Panodata Initiative.

1 Like

Hybrid “table+json(p)” Location Data

About

The “Location Data” options “table+json” and “table+jsonp” unlock the ability to use a metric source formatted as table as the primary source for location data information while looking up the corresponding label from a JSON document, extending what the plain “table” option already does by combining existing processing/transformation paths in a convenient manner.

TLDR;

If you prefer to read code, the corresponding software test for this feature might be good place to start.

/*
 * Hybrid "table+json(p)" Location Data
 *
 * We are testing the "table+json" and "table+jsonp" location data
 * sources here. They are used when basic location information is
 * coming from table data but humanized labels are resolved from a
 * JSON/JSONP data source by appropriate field name mapping.
 */

Details

This section gives a detailed overview over appropriate configuration of 1. the data source metric, 2. the JSON document for resolving corresponding humanized labels and 3. the respective Grafana Worldmap Panel settings to successfully map all fields and resources.

Configuration of data source metric, JSON document and Grafana Worldmap settings

Primary data source: Metric as Table

The primary location data source is determined through the metric we are querying from. Specifically, this is an InfluxDB data source which is formatted as “Table”. The response yields the fields/tags “station_id”, “geohash” and “metric”.

Request

SELECT last("P1") AS "metric" 
FROM "ldi_readings" 
WHERE ("station_id" =~ /^$ldi_station_id$/) AND $timeFilter 
GROUP BY "geohash", "station_id"

image

Response

Data source response
{
    "results": [
        {
            "series": [
                {
                    "columns": [
                        "time",
                        "metric"
                    ],
                    "name": "ldi_readings",
                    "tags": {
                        "geohash": "u0wt1tdd0x1d",
                        "station_id": "19"
                    },
                    "values": [
                        [
                            1555738566000,
                            5999.12
                        ]
                    ]
                },
                {
                    "columns": [
                        "time",
                        "metric"
                    ],
                    "name": "ldi_readings",
                    "tags": {
                        "geohash": "u0wt6pv2qqhz",
                        "station_id": "28"
                    },
                    "values": [
                        [
                            1555738438000,
                            9.13
                        ]
                    ]
                }
            ],
            "statement_id": 0
        }
    ]
}

Lookup data source: JSON document

The effective label is resolved by looking it up from a corresponding JSON document acquired and processed exactly like the “json(p) endpoint” options
image

which looks like

[
    {
        "key": "19",
        "name": "Alte Dorfstra\u00dfe, Birkach, Stuttgart, Baden-W\u00fcrttemberg, DE"
    },
    {
        "key": "28",
        "name": "Ulmer Stra\u00dfe, Wangen, Stuttgart, Baden-W\u00fcrttemberg, DE"
    },
]

The value from the table field designated by “Location Name Field” (here “station_id”, see response above)
image

will be used to run a lookup against the “key” field of the JSON document as depicted above. If there’s a match, the value associated with the corresponding name will be returned.

Full configuration screenshot: "Map Data Options" using "Location Data: table+json"

Example: LDI map Germany

We are using this to display synthesized address information on a map populated with data coming from luftdatenpumpe through InfluxDB and PostGIS data sources, see also LDI data plane v2.


– Source: luftdaten.info Karte Deutschland

Read more...

Introduction

In one of our data planes, we are feeding data from the luftdaten.info project into an InfluxDB database using luftdatenpumpe.

Details

As the data from InfluxDB is designated by "FORMAT AS Table" into the Worldmap Panel, we couldn’t find a way to acquire the Location Name from a JSON/JSONP data source as it is possible with the "FORMAT AS Time series" variant already.

This PR accounts for that. When choosing to load location data using the new mapping options "table+json" or "table+jsonp", it is possible to load location information from JSON. The value of the data field designated by the "Location Name Field" will be used as a key for looking up the appropriate record in the JSON dataset. If the lookup fails, the specific code path is backward compatible and should produce the same results as before by using the value of the "Location Name Field" itself.

Outcome

By applying that to our installation, we have been able to display a long location name acquired from ldi-stations.json on the map overlay of LDI stations map without having to store the name inside the InfluxDB measurement table itself. We don’t even want to keep a short name there, the lookup is solely done by using a single location identifier, the numeric station_id:

image

References

Optimizations for small maps

Introduction

As we are displaying a number of small-sized map widgets on dashboards like the weather.hiveeyes.org landing page or Vergleichende Stationsansicht: UBA ./. LDI korrigiert, we wanted to optimize different things on the display side.

All amendments can be conditionally configured through appropriate plugin settings, so they are completely optional and behave backward-compatible by not modifying the behavior for map panels already configured.

Overview

We are currently keeping the amendments to Worldmap in our develop branch.

Over the course of the next posts, we want to outline the changes and provide some background information about it.

See also

Disable auto-panning for popup labels

Problem

When moving the mouse at the corners of the map, Leaflet starts panning around in order to display the popup without clipping it. In general, this is a good thing.

However, it turns out this isn’t possible on a small view area at all: It happens way too quick when navigating through a small map and just makes the user crazy.

label-panning-woes

Solution


Documentation - Leaflet - a JavaScript library for interactive maps

We added the autoPanLabels option to accommodate for that, which defaults to true so it doesn’t change the default behavior.
image

1 Like

Disable automatic width-calculation of popup labels

Problem

The popup label content gets clipped on small maps.

Thoughts

  • As we just disabled auto panning, the popup size and/or layout should be adjusted appropriately to improve display on small maps.
  • Alternatively, the popup contents might also be displayed at an arbitrary out-of-band location.

Solution

We added the autoWidthLabels option to accommodate for that which defaults to true so it doesn’t change the default behavior. When set to false,
image
the popup layout looks more pleasant.

Before / After

image image
– Example: [Feinstaub PM10] Vergleich: Umweltbundesamt (UBA) vs. Korrektur luftdaten.info (LDI)

1 Like

Hide zoom control and attribution label

Problem

Both control elements (zoom control and attribution) cover too much space on small maps.
image

image

Thoughts

  • Either we should be able resize the controls or hide them altogether.
  • Alternatively, they could be displayed outside of the map in a separate footer area or even at an arbitrary out-of-band location.

Solution

We added the showZoomControl and showAttribution options to accommodate for that. Both default to true so they don’t change the default behavior. When set to false ,
image
small map widgets look more pleasant to us.

Before / After

image image

Appendix: At runtime

// Hide zoom control and attribution.
$('.leaflet-control-zoom').hide();
$('.leaflet-control-attribution').hide();

Out-of-band display of map legend

Problem

The legend sometimes covers too much room on small maps.
image

Thoughts

So let’s just display it elsewhere on the dashboard, right?

Solution

Let’s define a Text-Box which should act as a container element.
image

In the panel settings, the output can be redirected to another DOM element by specifying a jQuery element-selector expression.
image

Before / After


1 Like

Cool! Can you please spend a <center>-tag, too? For the pleasantness!

In order to not encourage copyright-infringements, I suggest a read-only multi-line text-field with the correct attribution (probably prefixed with the panel’s title) in Hypertext (or Markdown), introduced with the words “In order to respect the copyright of many cool people, you should now add this attribution-text somewhere else on the dashboard:”, suddenly appearing after unchecking the “show attribution”-box.

.oO( or shall it automagically create a text-field panel with the attribution-text/html, like within the new legend-option? )

1 Like

Out-of-band attribution for Grafana Map Panel and beyond

I second that. We will see how this would be implemented exactly, thanks for sharing your thoughts.

Labels barely visible on Grafana light theme

Problem

When using the Grafana light theme, popover labels are barely visible.

Solution

dtheb supplied a fix for that the other day. Thanks!

Before

image

After

image

1 Like

Clickthrough links

Introduction

Sticky labels

Daniel Lee already added the feature "stickyLabels": true for this. Thanks!

Research

We know this feature is well-requested already.

Solutions

Implementation

Screens

Variant 1: Assign URL from panel settings.
image

Variant 2: Assign URL from database field.
image

Description

There are two ways of attaching clickthrough-links to circles. Either assign the URL through the corresponding “Clickthrough URL” editor option or take the URL from a database field of the respective data point by using the “Link field” option. By running the outcome through Grafana’s variable interpolation machinery, both dashboard and data point variables will get interpolated at runtime into the strings typed in there as you can see from the first screenshot above.

All dashboard variables will get interpolated 1:1 while values coming from a data point will get prefixed by "__field_" to be able to tell them apart and avoid collisions.

Using the “Clickthrough Window” setting, the panel operator is able to control the designated target browsing context (formerly called “frame”), which is either a named window or one of the well known literal strings “_blank”, “_self”, “_parent” or “_top” with their respective semantics listed below. Using the named window variant, it is easily possible to support dual- and multiscreen-scenarios by just arranging multiple browser windows on different screens.

Currently, the clickthrough-links are attached to the respective circle. Injecting and attaching links into the popover label (see “Sticky labels”) is reserved for some future iteration on that.

Value Description
_self Open the link in the current frame. This is the default behavior.
_blank Open the linked document in a new tab or window, depending on browser configuration.
_parent Open the link in the parent frame, otherwise resorting to _self.
_top Open the link in the top-level frame, otherwise resorting to _self.
frame name Open the link in the named frame.

https://wiki.selfhtml.org/wiki/Referenz:HTML/Attribute/target (sic!)

Details

This is a simple implementation of a panel option being used to navigate to a new URL on click.

https://github.com/hiveeyes/grafana-worldmap-panel/commit/09432e8672d94658273d56856d94e892467f183d

This adds the possibility to obtain the clickthrough-link from the database through the linkField editor option.

Adds clickthrough links mappable from data for circles · panodata/panodata-map-panel@129dd98 · GitHub

Add basic variable interpolation based on keys from dataPoint.
Saying that, all metric values have been added to the dataPoint
beforehand by prefixing them with __field_ to allow for setting
things like clickthroughURL: /url/to/$__field_station_id if
there’s a field called station_id in the current datapoint.

Combine both clickthrough link implementations #129 and #190 · panodata/panodata-map-panel@7a2e825 · GitHub

Use the regular Grafana templating mechanism for interpolating variables into clickthrough links, thus interpolating dashboard and dataPoint variables now.

Use the regular Grafana templating mechanism for interpolating variables · panodata/panodata-map-panel@df86b0d · GitHub

Acknowledgements

Shoutouts to James Nicholls and Leonhard Haas for sharing the contributions referenced above through #129 and #190. Thanks!

1 Like

Variable interpolation all the way down

Apply Grafana-style variable interpolation to all panel settings.

This outlines various details about how the functionality for interpolating values into panel settings has been improved through variables originating from different kinds of sources.

As of now, the interpolation machinery honors the well known Grafana dashboard variables as well as request parameters (prefixed by "request_") and data point fields (prefixed by "__field_").

1. Pull variables from dashboard variables

This enables the interpolation of dashboard variables into each of the panel control settings (source, commit).

Example

An example would be to have a variable called “countrycode” which gets populated by a database query defined by an SQL statement.

SELECT
  country_code AS __value,
  country_name AS __text
FROM
  stations
ORDER BY
  country_code

Then, assigning a value like that to the JSON endpoint URL panel control setting, the machinery would interpolate the selected value assigned to the $countrycode variable appropriately.

image

2. Pull variables from data point fields

Add basic variable interpolation based on keys from dataPoint. Saying that, all metric values have been added to the data point beforehand by prefixing them with "__field_" (source).

Example

If there’s a field called station_id in the current data point, using this inside a panel control option is easy by addressing it with $__field_station_id.

image

3. Pull variables from URL request parameters

All request parameters from the request URL are now available in the variable interpolation dictionary, prefixed by "request_". This enables using request variables in all panel control options (source, commit).

Example

So, when invoking the dashboard with query parameters like
image
you would be able to use these variables from inside panel control options by assigning them appropriately.

image

4. Override panel control options with URL request parameters

By using request parameters prefixed with “panel-”, it is possible to override the panel control options at runtime (source, commit).

Examples

# Disable zoom control.
https://daq.example.org/d/D1Fx12kWk/magic-dashboard?panel-showZoomControl=false 

# Add clickthrough-link, interpolating the geohash value.
?panel-clickthroughUrl=/path/to/?geohash=$__field_geohash

# Crazyness:
# Interpolate a custom request variable into
# the `clickthroughUrl` setting at runtime.
https://daq.example.org/d/D1Fx12kWk/magic-dashboard?panel-clickthroughUrl=/path/to/?foobar=$request_foobar&foobar=hello

Add map centering to “First geohash”

image

Improvement

We already support the last geohash center, it might be useful to also center on the first geohash received (commit).

Acknowlegements

Thanks for sharing that through #156, Fabien Pomerol.

Add control options "mapFitData" and "mapZoomByRadius"

Both editor options provide a more convenient way to determine the viewport (center and zoom level) of the map view. Both features are using Leaflet’s getBoundsZoom() method, which

returns the maximum zoom level on which the given bounds fit to the map view in its entirety.

As the computation of "mapFitData" is based on the real data which actually covers a certain boundary, this should mostly do what I mean.

Fit to data

Enabling this option will choose an optimal center and zoom level for covering the displayed data (source).

image

Zoom radius (km)

This will use a bounding box of a respective circle with the designated radius around the chosen center for determining the viewport (source).

image

Reference

Add control options `mapFitData` and `mapZoomByRadius`. · panodata/panodata-map-panel@ebec3f6 · GitHub

Problem with “Missing geohash value”

TLDR;

One of our data feeds suddenly started yielding empty "geohash" values into the database. The Worldmap Panel was not amused about it.

Problem

Reason

Proposal

The Worldmap Panel should be able to compensate for that by operating it with an appropriate control option.


Solution: New options for ignoring empty or invalid geohash values

Complementing the options to ignore empty and zero metric values, the options to “Ignore empty geohash values” and “Ignore invalid geohash values” optionally should do what they promise (source, commit).

image

image

Improve Kiosk-mode display

Add ignoreEscapeKey option.

While this amends the global behaviour of Grafana and should probably be implemented elsewhere, it helps us now to prevent the user to leave the Kiosk mode.

https://github.com/hiveeyes/grafana-worldmap-panel/commit/eea48c41d5fd338f841ba29e3d031004ab61671d

Add hideTimepickerNavigation option.

This also amends global behaviour so it should be implemented elsewhere. However, it helps us now to improve Kiosk mode displays.

https://github.com/hiveeyes/grafana-worldmap-panel/commit/e379a8b0de93bbc3a3aca4cbe8d9119c6b37f774

Disable strokes or adjust stroke width

Request


Disable strokes

image

Before / After

The default is “strokes enabled” with a stroke width of 3 pixels (left side). After disabling strokes, the circles display without borders (right side).


Adjust stroke width

image

Pen Liner borders.
image

Adjust attribution text

These settings control which alternative text is displayed in the attribution area of the Leaflet map widget (commit).

image

image

Full changelog

v0.3.0-dev

  • Add new mapping options table+json and table+jsonp to retrieve
    location information from JSON endpoint, even for table data.
    The lookup key is the value obtained from the database field
    designated by tableQueryOptions.labelLocationKeyField.
  • Add autoPanLabels and autoWidthLabels to visual option settings.
  • Add zoomControl and attributionControl to visual option settings.
  • Optionally display legend in different DOM element.
  • Increase form field width for JSON endpoint url settings.
  • Fix conditional table options display in settings editor.
  • Add showTableOptions() conditional for signalling any type of table source.
  • Add software tests covering new control options
    • locationData: "table+json"
    • showZoomControl: false
    • showAttribution: false
    • legendContainerSelector
  • Add Makefile for macOS/Homebrew.
  • Fix popover labels text color for light theme (#169).
    Thanks, @dtheb.
  • Add clickthrough option for circles, use fixed url (#129).
    Thanks, @ryft.
  • Add clickthrough option for circles, use urls from data (#190).
    Thanks, @leonhardhaas.
  • Combine both clickthrough link implementations #129 and #190,
    add basic variable interpolation based on keys from dataPoint.
  • Introduce the regular Grafana templating mechanism for interpolating variables
    into clickthrough links. As we are now interpolating dashboard and dataPoint
    variables, the latter one will get prefixed with __field_ to avoid collisions.
  • Apply Grafana-style variable interpolation to all panel settings.
  • Add query parameters into the interpolation dictionary, prefixed by request_.
  • Use request parameters prefixed with “panel-” to optionally override the
    respective control options.
  • Improve map center control handling.
  • Improve clickthroughUrl interpolation.
  • Add control option clickthroughOptions.windowName to open clickthrough target in designated window.
  • Improve centering on Last Geohash.
  • Add support for First Geohash centering (#156).
    Thanks, @fabienpomerol.
  • Add control options mapFitData and mapZoomByRadius.
  • Repaint user interface.
  • Refactor machinery and user interface.
  • Add options ignoreEmptyGeohashValues and ignoreInvalidGeohashValues.
  • Add ignoreEscapeKey option.
  • Add hideTimepickerNavigation option.
  • Add circleOptions.strokeEnabled and circleOptions.strokeWeight.
  • Add options customAttribution and customAttributionText.

grafana-worldmap-panel/CHANGELOG.md at develop · hiveeyes/grafana-worldmap-panel · GitHub