Getting started with Flux

influxdb

#1

About Flux

Introduction to Flux

Flux (formerly IFQL) is the new InfluxDB query language and engine we already talked about at The history and future of InfluxDB and Temperatursummen in der Imkerei.

Flux is a lightweight scripting language for querying databases (like InfluxDB) and working with data. It’s part of InfluxDB 1.7 and 2.0, but can be run independently of those.

See also: Flux Specification.

Flux vs. InfluxQL

InfluxQL is InfluxData’s SQL-like query language for interacting with data in InfluxDB. Flux is an alternative to InfluxQL and other SQL-like query languages for querying and analyzing data. It uses functional language patterns making it incredibly powerful, flexible, and able to overcome many of the limitations of InfluxQL. This article outlines many of the tasks possible with Flux but not InfluxQL and provides information about Flux and InfluxQL parity.

Flux vs InfluxQL | InfluxData Documentation

Remark: Please follow this link to get an at-a-glance like introduction to Flux’s features by looking at things which haven’t been possible with InfluxQL.

Rationale for Flux

A Note on IFQL

The combination of schema-on-write, automatic indexing of tagsets , and SQL-like syntax produce a system that allows newcomers to be productive quickly, that feels familiar, and requires minimal setup to get started.

However, the pre-allocation of narrowly scoped iterators means high-cardinality queries, and queries that produce a very large number of groups are expensive to plan. The iterator structures can consume, worst case, GBs of RAM. Secondly, the iterator allocation during planning and other implementation details make multi-query resource management difficult. Finally, while SQL-like syntax is a good fit for simple queries, it becomes cumbersome for more sophisticated analytics. Time series queries are often sets of functions applied to groupings of filtered streams. Expressing these queries using select-project-join logic with advanced SQL partition and over clauses requires an experienced SQL programmer and is no longer beginner- friendly.

We recently announced a prototype query language, IFQL, to explore solutions to these problems: cheaper planning, better resource management, and easier expression of complex queries.

Querying Data Optimally | InfluxDB Internals 101 [Part Two] (Nov, 2017)

Documentation


Articles

2017

2018

The Design of IFQL, the New Influx Functional Query Language

by Paul Dix, the founder of InfluxData and principal architect of Flux (formerly IFQL):

The new InfluxDB query language is functional in nature. Conceptually it looks like D3 or jQuery. Functions are chained off each other. This gives greater flexibility for what the language can do and opens up new possibilities for UI builders. This talk will cover the details of the language and user interface ideas that can pair up nicely with it. We’ll look at query examples that perform math on metrics, interpolation, and similarity measures.

Important notes about Flux

Key concepts » Buckets

Flux introduces “buckets”, a new data storage concept for InfluxDB. A bucket is a named location where data is stored that has a retention policy. It’s similar to an InfluxDB v1.x “database,” but is a combination of both a database and a retention policy.

Get started with Flux » Key concepts » Buckets

Specify a time range

Flux requires a time range when querying time series data. “Unbounded” queries are very resource-intensive and as a protective measure, Flux will not query the database without a specified range.

Query InfluxDB with Flux » Specify a time range


Using Flux from Grafana
Upgrade to InfluxDB 1.7.2 and Grafana 5.4.2
Temperatursummen in der Imkerei
#2

Test drive

Despite it will be arriving soon in mainline as there is a schedule for the InfluxDB Master Branch to Switch to 2.0 on Jan 11 | InfluxData (thanks @weef), we are all eager to check it out.

As the Testdrive of InfluxData OSS 2.0 failed to get us Flux into Grafana through Flux (InfluxDB) [BETA] plugin for Grafana | Grafana Labs, we are now trying to just Upgrade to InfluxDB 1.7.2 and Grafana 5.4.2.


#3

Flux from the commandline

InfluxDB 1.7.2 supports Flux out-of-the-box already, just add "flux-enabled = true" to its configuration file.

See also Different ways to execute Flux queries » Influx CLI in “Flux mode” ff.

Invoke the REPL shell

influx -precision rfc3339 -type flux

Connected to http://localhost:8086 version 1.7.2
InfluxDB shell version: 1.7.2
Enter a Flux query

Execute Flux query

> from(bucket:"hiveeyes_open_hive_clemens") |> range(start: -1h) |> last()

Result

Result: _result
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2019-01-06T20:13:58.532965601Z  2019-01-06T21:13:58.532965601Z     Aussen-Feuchtigkeit       default_2_sensors  2019-01-06T21:06:40.000000000Z                            71
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2019-01-06T20:13:58.532965601Z  2019-01-06T21:13:58.532965601Z       Aussen-Temperatur       default_2_sensors  2019-01-06T21:06:40.000000000Z                           5.9
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2019-01-06T20:13:58.532965601Z  2019-01-06T21:13:58.532965601Z                 Gewicht       default_2_sensors  2019-01-06T21:06:40.000000000Z                         8.188
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2019-01-06T20:13:58.532965601Z  2019-01-06T21:13:58.532965601Z                Spannung       default_2_sensors  2019-01-06T21:06:40.000000000Z                          3.67

#4

Flux via InfluxDB HTTP API

See also Different ways to execute Flux queries » InfluxDB API.

Request

echo 'from(bucket:"hiveeyes_open_hive_clemens") |> range(start: -1h) |> last()' | \
    http localhost:8086/api/v2/query Content-Type:application/vnd.flux

Response

Headers

HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Type: text/csv; charset=utf-8
Date: Sun, 06 Jan 2019 22:48:01 GMT
Request-Id: 1eee2579-1205-11e9-8b65-00163e08ddd3
Transfer-Encoding: chunked
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.2
X-Request-Id: 1eee2579-1205-11e9-8b65-00163e08ddd3

Body

#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string
#group,false,false,true,true,false,false,true,true
#default,_result,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement
,,0,2019-01-06T21:48:01.457170176Z,2019-01-06T22:48:01.457170176Z,2019-01-06T22:46:40Z,73.3,Aussen-Feuchtigkeit,default_2_sensors
,,1,2019-01-06T21:48:01.457170176Z,2019-01-06T22:48:01.457170176Z,2019-01-06T22:46:40Z,5.3,Aussen-Temperatur,default_2_sensors
,,2,2019-01-06T21:48:01.457170176Z,2019-01-06T22:48:01.457170176Z,2019-01-06T22:46:40Z,8.188,Gewicht,default_2_sensors
,,3,2019-01-06T21:48:01.457170176Z,2019-01-06T22:48:01.457170176Z,2019-01-06T22:46:40Z,3.67,Spannung,default_2_sensors

#5

Flux via Grafana

When intending to use Grafana with Flux, we ask you to follow up reading …


#6

Flux by example

Introduction

When diving into Flux, please read the following sections carefully. They will guide you from basic examples to semi advanced ones.

Prepare data

To prepare some data points, just add them like

# Define ingest address for data channel.
target=https://weather.hiveeyes.org/api/umwelt/testdrive/flux/test-01/data

# Submit some measurement points.
echo '{"temperature": 42.42, "humidity": 84.84, "rssi": 50.50}' | http $target
sleep 1.5
echo '{"temperature": 44.42, "humidity": 80.84, "rssi": 55.55}' | http $target
sleep 1.5
echo '{"temperature": 46.42, "humidity": 70.78, "rssi": 60.80}' | http $target
sleep 1.5
echo '{"temperature": 53.42, "humidity": 50.50, "rssi": 57.30}' | http $target
sleep 1.5
echo '{"temperature": 51.39, "humidity": 48.00, "rssi": 53.53}' | http $target
sleep 1.5
echo '{"temperature": 36.36, "humidity": 75.75, "rssi": 64.64}' | http $target

[o] TODO: Add a more sensible example dataset. Does someone have an idea about a 3-5 column, reasonably-sized CSV dataset we could use here?


#7

Basic queries

These examples will show the legacy SQL-like InfluxQL and the new Flux query expressions and their results side by side. Please also have a look at the official documentation about how to Query InfluxDB with Flux | InfluxData Documentation.

As having a time-based filter component as minimum expression constraint is obligatory with Flux …

Flux requires a time range when querying time series data. “Unbounded” queries are very resource-intensive and as a protective measure, Flux will not query the database without a specified range.

Query InfluxDB with Flux | InfluxData Documentation

… we will use a simple filter like "now()-1h" for restricting the time range.

Expressions

InfluxQL

# Define InfluxQL expression.
expression='SELECT * FROM flux_test_01_sensors WHERE time > now()-12h;'

# Submit expression to database.
echo $expression | influx -precision rfc3339 -type influxql -database "umwelt_testdrive"

Flux

# Define Flux expression.
expression='
from(bucket:"umwelt_testdrive")
  |> range(start: -12h)
  |> filter(fn: (r) => r._measurement == "flux_test_01_sensors")
'

# Submit expression to database.
echo $expression | influx -precision rfc3339 -type flux

Results

InfluxQL

Querying by InfluxQL produces columnar data output (one column per field) by default.

name: flux_test_01_sensors

time                           humidity rssi  temperature
----                           -------- ----  -----------
2019-01-08T00:27:20.975467141Z 84.84    50    42.42
2019-01-08T00:27:54.828403444Z 80.84    55    44.42
2019-01-08T00:28:15.657730499Z 70       60.8  46.42
2019-01-08T00:28:33.315969281Z 50       57.3  53.42
2019-01-08T00:28:46.551228316Z 48       53.53 51.39
2019-01-08T00:29:06.491641575Z 75.75    64.64 36.36

Flux

Querying by Flux will yield timeseries-style results. More precisely, all results will get grouped into one timeseries table per field. As our schema has three fields "temperature", "humidity" and "rssi", we will receive three timeseries tables, respectively containing all data points for each field at a time.

Result: _result

Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                humidity    flux_test_01_sensors  2019-01-08T00:27:20.975467141Z                         84.84
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                humidity    flux_test_01_sensors  2019-01-08T00:27:54.828403444Z                         80.84
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                humidity    flux_test_01_sensors  2019-01-08T00:28:15.657730499Z                            70
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                humidity    flux_test_01_sensors  2019-01-08T00:28:33.315969281Z                            50
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                humidity    flux_test_01_sensors  2019-01-08T00:28:46.551228316Z                            48
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                humidity    flux_test_01_sensors  2019-01-08T00:29:06.491641575Z                         75.75

Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                    rssi    flux_test_01_sensors  2019-01-08T00:27:20.975467141Z                            50
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                    rssi    flux_test_01_sensors  2019-01-08T00:27:54.828403444Z                            55
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                    rssi    flux_test_01_sensors  2019-01-08T00:28:15.657730499Z                          60.8
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                    rssi    flux_test_01_sensors  2019-01-08T00:28:33.315969281Z                          57.3
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                    rssi    flux_test_01_sensors  2019-01-08T00:28:46.551228316Z                         53.53
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z                    rssi    flux_test_01_sensors  2019-01-08T00:29:06.491641575Z                         64.64

Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z             temperature    flux_test_01_sensors  2019-01-08T00:27:20.975467141Z                         42.42
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z             temperature    flux_test_01_sensors  2019-01-08T00:27:54.828403444Z                         44.42
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z             temperature    flux_test_01_sensors  2019-01-08T00:28:15.657730499Z                         46.42
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z             temperature    flux_test_01_sensors  2019-01-08T00:28:33.315969281Z                         53.42
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z             temperature    flux_test_01_sensors  2019-01-08T00:28:46.551228316Z                         51.39
2019-01-07T23:42:33.662834823Z  2019-01-08T00:42:33.662834823Z             temperature    flux_test_01_sensors  2019-01-08T00:29:06.491641575Z                         36.36

#8

Now, let’s switch to the HTTP interface like Querying our InfluxDB databases interactively. In order to do so, you will have to authenticate appropriately:

http --session=hiveeyes https://weather.hiveeyes.org/influx/ --auth <username>:<password>

#9

Projection

How to use Projection (relational algebra) - Wikipedia with InfluxDB/Flux. You can either

Keep specific columns

See also keep() function | InfluxData Documentation.

# Define Flux expression.
expression='
from(bucket:"umwelt_testdrive")
  |> range(start: -12h)
  |> filter(fn: (r) => 
    r._measurement == "flux_test_01_sensors"
  )
  |> keep(columns: ["_time", "_field", "_value"])
'

# Submit expression to database.
echo $expression | http --session=hiveeyes https://weather.hiveeyes.org/influx/api/v2/query Content-Type:application/vnd.flux

Drop specific columns

See also drop() function | InfluxData Documentation.

# Define Flux expression.
expression='
from(bucket:"umwelt_testdrive")
  |> range(start: -12h)
  |> filter(fn: (r) => 
    r._measurement == "flux_test_01_sensors"
  )
  |> drop(columns: ["_start", "_stop", "_measurement"])
'

# Submit expression to database.
echo $expression | http --session=hiveeyes https://weather.hiveeyes.org/influx/api/v2/query Content-Type:application/vnd.flux

#10

Cardinality

How to compute the Cardinality - Wikipedia of datasets with InfluxDB. In other words, this is about finding out how many records are stored in your database table.

InfluxQL

# Define InfluxQL expression.
expression='
SELECT count(*) 
  FROM flux_test_01_sensors 
  WHERE time > now()-12h
'
# Submit expression to database.
http --session=hiveeyes https://weather.hiveeyes.org/influx/query \
    db==umwelt_testdrive \
    q=="$expression"

Flux

# Define Flux expression.
expression='
from(bucket:"umwelt_testdrive")
  |> range(start: -12h)
  |> filter(fn: (r) =>
    r._measurement == "flux_test_01_sensors"
  )
  |> count() |> keep(columns: ["_value"])
'
# Submit expression to database.
echo $expression | http --session=hiveeyes https://weather.hiveeyes.org/influx/api/v2/query Content-Type:application/vnd.flux

#11

About numeric data types

@wtf told us that Flux is picky about data types. You should take care not mixing up Integers and Floats. See for yourself:

Yielding errors

> 5.0 / 6
Error: type error 1:1-1:8: float != int

> 5 / 6.0
Error: type error 1:1-1:8: int != float

Yielding results

> 5.0 / 6.0
0.8333333333333334
> 5 / 6
0

Take care about that one. When doing arithmetic operations on two Integers, the result stays in the Integer domain, actually behaving like Python:

>>> 5 / 6
0

#12

Keywords are case sensitive in Flux

https://github.com/influxdata/flux/pull/351/files

Acceptable with InfluxDB 1.7.2 / Flux 0.7.1

|> filter(fn: (r) => r._measurement == "mem" AND r._field == "used" )

Strict with InfluxDB 1.7.3 / Flux 0.12.0

|> filter(fn: (r) => r._measurement == "mem" and r._field == "used" )

P.S.: We encountered this when trying to upgrade to InfluxDB 1.7.3.