Unlocking and improving the Pythings SIM800 GPRS module for MicroPython

Andreas · February 18, 2020, 5:59pm

Introduction

Within MicroPython libraries for the SIM800 module, we have been researching appropriate pure-Python drivers suitable to unlock SIM800-support for Terkin.

@sarusso just introduced us to the SIM800L.py module he conceived the other day.

Thanks a bunch!

Andreas · February 18, 2020, 6:12pm

We will use that module to implement appropriate telemetry support in order to submit measurement data from the Terkin datalogger to the Kotori data historian using HTTP POST requests as outlined within HTTP - Kotori DAQ.

github.com/hiveeyes/terkin-datalogger

Add telemetry support for cellular networks (GPRS)

opened 06:30PM - 18 Feb 20 UTC

amotl

enhancement documentation

## TinyGSM integration The coolest thing would be to support quite a bunch of G…SM/GPRS modules by bringing the excellent [TinyGSM](https://github.com/vshymanskyy/TinyGSM) library to the world of MicroPython. We mentioned that idea within https://github.com/stinos/micropython-wrap/issues/3 and https://github.com/vshymanskyy/TinyGSM/issues/362 the other day. ## Pure-Python SIM800 support On the other hand, using a specialized pure-Python module for supporting the SIM800 module at least is probably one of the more lower hanging fruits right now. We researched this topic already [1] and now found the implementation by @sarusso to be the most promising to integrate - thanks a bunch! We will further discuss that topic on our forum [2]. [1] https://community.hiveeyes.org/t/micropython-libs-for-sim800-on-esp32/1492 [2] https://community.hiveeyes.org/t/unlocking-and-improving-the-pythings-sim800-gprs-module-for-micropython/2978 --- cc @poesel, @thiasB, @ClemensGruber

Andreas · February 19, 2020, 3:31am

This is just a first attempt and has not been tested yet. @clemens or @poesel might want to give this a try on one of their TTGO T-Call-SIM800x devices. Thanks again, @sarusso!

Andreas · February 19, 2020, 5:29pm

A post was merged into an existing topic: Portierung des Terkin-Datenlogger auf ESP32 WROVER mit MicroPython 1.11

poesel · February 19, 2020, 8:22pm

Success!
I flashed a T-Call with the uPy 1.12 idf3 binary (idf4 has no PPP and I think we need that here).
Then I synced your github unto the device. I commented the debugging messages back in but apart from that no changes. The pins are already correct.

example_usage()
Starting up...
Modem info: "SIM800 R14.18"
Network scan: "[{'name': 'vodafone', 'shortname': 'voda D2', 'id': '26202'}, {'name': 'D1', 'shortname': 'TMO D', 'id': '26201'}, {'name': 'E-Plus', 'shortname': 'E-Plus', 'id': '26203'}]"
Current network: "E-Plus"
Signal strength: "60.0%"

Modem IP address: "10.179.170.111"

Now running demo http GET...
Response status code: 200
Response content: <html><head><title>Current IP Check</title></head><body>Current IP Address: 5.35.166.143</body></html>

Now running demo https POST...
Response status code: 200
Response content: {"args":{},"data":{"myparameter":42},"files":{},"form":{},"headers":{"x-forwarded-proto":"https","host":"postman-echo.com","content-length":"19","accept":"*/*","content-type":"application/json","user-agent":"SIMCOM_MODULE","x-forwarded-port":"443"},"json":{"myparameter":42},"url":"https://postman-echo.com/post"}

First run was a failure because signal strength was 0%. But it worked on the second.

I have a hologram card and this is the first time it worked!

Grazie @sarusso!

sarusso · February 20, 2020, 10:20am

Amazing! Super-happy to read this!

p.s. I don’t think you need PPP as the communication is entirely serial with the SIM800L, so in theory idf4 should work as well, even if I am not sure what the benefits it would bring.

clemens · February 20, 2020, 8:43pm

Many thanks @Andreas! I’ve started to get my hands dirty on that within Entwicklung der Terkin-Sandbox - #20 by clemens ff.

clemens · February 20, 2020, 9:30pm

After some fiddling within Portierung des Terkin-Datenlogger auf Genuine MicroPython für ESP32 ff., I am now getting this:

   39.8760 [terkin.telemetry            ] INFO   : Sending HTTP request over GPRS APN internet.telekom to https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01/data
   40.1040 [terkin.telemetry            ] INFO   : Payload:     {"system.memfree": 3818592, "system.runtime": 20, "system.uptime": 43.213, "system.time": 635552948}
   40.8850 [terkin.telemetry            ] ERROR  : Telemetry to https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01 failed
Traceback (most recent call last):
  File "/lib/terkin/telemetry.py", line 148, in transmit
  File "/lib/terkin/telemetry.py", line 326, in transmit
  File "/lib/terkin/telemetry.py", line 417, in send
  File "/lib/terkin/network/gprs.py", line 72, in send_request
  File "/dist-packages/pythings_sim800.py", line 357, in http_request
  File "/dist-packages/pythings_sim800.py", line 173, in execute_at_command
GenericATError: Got generic AT error

   41.1580 [terkin.datalogger           ] WARNING: Telemetry status: FAILURE. 1 out of 1 targets failed. Status: {'https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01': False}
   41.3880 [terkin.device               ] INFO   : Start curating the garbage collector
   41.8120 [terkin.device               ] INFO   : Collecting garbage
   42.2730 [terkin.device               ] INFO   : Curating the garbage collector finished. Free memory: 3900608
   42.9730 [terkin.device               ] INFO   : Waiting for 54.705 seconds

Andreas · February 20, 2020, 9:39pm

It looks like SSL fails.

github.com

hiveeyes/pythings-sim800/blob/7ee6256e/SIM800L.py#L354-L362


      
          # ..Do we have to enable ssl as well?
          if url.startswith('https://'):
              logger.debug('Http request step #1.3 (enablessl)')
              self.execute_at_command('enablessl')
          elif url.startswith('http://'):
              logger.debug('Http request step #1.3 (disablessl)')
              self.execute_at_command('disablessl')
          else:
              raise Exception('I have no idea how to handle communication protocol for URL "{}"'.format(url))

clemens · February 20, 2020, 9:41pm

Even when switching to non-SSL:

  File "/dist-packages/pythings_sim800.py", line 360, in http_request
  File "/dist-packages/pythings_sim800.py", line 173, in execute_at_command
GenericATError: Got generic AT error

clemens · February 20, 2020, 10:30pm

Mit diesen Auskommentierungen in pythings_sim800.py

        # ..Do we have to enable ssl as well?
        if url.startswith('https://'):
            logger.debug('Http request step #1.3 (enablessl)')
            #self.execute_at_command('enablessl')
        elif url.startswith('http://'):
            logger.debug('Http request step #1.3 (disablessl)')
            #self.execute_at_command('disablessl')
        else:
            raise Exception('I have no idea how to handle communication protocol for URL "{}"'.format(url))

Läuft der sketch durch! Die Übeltäter sind

'enablessl':  {'string':'AT+HTTPSSL=1', 'timeout':3, 'end': 'OK'},
'disablessl': {'string':'AT+HTTPSSL=0', 'timeout':3, 'end': 'OK'},

Andreas · February 20, 2020, 10:34pm

$ mosquitto_sub -h swarm.hiveeyes.org -p 1883 -t '#' -v

hiveeyes/testdrive/area-005/t-call-cg-01/data.json {"system.memfree": 3894368.0, "system.time": 178.0, "system.uptime": 178.537, "system.runtime": 155.0}

Yeah!

clemens · February 20, 2020, 10:52pm

It started working after commenting out lines 357 and 360 within SIM800L.py.

   54.0850 [terkin.telemetry            ] INFO   : Payload:     {"system.memfree": 3807520, "system.runtime": 34, "system.uptime": 57.159, "system.time": 57}
   56.9340 [terkin.network.gprs         ] INFO   : [GPRS] HTTP response status code: 200
   57.1580 [terkin.network.gprs         ] INFO   : [GPRS] HTTP response content: [
    {
        "message": "Received #1 readings",
        "type": "info"
    }
]
   57.3910 [terkin.datalogger           ] INFO   : Telemetry status: SUCCESS (1/1)

:-)) Danke @Andreas!!

Andreas · February 20, 2020, 11:01pm

While it looks like this specific modem firmware does not support SSL at all, I can’t really believe it. However, we might want to investigate by invoking ATI or AT+CGMR.

Datasheet page 35

ATI: Display Product Identification Information

Datasheet page 68

AT+CGMR: Request TA Revision Identification of Software Release

clemens · February 20, 2020, 11:30pm

Bin gerade schnell zur Arduino-IDE gewechselt und habe mit der TinyGSM über

modem.sendAT("+CGMR");

das ausgegeben bekommen:

SIM800L16_20141031_1027

Nur der Vollständigkeit halber, muss wohl ein anderes Problem gewesen sein:

Andreas · February 21, 2020, 1:08am

I’ve shared our observations with @sarusso and LilyGO.

weef · February 21, 2020, 5:20am

@clemens, did you also try the example_SIM800L.py which worked for @poesel (it formerly was a fn call example_usage()) ? What does this script say on your T-call ?

Your connection to https://swarm.hiveyes.org failed, while https://postman-echo.com/post succeeded on poesel’s tests, which should be investigated first: you both have a TTGO Tcall from the same production run - therefore it’s not very likely that a different firmware in the modem is the reason.

poesel · February 21, 2020, 8:19am

Do we know that? We bought the devices seperately.

That came from the line:

#print(‘Modem info: “{}”’.format(modem.get_info()))

in example_SIM800.py. If there is anything else I can take a look at the board this evening.

clemens · February 21, 2020, 1:10pm

Unfortunately I got the old and limp salat:

Modem: SIM800 R13.08

Think I bought the T-Call last summer in the time it was announced here at hiveeyes while @poesel ordered it in the last weeks.

Hmm, is this a hardware issue or can it be updated via firmware?

@poesel you may have a look at the “firmware version” or whatever it is via AT+CGMR I did it with the TinyGSM lib but it should be possible via the pythings code, you may supplement it here:

github.com

pythings/Drivers/blob/a69878f92e36eba91b3038f5757208397f3913e5/SIM800L.py#L102-L106


      
          # Commands dictionary. Not the best approach ever, but works nicely.
          commands = {
                      'modeminfo':  {'string':'ATI', 'timeout':3, 'end': 'OK'},
                      'battery':    {'string':'AT+CBC', 'timeout':3, 'end': 'OK'},
                      'scan':       {'string':'AT+COPS=?', 'timeout':60, 'end': 'OK'},

Volodymyr Shymanskyy the author of TinyGSM writes at
https://github.com/vshymanskyy/TinyGSM/issues/29#issuecomment-328802556

On my SIM800 R14.18 it works fine.
The other one, SIM800 R13.08 reports that SSL is not supported.

So we should make steps in this direction. Btw. is there a date in SIM800L16_20141031_1027 and has my 2019 device a SIM800 module from 2014-10-31 or does it only look like a date by chance?

weef · February 21, 2020, 3:31pm

@clemens, @poesel: OK, I somehow thought you were buying this within a same order, apparently no: even very different fw versions.

But this is one more reason to switch over to PPP when running on an ESP32 and no longer depend on this old IP stack on the SIM800.

…but sadly only without PPP.

well, this is the only way to overcome the limitations of the inbuilt IP stack of the SIM800L in terms of reasonable transport layer security: this stack only “features” SSL2, SSL3 and TLS 1.0, and the firmware is about six years old. The three mentioned cryptographic protocols are considered deprecated; SSL2 and 3 are dead, TLS 1.0 ist dead by the BEAST attack also since at least six years, within this spring 2020 all major browser will switch off their support for TLS 1.0 (and even TLS 1.1).

Even if one gets a SIM800 fw updated, it then will only be able to run outdated ‘secure’ protocols and ciphers. So there no other chance remains to update to decent security except via PPP usage and having the ESP32 IP stack doing the work. This way, it even doesn’t matter what particular fw a respective SIM800 has, as its ‘secure’ features are simple not used (it simply of course needs PPP support in its fw).

Here is how loboris does this (w/ a SIM800L ;) ):