Unlocking and improving the Pythings SIM800 GPRS module for MicroPython

Introduction

Within MicroPython libraries for the SIM800 module, we have been researching appropriate pure-Python drivers suitable to unlock SIM800-support for Terkin.

@sarusso just introduced us to the SIM800L.py module he conceived the other day.

Thanks a bunch!

1 Like

We will use that module to implement appropriate telemetry support in order to submit measurement data from the Terkin datalogger to the Kotori data historian using HTTP POST requests as outlined within Data acquisition over HTTP — Kotori 0.24.0 documentation.

This is just a first attempt and has not been tested yet. @clemens or @poesel might want to give this a try on one of their TTGO T-Call devices. Thanks again, @sarusso!

3 Likes

A post was merged into an existing topic: Portierung des Terkin-Datenlogger auf ESP32 WROVER mit MicroPython 1.11

Success!
I flashed a T-Call with the uPy 1.12 idf3 binary (idf4 has no PPP and I think we need that here).
Then I synced your github unto the device. I commented the debugging messages back in but apart from that no changes. The pins are already correct.

example_usage()
Starting up...
Modem info: "SIM800 R14.18"
Network scan: "[{'name': 'vodafone', 'shortname': 'voda D2', 'id': '26202'}, {'name': 'D1', 'shortname': 'TMO D', 'id': '26201'}, {'name': 'E-Plus', 'shortname': 'E-Plus', 'id': '26203'}]"
Current network: "E-Plus"
Signal strength: "60.0%"

Modem IP address: "10.179.170.111"

Now running demo http GET...
Response status code: 200
Response content: <html><head><title>Current IP Check</title></head><body>Current IP Address: 5.35.166.143</body></html>

Now running demo https POST...
Response status code: 200
Response content: {"args":{},"data":{"myparameter":42},"files":{},"form":{},"headers":{"x-forwarded-proto":"https","host":"postman-echo.com","content-length":"19","accept":"*/*","content-type":"application/json","user-agent":"SIMCOM_MODULE","x-forwarded-port":"443"},"json":{"myparameter":42},"url":"https://postman-echo.com/post"}

First run was a failure because signal strength was 0%. But it worked on the second.

I have a hologram card and this is the first time it worked! :slight_smile:

Grazie @sarusso!

3 Likes

Amazing! Super-happy to read this! :smiley:

p.s. I don’t think you need PPP as the communication is entirely serial with the SIM800L, so in theory idf4 should work as well, even if I am not sure what the benefits it would bring.

Many thanks @Andreas! I’ve started to get my hands dirty on that within Entwicklung der Terkin-Sandbox ff.

After some fiddling within Portierung des Terkin-Datenlogger auf Genuine MicroPython für ESP32 ff., I am now getting this:

   39.8760 [terkin.telemetry            ] INFO   : Sending HTTP request over GPRS APN internet.telekom to https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01/data
   40.1040 [terkin.telemetry            ] INFO   : Payload:     {"system.memfree": 3818592, "system.runtime": 20, "system.uptime": 43.213, "system.time": 635552948}
   40.8850 [terkin.telemetry            ] ERROR  : Telemetry to https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01 failed
Traceback (most recent call last):
  File "/lib/terkin/telemetry.py", line 148, in transmit
  File "/lib/terkin/telemetry.py", line 326, in transmit
  File "/lib/terkin/telemetry.py", line 417, in send
  File "/lib/terkin/network/gprs.py", line 72, in send_request
  File "/dist-packages/pythings_sim800.py", line 357, in http_request
  File "/dist-packages/pythings_sim800.py", line 173, in execute_at_command
GenericATError: Got generic AT error

   41.1580 [terkin.datalogger           ] WARNING: Telemetry status: FAILURE. 1 out of 1 targets failed. Status: {'https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01': False}
   41.3880 [terkin.device               ] INFO   : Start curating the garbage collector
   41.8120 [terkin.device               ] INFO   : Collecting garbage
   42.2730 [terkin.device               ] INFO   : Curating the garbage collector finished. Free memory: 3900608
   42.9730 [terkin.device               ] INFO   : Waiting for 54.705 seconds

It looks like SSL fails.

Even when switching to non-SSL:

  File "/dist-packages/pythings_sim800.py", line 360, in http_request
  File "/dist-packages/pythings_sim800.py", line 173, in execute_at_command
GenericATError: Got generic AT error

Mit diesen Auskommentierungen in pythings_sim800.py

        # ..Do we have to enable ssl as well?
        if url.startswith('https://'):
            logger.debug('Http request step #1.3 (enablessl)')
            #self.execute_at_command('enablessl')
        elif url.startswith('http://'):
            logger.debug('Http request step #1.3 (disablessl)')
            #self.execute_at_command('disablessl')
        else:
            raise Exception('I have no idea how to handle communication protocol for URL "{}"'.format(url))

Läuft der sketch durch! Die Übeltäter sind

'enablessl':  {'string':'AT+HTTPSSL=1', 'timeout':3, 'end': 'OK'},
'disablessl': {'string':'AT+HTTPSSL=0', 'timeout':3, 'end': 'OK'},
1 Like
$ mosquitto_sub -h swarm.hiveeyes.org -p 1883 -t '#' -v

hiveeyes/testdrive/area-005/t-call-cg-01/data.json {"system.memfree": 3894368.0, "system.time": 178.0, "system.uptime": 178.537, "system.runtime": 155.0}

Yeah!

2 Likes

It started working after commenting out lines 357 and 360 within SIM800L.py.

   54.0850 [terkin.telemetry            ] INFO   : Payload:     {"system.memfree": 3807520, "system.runtime": 34, "system.uptime": 57.159, "system.time": 57}
   56.9340 [terkin.network.gprs         ] INFO   : [GPRS] HTTP response status code: 200
   57.1580 [terkin.network.gprs         ] INFO   : [GPRS] HTTP response content: [
    {
        "message": "Received #1 readings",
        "type": "info"
    }
]
   57.3910 [terkin.datalogger           ] INFO   : Telemetry status: SUCCESS (1/1)

:-)) Danke @Andreas!!

2 Likes

While it looks like this specific modem firmware does not support SSL at all, I can’t really believe it. However, we might want to investigate by invoking ATI or AT+CGMR.

Datasheet page 35

  • ATI: Display Product Identification Information

Datasheet page 68

  • AT+CGMR: Request TA Revision Identification of Software Release

Bin gerade schnell zur Arduino-IDE gewechselt und habe mit der TinyGSM über

modem.sendAT("+CGMR");

das ausgegeben bekommen:

SIM800L16_20141031_1027

Nur der Vollständigkeit halber, muss wohl ein anderes Problem gewesen sein:

I’ve shared our observations with @sarusso and LilyGO.

@clemens, did you also try the example_SIM800L.py which worked for @poesel (it formerly was a fn call example_usage()) ? What does this script say on your T-call ?

Your connection to https://swarm.hiveyes.org failed, while https://postman-echo.com/post succeeded on poesel’s tests, which should be investigated first: you both have a TTGO Tcall from the same production run - therefore it’s not very likely that a different firmware in the modem is the reason.

Do we know that? We bought the devices seperately.

That came from the line:

#print(‘Modem info: “{}”’.format(modem.get_info()))

in example_SIM800.py. If there is anything else I can take a look at the board this evening.

Unfortunately I got the old and limp salat:

Modem: SIM800 R13.08

Think I bought the T-Call last summer in the time it was announced here at hiveeyes while @poesel ordered it in the last weeks.

Hmm, is this a hardware issue or can it be updated via firmware?

@poesel you may have a look at the “firmware version” or whatever it is via AT+CGMR I did it with the TinyGSM lib but it should be possible via the pythings code, you may supplement it here:

Volodymyr Shymanskyy the author of TinyGSM writes at
https://github.com/vshymanskyy/TinyGSM/issues/29#issuecomment-328802556

On my SIM800 R14.18 it works fine.
The other one, SIM800 R13.08 reports that SSL is not supported.

So we should make steps in this direction. Btw. is there a date in SIM800L16_20141031_1027 and has my 2019 device a SIM800 module from 2014-10-31 or does it only look like a date by chance?

1 Like

@clemens, @poesel: OK, I somehow thought you were buying this within a same order, apparently no: even very different fw versions.


But this is one more reason to switch over to PPP when running on an ESP32 and no longer depend on this old IP stack on the SIM800.

…but sadly only without PPP.

well, this is the only way to overcome the limitations of the inbuilt IP stack of the SIM800L in terms of reasonable transport layer security: this stack only “features” SSL2, SSL3 and TLS 1.0, and the firmware is about six years old. The three mentioned cryptographic protocols are considered deprecated; SSL2 and 3 are dead, TLS 1.0 ist dead by the BEAST attack also since at least six years, within this spring 2020 all major browser will switch off their support for TLS 1.0 (and even TLS 1.1).

Even if one gets a SIM800 fw updated, it then will only be able to run outdated ‘secure’ protocols and ciphers. So there no other chance remains to update to decent security except via PPP usage and having the ESP32 IP stack doing the work. This way, it even doesn’t matter what particular fw a respective SIM800 has, as its ‘secure’ features are simple not used (it simply of course needs PPP support in its fw).


Here is how loboris does this (w/ a SIM800L ;) ):

2 Likes