Unlocking and improving the Pythings SIM800 GPRS module for MicroPython

We will use that module to implement appropriate telemetry support in order to submit measurement data from the Terkin datalogger to the Kotori data historian using HTTP POST requests as outlined within HTTP - Kotori DAQ.

This is just a first attempt and has not been tested yet. @clemens or @poesel might want to give this a try on one of their TTGO T-Call-SIM800x devices. Thanks again, @sarusso!

3 Likes

A post was merged into an existing topic: Portierung des Terkin-Datenlogger auf ESP32 WROVER mit MicroPython 1.11

Success!
I flashed a T-Call with the uPy 1.12 idf3 binary (idf4 has no PPP and I think we need that here).
Then I synced your github unto the device. I commented the debugging messages back in but apart from that no changes. The pins are already correct.

example_usage()
Starting up...
Modem info: "SIM800 R14.18"
Network scan: "[{'name': 'vodafone', 'shortname': 'voda D2', 'id': '26202'}, {'name': 'D1', 'shortname': 'TMO D', 'id': '26201'}, {'name': 'E-Plus', 'shortname': 'E-Plus', 'id': '26203'}]"
Current network: "E-Plus"
Signal strength: "60.0%"

Modem IP address: "10.179.170.111"

Now running demo http GET...
Response status code: 200
Response content: <html><head><title>Current IP Check</title></head><body>Current IP Address: 5.35.166.143</body></html>

Now running demo https POST...
Response status code: 200
Response content: {"args":{},"data":{"myparameter":42},"files":{},"form":{},"headers":{"x-forwarded-proto":"https","host":"postman-echo.com","content-length":"19","accept":"*/*","content-type":"application/json","user-agent":"SIMCOM_MODULE","x-forwarded-port":"443"},"json":{"myparameter":42},"url":"https://postman-echo.com/post"}

First run was a failure because signal strength was 0%. But it worked on the second.

I have a hologram card and this is the first time it worked! :slight_smile:

Grazie @sarusso!

3 Likes

Amazing! Super-happy to read this! :smiley:

p.s. I don’t think you need PPP as the communication is entirely serial with the SIM800L, so in theory idf4 should work as well, even if I am not sure what the benefits it would bring.

Many thanks @Andreas! I’ve started to get my hands dirty on that within Entwicklung der Terkin-Sandbox - #20 by clemens ff.

After some fiddling within Portierung des Terkin-Datenlogger auf Genuine MicroPython für ESP32 ff., I am now getting this:

   39.8760 [terkin.telemetry            ] INFO   : Sending HTTP request over GPRS APN internet.telekom to https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01/data
   40.1040 [terkin.telemetry            ] INFO   : Payload:     {"system.memfree": 3818592, "system.runtime": 20, "system.uptime": 43.213, "system.time": 635552948}
   40.8850 [terkin.telemetry            ] ERROR  : Telemetry to https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01 failed
Traceback (most recent call last):
  File "/lib/terkin/telemetry.py", line 148, in transmit
  File "/lib/terkin/telemetry.py", line 326, in transmit
  File "/lib/terkin/telemetry.py", line 417, in send
  File "/lib/terkin/network/gprs.py", line 72, in send_request
  File "/dist-packages/pythings_sim800.py", line 357, in http_request
  File "/dist-packages/pythings_sim800.py", line 173, in execute_at_command
GenericATError: Got generic AT error

   41.1580 [terkin.datalogger           ] WARNING: Telemetry status: FAILURE. 1 out of 1 targets failed. Status: {'https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-005/t-call-cg-01': False}
   41.3880 [terkin.device               ] INFO   : Start curating the garbage collector
   41.8120 [terkin.device               ] INFO   : Collecting garbage
   42.2730 [terkin.device               ] INFO   : Curating the garbage collector finished. Free memory: 3900608
   42.9730 [terkin.device               ] INFO   : Waiting for 54.705 seconds

It looks like SSL fails.

Even when switching to non-SSL:

  File "/dist-packages/pythings_sim800.py", line 360, in http_request
  File "/dist-packages/pythings_sim800.py", line 173, in execute_at_command
GenericATError: Got generic AT error

Mit diesen Auskommentierungen in pythings_sim800.py

        # ..Do we have to enable ssl as well?
        if url.startswith('https://'):
            logger.debug('Http request step #1.3 (enablessl)')
            #self.execute_at_command('enablessl')
        elif url.startswith('http://'):
            logger.debug('Http request step #1.3 (disablessl)')
            #self.execute_at_command('disablessl')
        else:
            raise Exception('I have no idea how to handle communication protocol for URL "{}"'.format(url))

Läuft der sketch durch! Die Übeltäter sind

'enablessl':  {'string':'AT+HTTPSSL=1', 'timeout':3, 'end': 'OK'},
'disablessl': {'string':'AT+HTTPSSL=0', 'timeout':3, 'end': 'OK'},
1 Like
$ mosquitto_sub -h swarm.hiveeyes.org -p 1883 -t '#' -v

hiveeyes/testdrive/area-005/t-call-cg-01/data.json {"system.memfree": 3894368.0, "system.time": 178.0, "system.uptime": 178.537, "system.runtime": 155.0}

Yeah!

2 Likes

It started working after commenting out lines 357 and 360 within SIM800L.py.

   54.0850 [terkin.telemetry            ] INFO   : Payload:     {"system.memfree": 3807520, "system.runtime": 34, "system.uptime": 57.159, "system.time": 57}
   56.9340 [terkin.network.gprs         ] INFO   : [GPRS] HTTP response status code: 200
   57.1580 [terkin.network.gprs         ] INFO   : [GPRS] HTTP response content: [
    {
        "message": "Received #1 readings",
        "type": "info"
    }
]
   57.3910 [terkin.datalogger           ] INFO   : Telemetry status: SUCCESS (1/1)

:-)) Danke @Andreas!!

2 Likes

While it looks like this specific modem firmware does not support SSL at all, I can’t really believe it. However, we might want to investigate by invoking ATI or AT+CGMR.

Datasheet page 35

  • ATI: Display Product Identification Information

Datasheet page 68

  • AT+CGMR: Request TA Revision Identification of Software Release

Bin gerade schnell zur Arduino-IDE gewechselt und habe mit der TinyGSM über

modem.sendAT("+CGMR");

das ausgegeben bekommen:

SIM800L16_20141031_1027

Nur der Vollständigkeit halber, muss wohl ein anderes Problem gewesen sein:

I’ve shared our observations with @sarusso and LilyGO.

@clemens, did you also try the example_SIM800L.py which worked for @poesel (it formerly was a fn call example_usage()) ? What does this script say on your T-call ?

Your connection to https://swarm.hiveyes.org failed, while https://postman-echo.com/post succeeded on poesel’s tests, which should be investigated first: you both have a TTGO Tcall from the same production run - therefore it’s not very likely that a different firmware in the modem is the reason.

Do we know that? We bought the devices seperately.

That came from the line:

#print(‘Modem info: “{}”’.format(modem.get_info()))

in example_SIM800.py. If there is anything else I can take a look at the board this evening.

Unfortunately I got the old and limp salat:

Modem: SIM800 R13.08

Think I bought the T-Call last summer in the time it was announced here at hiveeyes while @poesel ordered it in the last weeks.

Hmm, is this a hardware issue or can it be updated via firmware?

@poesel you may have a look at the “firmware version” or whatever it is via AT+CGMR I did it with the TinyGSM lib but it should be possible via the pythings code, you may supplement it here:

Volodymyr Shymanskyy the author of TinyGSM writes at
https://github.com/vshymanskyy/TinyGSM/issues/29#issuecomment-328802556

On my SIM800 R14.18 it works fine.
The other one, SIM800 R13.08 reports that SSL is not supported.

So we should make steps in this direction. Btw. is there a date in SIM800L16_20141031_1027 and has my 2019 device a SIM800 module from 2014-10-31 or does it only look like a date by chance?

1 Like

@clemens, @poesel: OK, I somehow thought you were buying this within a same order, apparently no: even very different fw versions.


But this is one more reason to switch over to PPP when running on an ESP32 and no longer depend on this old IP stack on the SIM800.

…but sadly only without PPP.

well, this is the only way to overcome the limitations of the inbuilt IP stack of the SIM800L in terms of reasonable transport layer security: this stack only “features” SSL2, SSL3 and TLS 1.0, and the firmware is about six years old. The three mentioned cryptographic protocols are considered deprecated; SSL2 and 3 are dead, TLS 1.0 ist dead by the BEAST attack also since at least six years, within this spring 2020 all major browser will switch off their support for TLS 1.0 (and even TLS 1.1).

Even if one gets a SIM800 fw updated, it then will only be able to run outdated ‘secure’ protocols and ciphers. So there no other chance remains to update to decent security except via PPP usage and having the ESP32 IP stack doing the work. This way, it even doesn’t matter what particular fw a respective SIM800 has, as its ‘secure’ features are simple not used (it simply of course needs PPP support in its fw).


Here is how loboris does this (w/ a SIM800L ;) ):

3 Likes

@weef sorry I must have mis-explained myself: I meant that the idf4 which is without PPP should work with this AT-based driver for the SIM800, but I was not sure about the benefits of using firmware idf4 over idf3 in general.

I fully agree on the benefits of using PPP (and therefore idf3) instead of relying on SIM800 internal SSL stack via AT commands, also because the SIM800 internal SSL stack does not perform certificate validation, which is something that should be possible to do at least in a not too far future. Agreed also on the protocol version concerns.

The main problem with what loboris does is that as far as I understood it is not a “pluggable driver” but requires an an entire firmware build + patches. Also, its Micropython fork [1] has diverged from the main repo [2], making it not much usable IMHO.

Having an AT-based, almost hardware-agnostic implementation allows for excellent porting across different boards and microcontrollers, this is why I followed this way even given the limitations.

I guess it depends on the use case, for sensitive data the SIM800 internal SSL implementation it is definitely not OK, for sending a bunch of metrics on a cloud service it might be a reasonable compromise.

This said, it would be great to have the Esp32 + SIM800 work out of the box with a vanilla MicroPython and proper SSL stack :slight_smile:

[1] GitHub - loboris/MicroPython_ESP32_psRAM_LoBo: MicroPython for ESP32 with psRAM support
[2] Integrating Loboris port back to official uP ESP32 port - MicroPython Forum

3 Likes