Erschließung von I2S-Support und FFT für MicroPython auf Pycom/ESP32

I have tested different pin combinations but sound does not get better. So I will try my Adafruit mic I have ordered a long time ago, same was used in the example and test this next (in case I will find it :-).

In the code is a comment

Testing was done with an INMP441 Microphone

but in the readme the Adafruit breakout is mentioned:

I2S Boards Tested

  • Adafruit I2S MEMS Microphone Breakout - SPH0645LM4H

I have ordered some INMP441 mics also, really inexpensive, you can get it for 3-4 EUR at ebay. And in a nice form factor that will fit in a Lockenwickler

According to Street Sense Schematic-CPU Module-V01.pdf, Mike Teachman is in fact using the INMP441 for his Street Sense - A portable electronic device to measure air and noise pollution.

Good luck!

Getting real

Introduction

@clemens, @einsiedlerkrebs and me asked ourselves how processing the audio data would actually be done after ingesting it through the MicroPython I2S module. As Python will be too slow, data will have to be channeled through C-level implementations from the I2S driver to the analysis algorithm, most probably using DMA.

Measuring noise level (dbA)

moddba.c seems to be the most recent thing Mike Teachman is working on within his Street Sense - A portable electronic device to measure air and noise pollution. Its purpose is to calculate dB(A) from a stream of audio samples.

Within this blog post, he is outlining how it works:

Details

Fresh from the trenches, the C-level module has been integrated into the MicroPython program through continuous calculation of dbA, every second · miketeachman/micropython-street-sense@d625d0a · GitHub. Enjoy these snippets which outline how audio data is fed from the I2S source into the analyser sink. This is just concise and beautiful.

This is actually implemented within another module called modi2stools.c.

1 Like

A little bit off-topic regarding our requirements, but it outlines activity on the work of I2S support by Mike Teachman.

Mike Teachman is still working on I2S and improved the code examples just recently, see Comparing adbc7b7308...2549264e08 · miketeachman/micropython-esp32-i2s-examples · GitHub.

1 Like

Is Mike Teachman’s I2S example working with the current pycom firmware? Or do we need a special firmware?

The uasyncio library probably needs support from Genuine MicroPython core, so it might not be available on Pycom MicroPython yet.

This is Arduino C, not MicroPython, but perhaps some parts we can have a look at (in the last third)

Und hier auch noch Beispiel-Code für Arduino-C für einen ESP32, Vorbilder waren die Adafruit-Sketches zu I2S

Seems that Mike Teachman’s read-mono-mic-write-internal-flash.py I’ve used 2019-11 is no longer available, but you can find now an example writing to SD

Make it working on a PyCom device

First I installed Andreas’ FiPy firmware version with I2S support:
FiPy-1.20.1.r1-0.7.0-vanilla-dragonfly-onewire-i2s.tar.gz, use this or a newer version.

The original code uses an other SD card interface, according to SD I changed some lines to fit it for the FiPy.

And I changed the pins for the I2S mic to:

P11 - SCK
P22 - WS
P21 - SD 

This is the modified code

# The MIT License (MIT)
# Copyright (c) 2019 Michael Shi
# Copyright (c) 2020 Mike Teachman
# https://opensource.org/licenses/MIT

# Purpose:
# - read 32-bit audio samples from the left channel of an I2S microphone
# - snip upper 16-bits from each 32-bit microphone sample
# - write 16-bit samples to a SD card file using WAV format
#
# Recorded WAV file is named:
#   "mic_left_channel_16bits.wav"
#
# Hardware tested:
# - INMP441 microphone module
# - MSM261S4030H0 microphone module

import os
from machine import Pin
from machine import SD
from machine import I2S

#======= USER CONFIGURATION =======
RECORD_TIME_IN_SECONDS = 5
SAMPLE_RATE_IN_HZ = 22050
#======= USER CONFIGURATION =======

WAV_SAMPLE_SIZE_IN_BITS = 16
WAV_SAMPLE_SIZE_IN_BYTES = WAV_SAMPLE_SIZE_IN_BITS // 8
MIC_SAMPLE_BUFFER_SIZE_IN_BYTES = 4096
SDCARD_SAMPLE_BUFFER_SIZE_IN_BYTES = MIC_SAMPLE_BUFFER_SIZE_IN_BYTES // 2 # why divide by 2? only using 16-bits of 32-bit samples
NUM_SAMPLE_BYTES_TO_WRITE = RECORD_TIME_IN_SECONDS * SAMPLE_RATE_IN_HZ * WAV_SAMPLE_SIZE_IN_BYTES
NUM_SAMPLES_IN_DMA_BUFFER = 256
NUM_CHANNELS = 1

# snip_16_mono():  snip 16-bit samples from a 32-bit mono sample stream
# assumption: I2S configuration for mono microphone.  e.g. I2S channelformat = ONLY_LEFT or ONLY_RIGHT
# example snip:
#   samples_in[] =  [0x44, 0x55, 0xAB, 0x77, 0x99, 0xBB, 0x11, 0x22]
#   samples_out[] = [0xAB, 0x77, 0x11, 0x22]
#   notes:
#       samples_in[] arranged in little endian format:
#           0x77 is the most significant byte of the 32-bit sample
#           0x44 is the least significant byte of the 32-bit sample
#
# returns:  number of bytes snipped
def snip_16_mono(samples_in, samples_out):
    num_samples = len(samples_in) // 4
    for i in range(num_samples):
        samples_out[2*i] = samples_in[4*i + 2]
        samples_out[2*i + 1] = samples_in[4*i + 3]

    return num_samples * 2

def create_wav_header(sampleRate, bitsPerSample, num_channels, num_samples):
    datasize = num_samples * num_channels * bitsPerSample // 8
    o = bytes("RIFF",'ascii')                                                   # (4byte) Marks file as RIFF
    o += (datasize + 36).to_bytes(4,'little')                                   # (4byte) File size in bytes excluding this and RIFF marker
    o += bytes("WAVE",'ascii')                                                  # (4byte) File type
    o += bytes("fmt ",'ascii')                                                  # (4byte) Format Chunk Marker
    o += (16).to_bytes(4,'little')                                              # (4byte) Length of above format data
    o += (1).to_bytes(2,'little')                                               # (2byte) Format type (1 - PCM)
    o += (num_channels).to_bytes(2,'little')                                    # (2byte)
    o += (sampleRate).to_bytes(4,'little')                                      # (4byte)
    o += (sampleRate * num_channels * bitsPerSample // 8).to_bytes(4,'little')  # (4byte)
    o += (num_channels * bitsPerSample // 8).to_bytes(2,'little')               # (2byte)
    o += (bitsPerSample).to_bytes(2,'little')                                   # (2byte)
    o += bytes("data",'ascii')                                                  # (4byte) Data Chunk Marker
    o += (datasize).to_bytes(4,'little')                                        # (4byte) Data size in bytes
    return o

bck_pin = Pin('P11')
ws_pin = Pin('P22')
sdin_pin = Pin('P21')

audio_in = I2S(
    I2S.NUM0,
    bck=bck_pin, ws=ws_pin, sdin=sdin_pin,
    standard=I2S.PHILIPS,
    mode=I2S.MASTER_RX,
    dataformat=I2S.B32,
    channelformat=I2S.ONLY_LEFT,
    samplerate=SAMPLE_RATE_IN_HZ,
    dmacount=50,
    dmalen=NUM_SAMPLES_IN_DMA_BUFFER
)

# configure SD card
#   slot=2 configures SD card to use the SPI3 controller (VSPI), DMA channel = 2
#   slot=3 configures SD card to use the SPI2 controller (HSPI), DMA channel = 1
sd = SD()
os.mount(sd, "/sd")
wav = open('/sd/mic_left_channel_16bits.wav','wb')

# create header for WAV file and write to SD card
wav_header = create_wav_header(
    SAMPLE_RATE_IN_HZ,
    WAV_SAMPLE_SIZE_IN_BITS,
    NUM_CHANNELS,
    SAMPLE_RATE_IN_HZ * RECORD_TIME_IN_SECONDS
)
num_bytes_written = wav.write(wav_header)

# allocate sample arrays
#   memoryview used to reduce heap allocation in while loop
mic_samples = bytearray(MIC_SAMPLE_BUFFER_SIZE_IN_BYTES)
mic_samples_mv = memoryview(mic_samples)
wav_samples = bytearray(SDCARD_SAMPLE_BUFFER_SIZE_IN_BYTES)
wav_samples_mv = memoryview(wav_samples)

num_sample_bytes_written_to_wav = 0

print('Starting')
# read 32-bit samples from I2S microphone, snip upper 16-bits, write snipped samples to WAV file
while num_sample_bytes_written_to_wav < NUM_SAMPLE_BYTES_TO_WRITE:
    try:
        # try to read a block of samples from the I2S microphone
        # readinto() method returns 0 if no DMA buffer is full
        num_bytes_read_from_mic = audio_in.readinto(mic_samples_mv, timeout=0)
        if num_bytes_read_from_mic > 0:
            # snip upper 16-bits from each 32-bit microphone sample
            num_bytes_snipped = snip_16_mono(mic_samples_mv[:num_bytes_read_from_mic], wav_samples_mv)
            num_bytes_to_write = min(num_bytes_snipped, NUM_SAMPLE_BYTES_TO_WRITE - num_sample_bytes_written_to_wav)
            # write samples to WAV file
            num_bytes_written = wav.write(wav_samples_mv[:num_bytes_to_write])
            num_sample_bytes_written_to_wav += num_bytes_written
    except (KeyboardInterrupt, Exception) as e:
        print('caught exception {} {}'.format(type(e).__name__, e))
        break

wav.close()
os.umount("/sd")
sd.deinit()
audio_in.deinit()
print('Done')
print('%d sample bytes written to WAV file' % num_sample_bytes_written_to_wav)

and this a quite good recording:

I got not always such a comfortable quality and I have to investigate what the reasons are. One is: you have to keep a distance of 20 cm to get a nice recording, at least for speach.

After some more testing the audio quality is really good and stable. Perhaps a wiring problem before. You can also record sound direct in front of the I2S mic!

[edit] @cedric.cfk reported a bad audio quality, sounds similar to my first recordings. I tried to reproduce it by shaking the wires and connectors a bit and it seems to produce this metallic sound:

[edit2] Just FYI, it was actual a connection problem caused by bad cables (jumper wire) on @cedric.cfk’s hardware setup.

2 Likes

There is now an updated test / example code version for the first step – recording to SD card: bee-observer_bee-monitoring-micropython/mic-to-sd_adapted-fipy_2-advanced at master · ClemensGruber/bee-observer_bee-monitoring-micropython · GitHub

  • records x times / files from the I2S mic
  • with an adjustable length (seconds)
  • increment file naming
  • WiFi connection (in boot.py) to download files via FTP and play it on your computer
  • Wifi and recording indicator via RGB-LED

Wiring is

For the FiPy you have to use our home brewed firmware with I2S dirver, the official PyCom firmware has this driver not build in! Use at least version FiPy-1.20.1.r1-0.7.0-vanilla-dragonfly-onewire-i2s.tar.gz or better a newer release, see Dragonfly firmware for Pycom/ESP32

1 Like

After getting a good audio stream out of the INMP441 the next “natural” step would be using this stream for a FFT analysis. Due to performance reasons we should count on this:

The official DSP lib from espressif contains a FFT analysis. But this module is not in our firmware yet.

Can someone guide us a bit more detailed through the process of firmware building and including this module?

[edit] Is ulab, see Treiber- und Modulsammlung für MicroPython - #8 by Andreas an alternative? Performance?

1 Like

Yes :slight_smile:

ulab mpy forum

@pythoncoder would surely like to know if an FFT costs a king’s ransom. No, it doesn’t. In fact, a 1024-point float transform can be gotten in less than 2 ms on the pyboard.

1 Like

We are a bit in a dilemma! We have a PyCom firmware (from @Andreas) with an I2S dirver but without FFT support and wie have a firmware with FFT / ulab but without I2S, and we have the official DSP lib which is not integrated in any PyCom firmware yet.

I have asked rcolistete if he has a firmware version with ulab and I2S Pycom firmwares with single/double precision and ulab module | Pycom user forum I pointed him to the pull request from Andreas, hope this is sufficient for a check of feasibility. I feel not so comfortable with this direction becaus I think he has not integrated all the good and helpfull stuff, Andreas hast build in for stability in the last month.

So my favorite would be to integrate the DSP lib (and / or ulab) in Andreas’ firmware code. But as you may know: Busy by other things.

1 Like

@MKO hat sich das ganze mal angesehen und der Weg könnte sein:

Dragonfly mit ulab pimpen?

Source code zur Dragonfly firmware verwenden (nicht verwirren lassen, der branch heißt squirrel beinhaltet aber den latest and greates dragonfly Stand):

Im Endeffekt brauchen wir, so hoffe ich, nur noch diesen commit:

… und den code zum ulab-Modul hier:

Anleitungen dazu hat @MKO auch gefunden, um daraus wieder eine Firmware zu klöppeln. Wär auf alle Fälle einen Versuch wert.

I2S in die aktuelle firmware mit ulab integrieren?

Noch ein Nachtrag zur Idee das anders rum zu machen, die ulab-PyCom-Firmware mit I2S anzureichern:

ulab hat eine neuere Version als unsere home brewed dragonfly als Grundlage. Hier Pycom · master · Roberto Colistete Junior / MicroPython Firmwares · GitLab schreibt ulab 1.20.2.rc10, Andreas hat als Grundlage 1.20.2.rc6 verwendet. Uns bringt die neueste PyCom firmware aber nichts, wenn die Nachbesserungen von Andreas zu onewire, LTE, core panics … wie hier beschrieben nicht drinnen sind. :-( also bleibt nur der Weg oben.

1 Like

Habe jetzt mal versucht beide Zweige zu verbinden, was auch geklappt haben müsste, so hoffe ich.


Ich versuche jetzt mit WSL2 die firmware zu Compilen. Zum testen erstmal ohne das ulab Modul.

Allerdings habe ich ein paar Probleme.
Ich bin strikt nach der Anleitung der README.md vorgegangen.
musste allerdings trotzdem den IDF Hasch im Makefile anpassen.
Irgendwas stimmt aber anscheinend mit meinem WSL nicht.
auf xtensa-esp32-elf-gcc wird der Zugriff verweigert.


Ich bin leider kein Linux Experte und weiß nicht so recht weiter.
Jemand eine Idee?

1 Like

Hmm, liegt es am aktuellen code oder an der Maschinerie? Du versuchst testweise den den alten Stand von Andreas zu compilieren, ja? Dann dürfte es nicht am neuen code liegen, sondern an der Firmware-Maschine.

Liegt wohl nicht am Code wenn ich in der Bash direkt eingebe

xtensa-esp32-elf-gcc

eingebe passiert das gleiche

-bash: /home/user/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc: Permission denied 

Xtensa kann anscheinend nicht geöffnet werden

das einzige was mir noch einfallen würde ich habe die 32Bit version von Xtensa genommen, da die 64Bit Version wegen Fehlern nicht entpackt werden konnte. Weiß nicht ob Linux da empfindlich ist.

Dir fehlen die Rechte, um den gcc zu starten.
Mach mal folgendes:

cd /home/user/xtensa-esp32-elf
chmod -R a+rwx *

Das ist der Holzhammer, der Dir alle Rechte (lesen, schreiben, ausführen) an allen Dateien unterhalb des Verzeichnisses gibt.
Falls da auch Permission denied kommt, hast Du das vermutlich als root installiert. Dann musst Du root werden (‘su’ + passwort). Dann in demselben Verzeichnis ‘chown -R user *’. Danach gehören alle Dateien dem user ‘user’. Danach mit CTRL-d wieder von root zu user wechseln.

leider war ich etwas ungeduldig und habe mein wsl zurückgesetzt und nochmal komplett neu aufgesetzt.
er kompiliert jetzt auch. bricht dann aber ab. Es fehlt ihn nun anscheinend pyserial

1407640  223256   54756 1685652  19b894 build/WIPY/release/application.elf
Building partitions from lib/partitions_4MB.csv...
Building partitions from lib/partitions_8MB.csv...
IMAGE build/WIPY/release/wipy.bin
Pyserial is not installed for /usr/bin/python. Check the README for installation instructions.
Traceback (most recent call last):
  File "/home/user/pycom-esp-idf/components/esptool_py/esptool/esptool.py", line 37, in <module>
    import serial
ImportError: No module named serial
make: *** [application.mk:681: build/WIPY/release/wipy.bin] Error 1

Leider brauche ich für die instalation von pyserial pip.
Pip läßt sich aber nicht mehr installieren.
image

habe jetzt schon alles mögliche versucht