Erschließung von I2S-Support und FFT für MicroPython auf Pycom/ESP32

FFT auf ESP32

Et voilà: ESP-DSP: The official DSP library for the ESP32 [1].


  1. Müsste noch auf MicroPython adaptiert werden… ↩︎

Just wanted to add a relevant discussion on the Pycom Forum here:

@Ron: Either you might know everything already or you might find this an interesting read for some aspects.

1 Like

Just for the sake of completeness and because I don’t know where else to put that, I also want to drop Free small FFT in multiple languages here.

However, I actually wanted to outline that after putting Support modonewire from vanilla MicroPython by amotl · Pull Request #356 · pycom/pycom-micropython-sigfox · GitHub forward, we would be reasonably trained of trying this:

1 Like

It looks like this is almost finished, especially since it is already used by Street Sense - A portable electronic device to measure air and noise pollution.

There is also another small project building upon it:

The world’s best motion-activated blinkenlights for pinewood derby cars.

Eric Poulsen / Project Disco · GitLab

1 Like

Testing I2S support on Pycom devices

Introduction

Pre-flight tests

>>> from machine import I2S
>>> dir(I2S)
['__class__', '__name__', 'readinto', 'write', 'ALL_LEFT', 'ALL_RIGHT', 'B16', 'B24', 'B32', 'LSB', 'MASTER_RX', 'MASTER_TX', 'NUM0', 'NUM1', 'ONLY_LEFT', 'ONLY_RIGHT', 'PHILIPS', 'RIGHT_LEFT', 'deinit', 'init']

In the loop

We haven’t actually tested this for real. However, the little example program didn’t crash the machine, which is at least something. @clemens and anyone interested: You might want to try this at home – thanks already!

Instructions

The ready-made firmware image [1] might make your life easier. The example programs [2] will have to be adjusted to use the Pycom-specific pin definition like bck_pin = Pin('P9') instead of e.g. bck_pin = Pin(32).


  1. FiPy-1.20.1.r1-0.7.0-vanilla-dragonfly-onewire-i2s.tar.gz ↩︎

  2. GitHub - miketeachman/micropython-esp32-i2s-examples: Usage and examples for I2S support on the ESP32 microcontroller ↩︎

Many thanks @Andreas! I have no audio playback hardware for the FiPy here so I will test the

Recording to flash example

instead (this is also the domain we need later ;-).

What pins?

In the recording example we have – beside the Vcc and GND pins – this pins:

bck_pin = Pin(14)
ws_pin = Pin(13)
sdin_pin = Pin(27)

The pin definition in Andreas’ sketch (P9 to P11) is a placeholder only. So let’s try to find out what pins we have to use on the FiPy!

It seems that we do not need special pins for I2S, see:
https://github.com/espressif/esp-idf/tree/93a8603c5/examples/peripherals/i2s

The following table describes the pins we use by default (Note that you can also use other pins for the same purpose).

I did not find any proofed pin definitions for the PyCom devices, so in case it is not working we may have a look at the ESP32 examples and used pins, eg. this and that.

So I will stay on the pins Andreas mentioned but in an other order due to better physically wiring:

bck_pin =  Pin('P10')
ws_pin =   Pin('P11')
sdin_pin = Pin('P9')

Minor changes

First I got this error message:

Traceback (most recent call last):
 File "main.py", line 72, in <module>
OSError: [Errno 19] ENODEV

In case I delete in line 72 the slash / it works better and produces a wave file!

s = open('/mic_recording.wav','wb')

Done 256000 bytes written to flash

Finally I changed the channel

channelformat=I2S.ONLY_LEFT,

First recording

I got this recorded file:

So it sound not perfect yet but there is some recording via an I2S mic to FiPy’s flash!

The original code was for a INMP441 mic, I’m using a ICS43432 so this can be a source of error (besides the pins) also.

Btw. I could download the wav file via Atom / pymkr also but it was corrupted, downloading via FTP leads to a playable file.

1 Like

Thanks for letting me know. Cheers! We might want to tell some people on the Pycom user forum accordingly [1].

When connecting the pins, there actually are design mistakes you would want to avoid. Regarding this, I collected the resources you referenced already and added some additional ones:


  1. ↩︎

I have tested different pin combinations but sound does not get better. So I will try my Adafruit mic I have ordered a long time ago, same was used in the example and test this next (in case I will find it :-).

In the code is a comment

Testing was done with an INMP441 Microphone

but in the readme the Adafruit breakout is mentioned:

I2S Boards Tested

  • Adafruit I2S MEMS Microphone Breakout - SPH0645LM4H

I have ordered some INMP441 mics also, really inexpensive, you can get it for 3-4 EUR at ebay. And in a nice form factor that will fit in a Lockenwickler

According to Street Sense Schematic-CPU Module-V01.pdf, Mike Teachman is in fact using the INMP441 for his Street Sense - A portable electronic device to measure air and noise pollution.

Good luck!

Getting real

Introduction

@clemens, @einsiedlerkrebs and me asked ourselves how processing the audio data would actually be done after ingesting it through the MicroPython I2S module. As Python will be too slow, data will have to be channeled through C-level implementations from the I2S driver to the analysis algorithm, most probably using DMA.

Measuring noise level (dbA)

moddba.c seems to be the most recent thing Mike Teachman is working on within his Street Sense - A portable electronic device to measure air and noise pollution. Its purpose is to calculate dB(A) from a stream of audio samples.

Within this blog post, he is outlining how it works:

Details

Fresh from the trenches, the C-level module has been integrated into the MicroPython program through continuous calculation of dbA, every second · miketeachman/micropython-street-sense@d625d0a · GitHub. Enjoy these snippets which outline how audio data is fed from the I2S source into the analyser sink. This is just concise and beautiful.

This is actually implemented within another module called modi2stools.c.

1 Like

A little bit off-topic regarding our requirements, but it outlines activity on the work of I2S support by Mike Teachman.

Mike Teachman is still working on I2S and improved the code examples just recently, see Comparing adbc7b7308...2549264e08 · miketeachman/micropython-esp32-i2s-examples · GitHub.

1 Like

Is Mike Teachman’s I2S example working with the current pycom firmware? Or do we need a special firmware?

The uasyncio library probably needs support from Genuine MicroPython core, so it might not be available on Pycom MicroPython yet.

This is Arduino C, not MicroPython, but perhaps some parts we can have a look at (in the last third)

Und hier auch noch Beispiel-Code für Arduino-C für einen ESP32, Vorbilder waren die Adafruit-Sketches zu I2S

Seems that Mike Teachman’s read-mono-mic-write-internal-flash.py I’ve used 2019-11 is no longer available, but you can find now an example writing to SD

Make it working on a PyCom device

First I installed Andreas’ FiPy firmware version with I2S support:
FiPy-1.20.1.r1-0.7.0-vanilla-dragonfly-onewire-i2s.tar.gz, use this or a newer version.

The original code uses an other SD card interface, according to SD I changed some lines to fit it for the FiPy.

And I changed the pins for the I2S mic to:

P11 - SCK
P22 - WS
P21 - SD 

This is the modified code

# The MIT License (MIT)
# Copyright (c) 2019 Michael Shi
# Copyright (c) 2020 Mike Teachman
# https://opensource.org/licenses/MIT

# Purpose:
# - read 32-bit audio samples from the left channel of an I2S microphone
# - snip upper 16-bits from each 32-bit microphone sample
# - write 16-bit samples to a SD card file using WAV format
#
# Recorded WAV file is named:
#   "mic_left_channel_16bits.wav"
#
# Hardware tested:
# - INMP441 microphone module
# - MSM261S4030H0 microphone module

import os
from machine import Pin
from machine import SD
from machine import I2S

#======= USER CONFIGURATION =======
RECORD_TIME_IN_SECONDS = 5
SAMPLE_RATE_IN_HZ = 22050
#======= USER CONFIGURATION =======

WAV_SAMPLE_SIZE_IN_BITS = 16
WAV_SAMPLE_SIZE_IN_BYTES = WAV_SAMPLE_SIZE_IN_BITS // 8
MIC_SAMPLE_BUFFER_SIZE_IN_BYTES = 4096
SDCARD_SAMPLE_BUFFER_SIZE_IN_BYTES = MIC_SAMPLE_BUFFER_SIZE_IN_BYTES // 2 # why divide by 2? only using 16-bits of 32-bit samples
NUM_SAMPLE_BYTES_TO_WRITE = RECORD_TIME_IN_SECONDS * SAMPLE_RATE_IN_HZ * WAV_SAMPLE_SIZE_IN_BYTES
NUM_SAMPLES_IN_DMA_BUFFER = 256
NUM_CHANNELS = 1

# snip_16_mono():  snip 16-bit samples from a 32-bit mono sample stream
# assumption: I2S configuration for mono microphone.  e.g. I2S channelformat = ONLY_LEFT or ONLY_RIGHT
# example snip:
#   samples_in[] =  [0x44, 0x55, 0xAB, 0x77, 0x99, 0xBB, 0x11, 0x22]
#   samples_out[] = [0xAB, 0x77, 0x11, 0x22]
#   notes:
#       samples_in[] arranged in little endian format:
#           0x77 is the most significant byte of the 32-bit sample
#           0x44 is the least significant byte of the 32-bit sample
#
# returns:  number of bytes snipped
def snip_16_mono(samples_in, samples_out):
    num_samples = len(samples_in) // 4
    for i in range(num_samples):
        samples_out[2*i] = samples_in[4*i + 2]
        samples_out[2*i + 1] = samples_in[4*i + 3]

    return num_samples * 2

def create_wav_header(sampleRate, bitsPerSample, num_channels, num_samples):
    datasize = num_samples * num_channels * bitsPerSample // 8
    o = bytes("RIFF",'ascii')                                                   # (4byte) Marks file as RIFF
    o += (datasize + 36).to_bytes(4,'little')                                   # (4byte) File size in bytes excluding this and RIFF marker
    o += bytes("WAVE",'ascii')                                                  # (4byte) File type
    o += bytes("fmt ",'ascii')                                                  # (4byte) Format Chunk Marker
    o += (16).to_bytes(4,'little')                                              # (4byte) Length of above format data
    o += (1).to_bytes(2,'little')                                               # (2byte) Format type (1 - PCM)
    o += (num_channels).to_bytes(2,'little')                                    # (2byte)
    o += (sampleRate).to_bytes(4,'little')                                      # (4byte)
    o += (sampleRate * num_channels * bitsPerSample // 8).to_bytes(4,'little')  # (4byte)
    o += (num_channels * bitsPerSample // 8).to_bytes(2,'little')               # (2byte)
    o += (bitsPerSample).to_bytes(2,'little')                                   # (2byte)
    o += bytes("data",'ascii')                                                  # (4byte) Data Chunk Marker
    o += (datasize).to_bytes(4,'little')                                        # (4byte) Data size in bytes
    return o

bck_pin = Pin('P11')
ws_pin = Pin('P22')
sdin_pin = Pin('P21')

audio_in = I2S(
    I2S.NUM0,
    bck=bck_pin, ws=ws_pin, sdin=sdin_pin,
    standard=I2S.PHILIPS,
    mode=I2S.MASTER_RX,
    dataformat=I2S.B32,
    channelformat=I2S.ONLY_LEFT,
    samplerate=SAMPLE_RATE_IN_HZ,
    dmacount=50,
    dmalen=NUM_SAMPLES_IN_DMA_BUFFER
)

# configure SD card
#   slot=2 configures SD card to use the SPI3 controller (VSPI), DMA channel = 2
#   slot=3 configures SD card to use the SPI2 controller (HSPI), DMA channel = 1
sd = SD()
os.mount(sd, "/sd")
wav = open('/sd/mic_left_channel_16bits.wav','wb')

# create header for WAV file and write to SD card
wav_header = create_wav_header(
    SAMPLE_RATE_IN_HZ,
    WAV_SAMPLE_SIZE_IN_BITS,
    NUM_CHANNELS,
    SAMPLE_RATE_IN_HZ * RECORD_TIME_IN_SECONDS
)
num_bytes_written = wav.write(wav_header)

# allocate sample arrays
#   memoryview used to reduce heap allocation in while loop
mic_samples = bytearray(MIC_SAMPLE_BUFFER_SIZE_IN_BYTES)
mic_samples_mv = memoryview(mic_samples)
wav_samples = bytearray(SDCARD_SAMPLE_BUFFER_SIZE_IN_BYTES)
wav_samples_mv = memoryview(wav_samples)

num_sample_bytes_written_to_wav = 0

print('Starting')
# read 32-bit samples from I2S microphone, snip upper 16-bits, write snipped samples to WAV file
while num_sample_bytes_written_to_wav < NUM_SAMPLE_BYTES_TO_WRITE:
    try:
        # try to read a block of samples from the I2S microphone
        # readinto() method returns 0 if no DMA buffer is full
        num_bytes_read_from_mic = audio_in.readinto(mic_samples_mv, timeout=0)
        if num_bytes_read_from_mic > 0:
            # snip upper 16-bits from each 32-bit microphone sample
            num_bytes_snipped = snip_16_mono(mic_samples_mv[:num_bytes_read_from_mic], wav_samples_mv)
            num_bytes_to_write = min(num_bytes_snipped, NUM_SAMPLE_BYTES_TO_WRITE - num_sample_bytes_written_to_wav)
            # write samples to WAV file
            num_bytes_written = wav.write(wav_samples_mv[:num_bytes_to_write])
            num_sample_bytes_written_to_wav += num_bytes_written
    except (KeyboardInterrupt, Exception) as e:
        print('caught exception {} {}'.format(type(e).__name__, e))
        break

wav.close()
os.umount("/sd")
sd.deinit()
audio_in.deinit()
print('Done')
print('%d sample bytes written to WAV file' % num_sample_bytes_written_to_wav)

and this a quite good recording:

I got not always such a comfortable quality and I have to investigate what the reasons are. One is: you have to keep a distance of 20 cm to get a nice recording, at least for speach.

After some more testing the audio quality is really good and stable. Perhaps a wiring problem before. You can also record sound direct in front of the I2S mic!

[edit] @cedric.cfk reported a bad audio quality, sounds similar to my first recordings. I tried to reproduce it by shaking the wires and connectors a bit and it seems to produce this metallic sound:

[edit2] Just FYI, it was actual a connection problem caused by bad cables (jumper wire) on @cedric.cfk’s hardware setup.

2 Likes

There is now an updated test / example code version for the first step – recording to SD card: bee-observer_bee-monitoring-micropython/mic-to-sd_adapted-fipy_2-advanced at master · ClemensGruber/bee-observer_bee-monitoring-micropython · GitHub

  • records x times / files from the I2S mic
  • with an adjustable length (seconds)
  • increment file naming
  • WiFi connection (in boot.py) to download files via FTP and play it on your computer
  • Wifi and recording indicator via RGB-LED

Wiring is

For the FiPy you have to use our home brewed firmware with I2S dirver, the official PyCom firmware has this driver not build in! Use at least version FiPy-1.20.1.r1-0.7.0-vanilla-dragonfly-onewire-i2s.tar.gz or better a newer release, see Dragonfly firmware for Pycom/ESP32

1 Like

After getting a good audio stream out of the INMP441 the next “natural” step would be using this stream for a FFT analysis. Due to performance reasons we should count on this:

The official DSP lib from espressif contains a FFT analysis. But this module is not in our firmware yet.

Can someone guide us a bit more detailed through the process of firmware building and including this module?

[edit] Is ulab, see Treiber- und Modulsammlung für MicroPython an alternative? Performance?

1 Like

Yes :slight_smile:

ulab mpy forum

@pythoncoder would surely like to know if an FFT costs a king’s ransom. No, it doesn’t. In fact, a 1024-point float transform can be gotten in less than 2 ms on the pyboard.

1 Like