Audio acquisition and analysis with ARM Cortex-M0 and I2S microphone

,

Yeah, these are the usual things we also apply as workarounds for getting things going before getting in touch with the real problem. But obviously - and we are probably on the same page with that - this is just a sign of problems which are rooted a bit deeper. Most probably it’s about timing of things. In other words (and without any offense): It just has been a matter of luck that it has worked before and it even might stop working there if timing behavior changes slightly through factors beyond their immediate scope.

The good thing is that we know already what we have to ramble about: Slightly different hardware, so different drivers and in essence different timing behavior of the things involved. Specifically: As I learned from reading your code is that LMIC more or less completely takes over the main loop and introduces multitasking functionality by means of "os_init", "os_runloop_once" and "os_setTimedCallback" and friends. While this completely makes sense to actually drive the LoRa stack, it does not shield you from doing minor but important things wrong on the task scheduling side when interacting with your sensor domain.

I’m confident we will find the culprit, but I have to tackle the learning curve of sublimating myself more deeply into the topic first.

It might also help to have the other firmware which is currently working flawlessly on your hardware inside the hive available to be able to compare them side-by-side.

Hi Andreas, great analysis!

Indeed, as part of my debugging efforts I put in those statements and concluded that this would be something to mention to you, as it could contain a hint of sorts. I noticed other things in the past that might indicate a timing problem. For example, not being able to join the TTN network if I had the microphone running. I assumed because the join accept message returned from the gateway wasn’t received due the fact that the time slot in which the message was expected inadvertently got messed with. This triggered me to I2S.begin() (and end()) within a the function needing I2S and not as part of the setup() routine. However, I lack the understanding of interrrupt and timers to figure this out. Much appreciated that you have a look!!

As a side note: for the same reason (lack of understanding) I wouldn’t even know how to do things like getting the hardware in and out of deepsleep mode.

Enjoy the weekend!

Wouter.

1 Like

Sigh. It’s always like that, #SNAFU.

Let’s keep that for later, ok? ;]


Have a good weekend everyone!

@wjmb, do you know about the GitHub - adafruit/Adafruit_ZeroI2S: I2S audio playback library for the Arduino Zero / Adafruit Feather M0 (SAMD21 processor). already?

Especially the mentioning of DMA / interrupt support sparked my interest. Don’t be afraid of “I2S audio playback library” in the first line of the description, they are later telling us about Both Transmit (audio/speaker output) & Receive (audio/mic input) support.

I don’t want to say that this would be the right way to tackle the problem yet, but it’s definitively worth a look as the authors obviously did a significant amount of yak shaving already which really could help us in our endeavors.

As it seems to be the sister library to GitHub - adafruit/Adafruit_ZeroFFT: simple FFT for cortex m0 (which you are already using, right?), it might not be the worst choice either ;]. Looks like deanm1278 · GitHub is the mastermind behind both pieces.


While I believe to have a reasonable good understanding of the runtime behavior of MCU things regarding interrupts and multitasking, I would never be able to write such code as it requires a deeper understanding of the hardware you are programming there.

We will have to figure out by trial & error and educated guessing, but I’m confident we can make it through, eventually by joining our humble debugging abilities.

Hi Andreas,

Good find! I’ve seen it before, but haven’t payed it a lot of attention because I couldn’t see an input example. I remember it, as I did wonder about the following lines of code

/* max volume for 32 bit data */
#define VOLUME ( (1UL << 31) - 1)

where I couldn’t figure out what 1UL is. This was relevant to me as I couldn’t figure out the way a 32-bit word was translated from the microphone. For the SPH0645LM4H-B microphone, the data starts right a the first bit and is 18 bits wide. People in forums complain about it not following the standard, whereas the ICS43432 does. In the latter the first bit has no meaning as far as I can tell and only then follows the MSB and rest of 24 bit data. Skipping this first bit is what seems to be implied by PHILIPS_MODE. So I have been looking into various I2S libraries to try to figure out if this first bit is already removed from the data that the library returns. And, if yes, if it also shifted the data for the LSB to be the last bit of what is returned, or if the last bit returned is in fact the last bit of the 32 bit word communicated by the microphone and shifting is left for the end user. Does this somehow make sense?

The library seems nice and slim as well (unlike the Arduino - ArduinoSound library) and isn’t a wrapper for what I use already. I will have a better look at it as well.

Best regards,

Wouter.

1 Like

To make it a bit more visual, this is from the ICS43432 datasheet:

And this from the SPH0645LM4H-B datasheet.

1 Like

Totally makes sense. Thanks for sharing these insights.

What is the benefit of the Adafruit lib over the default Arduino lib?

It seems to be conceived for the Zero and M0 in mind and offers things like DMA / interrupt support. As the Arduino lib has to support more devices I suppose, I believe the Adafruit library is more the “pro one”, as all things (hardware and software) from Adafruit in comparison to Arduino generally are, no?

Not sure … the Arduino mentions SAMD21 as their target, same as Adafruit. Both libraries seem to contain compiler directives for both SAMD21 and SAMD51

No, it’s for the M0 only, so the same?!


The same(?) issue reported by others

I have tried to track the failure by inputing some Serial.println on the different classes / functions and the execution seems to freeze after the I2S callback to the AudioInI2SClass::OnReceive.

How long does it usually take for the sketch to stall?

It can vary, but normally it takes less than 10-15secs to stall (without Serial).

Following up from ICS43432 I2S - FFT Analysis · Issue #3 · arduino-libraries/ArduinoSound · GitHub, this reads very interesting:

In case of _dmaTransferInProgress == true, due to a faulty DMA callback, the I2S.read() always returns zeroes since no buffer is available after emptying the allocated _doubleBuffer. This issue takes place normally at high sampling rates (our use case is 44.1kHz) for the microphone and, from our point of view, randomly. We assume though that since the DMA.onTransferComplete is triggered by a DMA software trigger, the i2sd.data() itself shouldn’t be the rootcause, but this is just an assumption out of our lack of expertise about the I2S and DMA interactions in depth.


Have you been there already, @wjmb?

Thanks … I will have a good read there (a superficial one already conjures up a few deja-vu feelings ;-) … especially about the 512 buffer size and only getting 64 good samples out of that. Before I was aware of that I spend many days trying to figure out why the spectrum had soo many high frequencies that folded back as artefacts into the spectral range of interest. This was because essentially I was sampling a block-wave with 64 good samples and 256 (left channel of 512 samples) minus 64 = 192 bogus ones. In the end I verified that the 64 samples were continuous with the 64 obtained from the next buffer read and took for granted that the 512 size doesn’t make sense to me … still, not understanding this is one of the red flags that come back up in my mind while I don’t fully trust the FFT.

1 Like

No, not yet.

Also something worth a look from Fab Lab Bcn fame.

P.S.: OSBH also was kind of born there, at least what we learned from history.

Problem summary

From still looking at this from 10.000 ft, the discovery made by andrewjfox · GitHub sounds perfectly reasonable:

This may not be the issue, but I have found that if the buffer is not emptied (using i2s.read()) before the next cycle then the library can stop sampling, and the only way to get it going again is to restart the i2s (i2s.end → is2.begin). The I2S library uses a double buffer so while one buffer is available to read, the other is being filled by the DMA. If the buffer is not available then the process stops.

If you’re doing lots of processing on the buffered data, before going back to read next buffer, the delay between emptying the buffer might be too long.

https://github.com/arduino/ArduinoCore-samd/issues/294#issuecomment-375116660

Advice

So, assuming this might be the reason, we should either reduce the processing time, either by having a look at the FFT processing routine or by checking back with possible negative influences coming from the LMIC scheduler. If we are sure everything is absolutely correct with this and we just can’t make the FFT more efficient, we should look into the concept of bottom-half processing.

This concept is used by grown-up operating systems to counter

the main problems with interrupt handling when performing longish tasks within a handler. […]
[The solution is to split] the interrupt handler into two halves: The so-called top half is the routine that actually responds to the interrupt—the one you register with request_irq. The bottom half is a routine that is scheduled by the top half to be executed later, at a safer time.

There’s a comprehensive but well-written introduction into the topic at Tasklets and Bottom-Half Processing - Linux Device Drivers, Second Edition [Book] - enjoy reading!

If there are no general objections from your side and if we can’t solve the problem by some other reasonably satisfying workaround, we might check whether we might be able to apply this concept onto primitives provided by ArduinoCore-samd and/or the LMIC layer.

Disclaimer

Please recognize I’m writing this solely on the basis of random guesses after reading this small amount of resources circling around the possible issue with reading from I2S we listed above. Nevertheless, please let me know whether this resonates with you.

Magic … this already shows there is an issue: it consistently stalls after writing the RMS value of the second time to serial. I.e. when I2S.end() is called. I will shuffle around with I2S.begin() and end() to make sure there is no computation done in between. More later.

1 Like

I have used this FFT testing code (without FFT / TTN):