Best buffering practice for continuous SD card writing

Question

In a simplified example, I have a microcontroller functioning as a data acquisition device - sensor data is received over UART, and then saved to an SD card.

This received data is in the form of a 12 byte struct, received at 1kHz. The microcontroller receiving the data is an esp32, with an SD card connected over spi. Not an uncommon issue, but if a typical SD card write is triggered everytime new data is received, it will take longer than the gap between samples to write the data.

What is typically considered the best practice for this type of system, presuming the data will always be received faster than it can be written. I have seen examples of circular buffers, but presumably these will eventually fill completely and samples will be missed.

In my particular example, I have the bonus second cpu core in the esp32. Therefore, my thinking is to use 2 alternating buffers. Data will be received on core 1, and placed into the active buffer. When the buffer is full, incoming data will be written to the second buffer, and the first will be written to the SD card by core 2.

Is this a sensible solution to the problem? I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

Answer 1

Is this a sensible solution to the problem?

No, unfortunately not. The size of the buffer is limited by the available RAM on your chip. Jumping between multiple buffers or cores will not change that, since it is a hardware limitation.

I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

When you are continuously receiving faster than you can write, you only have 2 options:

• Improve write speed
• Reduce the needed data size

There is no other way, and that's a logical conclusion, not so much a programmical or hardware conclusion. It that situation the amount of "buffered" data will increase indefinitely, so you would need an infinite amount of memory. And that doesn't exist in real world. The same principle would apply to a high end PC with lots of memory, it will just be full after longer time, than with an ESP.

• Improving write speed:

  • On various sites I have read, that you can use the SD card with different SPI frequencies. A higher frequency would mean a higher transaction speed. You can investigate further at this point.
  • Writing a full 512 byte page at once and only closing and opening the file again, if really needed, will improve the overall speed.

• Reduce the needed data size

  • Maybe you can compromise with your data resolution and then pack those values as dense as possible (like packing two 4-bit values into a single byte of transmission).
  • You can research for compression code, so that you can compress your data lossless before transmitting it. Though the result will always depend on the kind of your data. Also mind, that in this case you are trading transmission size for computation time and RAM.

About buffers:

I would say, a normal circular buffer does not bring you much advantages here, since with the SD card you don't want to do bytewise data transmission. The 2 buffer approach, that you mentioned, seems good to me. Though doing handling with different cores does not make much sense to me. The UART receiving itself is done by hardware and some ISRs. You can do all the SD card transmissions in your main code, just making sure, that you regularly copy the received data from the Serial buffer into your buffer.

Answer 2

Buffers will help you if at some point the input stream stops, so the "slower" device (e.g. the SD card) can catchup. If the data will always be received faster than it can be written (note always) then no buffer will help you.

But I assume you meant that there times where you get bursts of input data at 1Khz, then it will stop. So, as long as the average input data rate is lower than the ability for the SD to persist the data, you will be fine.

The SD card library already has a buffer (as @Majenko pointed out), so most likely you won't need to do anything. Seem like something you can test relatively simply.

The 2 core approach sounds somewhat overkill (not to mention some synchronization issues you will have to deal with)

Answer 3

Forgive my non-technical analogy, but this is how my brain works. Hope you can get past the conclusion that "this isn't a technical answer", until you at least see it from this different perspective.

Buffers, input streams, and sampling rates, storage rates all relate strongly to filling up a bath. Taps, water, baths and plug holes? WTF?

Try slightly blocking the plug hole in your bath, and turn both taps on full.

The water that's drained away is the rate of which data is stored to the SD card. That's the storage-rate.

The remaining water which doesn't drain away fast enough causes the bath the fill up. The water is the sampling rate, the bath is the buffer.

If the taps remain on, and water is leaving the taps faster than the water can escape from the partially blocked hole will cause the bath to fill up with water.

So, if the blockage isn't cleared, the bath will eventually overflow.

Either you turn the taps down, use a bigger bath, or increase the drainage rate.

Answer 4

One approach would be to use Circular Buffering. It involves using a fixed-size buffer that wraps around itself, allowing data to be continuously written and overwritten. This approach eliminates the need for costly memory reallocation or shifting operations. As new data is written, older data is overwritten in a circular fashion. But, you will need to choose the appropriate buffer size. A larger buffer size can help handle bursts of data However, larger buffers also consume more memory.

If one buffer isn't enough, then you could use Two buffers instead. While one buffer is being written to the SD card, the other buffer can be used to receive new incoming data. Once the write operation is completed, the roles of the buffers are swapped. This technique can help improve efficiency by minimizing data transfer and write latency, which is the goal you aim at. I've used this type of buffer for audio data acquisition, and it worked well, got rid of the buffer overflow issues.

The other method would be to use DMA, if your microcontroller or system supports it, which I think it does, using DMA for SD card writing can offload the data transfer task from the CPU. DMA allows direct memory access without CPU intervention, enabling faster and more efficient data transfers. DMA can significantly reduce the overhead associated with continuous SD card writing.

For DMA, you might want to check this: https://github.com/RobTillaart/DAC8552/issues/8 and this: https://github.com/espressif/esp-idf/blob/master/examples/peripherals/spi_master/lcd/main/spi_master_example_main.c