How can I use my Raspberry Pi 4 for a write intensive application?

Question

Context

I have just found out the hard way that SD cards have a real life attainable write limit.

I use my Rasbperry Pi 4 (amongst other things) to scrape a website 24/7 with a frequency of 1 minute, which results in a ~1 MB file written every minute. I've done this for approximately 1.5 years, and now I notice the following issues:

• the card always boots read-only
• fsck tries to fix it and says it succeeded
• card appears to be mounted read-write
• after a reboot all information is gone and nothing is written to the card
• Running fsck from an external system on the card shows "unable to set superblock flags"

From the internet I learned these are all signs of a hardware issue, and that the SD card prevents writes after it has reached a certain number of write cycles. The card I used is a 32GB SanDisk Extreme Pro.

Question

I want to keep using my Raspberry Pi for the same, write intensive, purpose. What can I do, in terms of storage selection or configuration, to extend the writeable life of my Raspberry storage significantly?

What I've looked into

• I found this question: How can I extend the life of my SD card?. However, it focusses on how you can reduce writes to extend the life of the storage medium. I have no intention of reducing writes (as that is the core of the application).
• I found this post about moving the filesystem to an external USB SSD. It's unclear to me if this would change anything as:
  1. External SSD's are also flash memory (right?) and might have the same write limitations.
  2. It's not clear to me if the scraped files would still "go through" the SD card when going to the USB mounted storage.

Answer 1

SSDs have wear levelling technology and quote their expected life usually having at least 60Tb of writes before failure. If my calculations are correct and using your 1Mb per minute estimate that should give you around 120 years before failure.

Answer 2

There are more urban myths about the Pi and SD Cards than fact.

It is true that they can and do wear out (I have 2 failures in 9 years) but the risk is much over-rated.

If a card does become read-only, this is often a sign that it has worn out. In this rare case it is unrepairable, and the SD Card firmware does this to protect existing data.

Many of the suggestions about extending SD Card life are futile.

There are a few things you can do:-

1. Use quality SD Card (I currently use SanDisk® High Endurance microSD Card.
2. Use larger cards; these spread the load and require less frequent block erase.
3. Backup your system regularly, preferably using a file based backup. Backup image of SD Card
4. Rotate between backup and cards freshly imaged from backup. (This keeps files defragmented and minimises the need for blocks to be re-written.)

Without knowledge of what you are doing it is difficult to be specific, but there are means of internally buffering data and only writing actual changes (but this is not Pi specific).

You could use external storage; a NAS or disk mounted on the Pi.

NOTE moving the OS to external storage (at least on the Pi4) is likely to reduce performance, but putting your data on an external device is OK for a low demand situation ~1 MB every minute is OK.