Montag, 31. August 2015

TLC Bitrot

Recently I got a new laptop (Dell Ultrabook) powered by a Samsung PM851 SSD, quite similar (OEM) to the Samsung 840. It is equipped with TLC flash, and because of bitrot the SSD becomes slower and slower as data ages - complex ECC is performed by the controller to recover the degraded data. This has been discussed quite extensivly in media, e.g. at Anandtech, and Samsung has tried several FW attempts to mitigate this hardware issue. What hasn't been discussed at all, is that most likely these drives will start to loose data when powered off for longer time spans.

Recently I was curious how cheap SD-cards behave in that regard, as Samsung for sure chose their highest-quality TLC flash for the 840/840 Evo- while most SD cards do not get what is considered  SSD-grade flash. I benchmarked the 32GB microSD card I had used in my Android smartphone for the last 6 months. The card was brand new when inserted and only used to store static data like pictures and music - so the flash cells didn't experience any significant stress, here's the result:

So even for only 6 months old data, there are areas where data have degraded to a level the SD card can only read it at a rate of 400kB/s, although the SD card was never exposed to any kind of serious write workload. This has not been an isolated case, we have several similar cards in-house and they all show the same behaviour - they are a time bomb barely good enough for storing music. The cards showing this kind of behaviour use IMFT (Intel-Micron-Flash-Technology) TLC flash, unfortunately the lithography node is unknown. After re-writing the first half of the card using dd, read-performance was restored.

For comparison the results of a MLC based SD card which was exposed to write-intensive workload for quite some time (linux rootfs) and unused for 12 months:

To sum it up: TLC flash is not worth the trouble, not for SD cards and for sure not for SSDs. However, while MLC-powered SSDs aren't a lot more expensive than their TLC-counterparts, for SD-Cards you have to pay quite a premium, because there MLC is only used when required to reach high advertised write-speeds.

Sonntag, 28. Dezember 2014

Set -XX:-UsePerfData for 24/7 java-apps on your raspberry

The "official" raspberry pi distribution raspbian doesn't use tmpfs for /tmp. Therefore, all applications storing temporary stuff in /tmp cause writes to the rootfs which in most cases means to the SD card.

Usually the few kilobyte every minute are no big deal.
However, if you plan to deploy raspberry pi based systems running java applications 24/7 for years every unnescessary periodically occurring write counts.

So, beside using only high-quality SD cards (with MLC or SLC flash), you might want to run your java apps with -XX:-UsePerfData specified. This flag instructs hotspot to no longer generate temporary files for performance monitoring in /tmp.

Freitag, 12. Dezember 2014

Odroid C1 - finally a worthy Raspberry Pi successor

For quite some time I was unhappy about the state of ARM based linux boards:

There was/is the Raspberry Pi which is cheap and widely supported, however extremly underpowered and its creators are reluctant to move to a more powerful Soc (the soc manufacturer employs the Raspberry Pi's main developer, so no suprise). The Raspberry Pi features a single-core ARM11 CPU running at 700Mhz - the same CPU design which was used for the original IPhone in 2007. Worse, due to its age, it doesn't support the ARMv7 instruction set.

The other choice one had were boards with powerful dual and quad-code socs featuring powerful out-of-order cores running at 1-1,5ghz. The downside: Price. Those boards were in the range of 50-100$. For 41€ I can get a mainboard with a dual-core x86 APU (+20€ für 2GB DDR3 ram), where OpenGL is supported by open-source drivers, so no ARM in this price range.

However, recently Hardkernel has announced the Odroid C1. The Ordoid C1 seems to be a perfect compromise between price and performance. For the same price of a Raspberry Pi you get quad-core design. Granted, you only get Cortex-A5 cores (in-order, single-issue) which perform clock-for-clock more or less like the ARM11 found in the raspberry - however you get 4 of them clocked at twice the frequency. Plus, you get twice the amount of RAM (1GB instead of 512MB).

ODroid C1 (Source: Hardkernel)

Montag, 24. November 2014

Nilfs2 is a great filesystem for slow pen drives and sd-cards

To extend the capacity limited SSD of my Notebook I tend to move a lot of seldom-used data (e.g. the installation-directory of my Steam installation) to compact usb pen drives and sd cards. These cheap flash devices all have a very weak FTL (flash translation layer) in common and are extremly slow for small random writes (like filesystem metadata updates).

Initially I used ext4 for the pen-drive which caused Steam to be nearly unuseable - often the user interface was blocked for 10s of seconds and launching games took minutes.
Out of curiosity I re-formatted the pen-drive with NILFS2 (a log-structured file-system) which writes data only in linear in a log-structured way. Now the FTL is freed from managing randomly written chunks and the drive performs only the linear writes it was designed for. Games launch instantly and Steam doesn't experience any hangs or issues. Beside the higher performance, write amplification is decreased a lot by not forcing read-modify-write cycles when modifying data randomly.

Kudos to the Nilfs2 developers!

Samstag, 1. November 2014

The dangers of gift usb pen-drives

I've seen quite a few (~5) usb pen drives dying the last months, usually after what I would consider as light useage. They did not go read-only, but instead lost all content and dropped dead.
All those pen drives had in common, they were/had:
  • very small (1-4 GB) 
  • free / gifts
  • Alcor Micro controllers

The reason they all have small capacity stems from the fact that junk flash is used, where only a small portion is functional.
The image below shows an open Digikey 1GB pen-drive (I got two of these, both died with < 5GB written). The flash is unlabelled, only marked with what appears crayon.  Using ChipGenius to read the flash configuration, it turns out the chip is actually an 8GB Sandisk part.

The shown part is from digikey, but I've head the same stuff from Freescale and other companies, all were highly unreliable. I've had gift pen-drivers which looked exactly the same and were fine (but didn't have "junk flash"), so I guess some manufacturer has started building that stuff in high volume.

Bottom line: Those pen drives are gutter crap. If your data/time is valueable, don't use them. In don't understand why companies like Digikey or Freescale destroy their reputation with crap like this - better no gifts than stuff like this.

Samstag, 13. September 2014

Taming the Nokia-770's OOM-Killer

The Nokia 770's kernel ( does seem to have some issues with the OOM killer. Even with tons of free swap space availabe, it decides to kill larger processes, as I reported back in 2009 on the llkm:

Kernel devs thought this was a bug, however without the possibility updating to a more recent kernel, this didn't help.

However, I recently came across a message of another 770 user who had issues with debian's locale generation process on the 770, also killed by the OOM killer despite free swap. He found out that setting swappiness to 100 solved the issues:

So, although modifying swappiness still doesn't fix the actual kernel bug, the tendency of the kernel to keep more memory available for the  page-cache with a higher swappiness-value seems to make the issue appear a lot less frequently (at the cost of decreased SD life).

Therefore, if you experience OOM issues on your Nokia-770 despite free swap, try:

echo 100 > /proc/sys/vm/swappiness 

Donnerstag, 11. September 2014

Running debian wheezy ( chroot ) on the Nokia 770

After updating my Nokia 770 to OS2008HE and playing arround with it a bit, I thought about using it as home server (again). However, instead of installing Mer, I decided to go for a fully-fledged Debian and use it in a chroot-environment - as I do with my OpenWRT router. This way I get (security) updates timely, and also packages are typically well tested - while not touching the kernel or the device specific drivers.

Thanks to debootstrap, creating a minimum debian root-fs for ARMel is really easy:
debootstrap --foreign --arch=armel wheezy deb770

However, when trying to chroot on the 770, the' kernel shows its age:
chroot /media/mmc/debian
FATAL: kernel too old
The most recent version of debian still supporting 2.6.16 is debian Lenny, which is unsupported as of mid 2012. The main incompatibility stems from the minimum kernel version debian's glibc is compiled with - for debian wheezy userland it doesn't make much sence to support 2.6.16 when wheezy was shipped with Linux-3.2.

However, thanks to debian's excellent build system (and linux' flexibility in general) it is no big deal to re-build glibc with support for older kernels (if you have an ARM based system with a recent kernel, don't know about cross compilation).
Therefore, I simply chrooted into wheezy on my raspberry Pi, downloaded the source package (eglibc) and modified the following config files:

debian/sysdeps/ -> MIN_KERNEL_SUPPORTED = 2.6.16
debian/ -> replace every kernel version starting with 2.6. with 2.6.16

and followed the building tutorial:

After a few hours, you'll find debian packages of your rebuilt glibc, which can be installed with "dpkg -i". A good idea is to lock those packages for updates, otherwise upgrading your system using apt might break everything again. After you've built a working rootfs once, you can also re-compile and update glibc directly on the N770.

For those of you not eager to spend hours compiling glibc, I made the nokia770 compatible eglibc-packages (linux-2.6.16+, ARMv4te) available here:
However, for installation a system with newer kernel is still required (as dpkg requires glibc too), therefore I'll prepare a patched rootfs soon.