21

I purchased a 64 GB SD card from eBay. It works fine when I burn an Arch Linux ARM image to it and use it to boot up my Raspberry Pi.

However, when I try to create a single ext4 partition on it to use all capacity of the card, errors occur. mkfs.ext4 always finishes happily; however, the partition cannot be mounted, always throwing an error and dmesg shows kernel messages includes Cannot find journal. This has proved to be the case on at least two platforms: Arch Linux ARM and Ubuntu 13.04.

On the other hand, I can create and mount a FAT32 partition without error (a full capacity check has not been done).

I heard that some bad guys can change the SD card interface to report a wrong capacity to the OS (i.e. the card is really only 2 GB but it reports itself as a 64 GB) in order to sell the card at a better price.

I know that tools like badblocks exist for me to check the SD card for bad blocks. Can badblocks detect problems like this? If not, what other solutions exist for me to test the card?

I'd ideally like to know whether I was cheated or not; if the result shows I just received a bad item, I can return to the seller only, rather report to eBay that somebody tried to cheat me.

UPDATE

operations and messages:

~$ sudo mkfs.ext4 /dev/sde1
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
4096000 inodes, 16383996 blocks
819199 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
500 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done   

~$ dmesg | tail
...
[4199.749118]...
~$ sudo mount /dev/sde1 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/sde1,
   missing codepage or helper program, or other error
   In some cases useful info is found in syslog - try
   dmesg | tail  or so

~$ dmesg | tail
...
[ 4199.749118]...
[ 4460.857603] JBD2: no valid journal superblock found
[ 4460.857618] EXT4-fs (sde1): error loading journal

UPDATE

I have run badblocks /dev/sde but it reports no error. That means the remain causes are:

  • The SD car is good but for some reason mke2fs or mount or the kernel have a bug that causes the problem.

  • I was cheated in a way that badblocks that cannot detect the defeat. This is plausible because I think badblocks is just doing some in-place write-read test. However, the cheater can make the access to outbound areas link back to some inbound block. In this case a in-place write-read check is not able to detect the problem.

If there is no application can do the proper test, I think I can try to write a simple C program to test it.

8
  • Have you tried it in a SDXC USB card reader? May 22, 2013 at 4:14
  • Also, do any messages appear in the system log at the same time as the errors? May 22, 2013 at 4:14
  • I tried both with the native Raspberry Pi card reader, and an external card reader for my Ubuntu desktop as well. I have said that dmesg shows kernel messages and I am sure it is appear at the same time as the errors because I did that before and after and compared them. I have not check syslog because I believe dmesg will show the messages. May 22, 2013 at 4:21
  • Does it show any other messages? May 22, 2013 at 4:22
  • The external card reader I used is working for my other cards, includes SDXC cards. However that problematic one has one difference: it is a micro SD card with a SD adapter. May 22, 2013 at 4:24

6 Answers 6

32

If anyone sees this later: Someone wrote an open source tool called "F3" to test capacity of SD cards and other such media. It can be found on the project hompage and in Github.

2
6

The cheating has now been confirmed by the following steps:

  • Generate a random data file.  (4194304 = 4 × 1024 × 1024 = 4 MiB, total size = 40 × 4 MiB = 160 MiB)

    Command:

    dd if=/dev/urandom of=test.orig bs=4194304 count=40
    40+0 records in
    40+0 records out
    167772160 bytes (168 MB) copied, 11.0518 s, 15.2 MB/s
    
  • Copy the data to the SD card.  (2038340 × 4096 = 8153600 KiB = 7962.5 MiB)

    Command:

    sudo dd if=test.orig of=/dev/sde seek=2038399 bs=4096
    40960+0 records in
    40960+0 records out
    167772160 bytes (168 MB) copied, 41.6087 s, 4.0 MB/s
    
  • Read the data back from the SD card.

    Command:

    sudo dd if=/dev/sde of=test.result skip=2038399 bs=4096 count=40960
    40960+0 records in
    40960+0 records out
    167772160 bytes (168 MB) copied, 14.5498 s, 11.5 MB/s
    
  • Show the result

    Command:

    hexdump test.result | less
    ...
    0000ff0 b006 fe69 0823 a635 084a f30a c2db 3f19
    0001000 0000 0000 0000 0000 0000 0000 0000 0000
    *
    1a81000 a8a5 9f9d 6722 7f45 fbde 514c fecd 5145
    
    ...
    

What happened? We observed a gap of zeros. This is an indicator that the random data have not been actually written to the card. But why do the data come back after 1a81000? Obviously the card has an internal cache.

We can also try to investigate the behaviour of the cache.

hexdump test.orig | grep ' 0000 0000 '

provides no result, which means that the generated rubbish does not have such a pattern. However,

hexdump test.result | grep ' 0000 0000 '
0001000 0000 0000 0000 0000 0000 0000 0000 0000
213b000 0000 0000 0000 0000 0000 0000 0000 0000
407b000 0000 0000 0000 0000 0000 0000 0000 0000
601b000 0000 0000 0000 0000 0000 0000 0000 0000

have 4 matches.

So this is why it passes badblocks check.  Further tests can show that the actual capacity is 7962.5 MB, or slightly less than 8 GB.

I conclude that this is very unlikely to be just random hardware failure, but more likely to be a kind of cheating (i.e., fraud).  I would like to know what action I can take to help other victims.

Update 11/05/2019

  • People asked me about how do I figured out the correct seek parameter is 2038399. I did a lot more experience than I have shown in the above. Basically you have to guess in the first place. You have to guess a proper size of data, and you have to guess where the data corruption was. But you can always use bisection method to help.

  • In the comment below I thought I was assumed that the second step above (copy the data to SD card) only copies 1 sector. But I was not make this mistake in my experiement. Instead, the seek was to show that in the "show result" step the offset 1000 is simply happen in the second sector of the data. If the seek is 2038399 sectors, the corruption is at the 2038400th sector.

6
  • (1) Where do the numbers 2038340 and 2038399 come from? (2) Why do you use  bs=4194304 count=40 when reading from /dev/urandom but  bs=4096 count=40960  when writing to and reading from the SD card? (They are mathematically equivalent; 167772160 bytes each.) May 6, 2019 at 18:47
  • 1) I used bisec tecnique to figureout thie offset. As the bisec procedure is too verbose for the answer, I simply put them as an evident without further exaplaination. 2) Yes the calculation is equivlent; but I don't know whether I should match the sector size of the card, which I believe is 4096. May 10, 2019 at 10:29
  • Oh, for point 2) I do use seek, so I only wrote 1 sector to the card, which saves the amount of data transfer. Of cause, when experimenting I used bigger data block, this is why the generated data file is 160MiB. May 10, 2019 at 10:46
  • Your second comment doesn’t make any sense. The second command in your answer — the one that writes to the card — is sudo dd if=test.orig of=/dev/sde seek=2038399 bs=4096. And obviously you’re right; it uses seek. And, yeah, technically, it doesn’t use count. … (Cont’d) May 10, 2019 at 17:49
  • (Cont’d) …  But you say “I only wrote 1 sector to the card, which saves the amount of data transfer.” That’s clearly wrong; without a count specification, dd transfers the entire input (i.e., it transfers up to EOF or an error). So that command transfers the entire content of test.orig, which is 40960 records of 4096 bytes each, for a total of 167772160 bytes (as I said). May 10, 2019 at 17:49
3

First of all, read the F3 answer by @Radtoo. It is the correct way.

I have somehow missed it, and tried my own way:

  1. create 1gb test file: dd if=/dev/urandom bs=1024k count=1024 of=testfile1gb

  2. write copies of that file to sdcard (64 is sdcard size in gb): for i in $(seq 1 64); do dd if=testfile1gb bs=1024k of=/media/sdb1/test.$i; done

  3. check md5 of files (all but the last, incomplete, should match): md5sum testfile1gb /media/sdb1/test.*

2
2

The simplest way to test the full capacity of an SD card is to fill it up with files, then verify the files are correct: diff -qr /directory/on/computer /directory/on/SD

Alternatively, you can use programs to write patterns, or chains of hashes, to a file, and then verify that they are correct.

As @Earthy Engine pointed out, it's important to fill up the SD card, then read the data, as traditional approaches that simply write a small block of data, then read it, are fooled by fake SSD cards.

2

I wrote a small script that does the following.

-creates a temporary directory to the target USB or SC card

-creates a 5MB randomly generated reference file with md5sum checksum

-copies the reference file to the target and generates a md5sum check from the target to confirm read/write success

-fills the target to capacity (100%) or stops when a checksum error occurs

-once the script stops naturally, it displays the target reported size, Used and Free amounts.

With this script I have concluded that I was ripped off by an ebay seller who passed a 8GB microSD for a 64GB

#!/bin/bash
#Save file as 'filltext' and remember to set the executable flag to run it
if [ -d "$1" ]; then
 if [ -d "$1/tmp" ]; then
  echo "."
 else
  mkdir $1/tmp
 fi

#Make a tmp file and fill it with 3MB of junk
 TMPTSTR=$(mktemp)      
 base64 </dev/urandom  | head -c 5000000 > $TMPTSTR

 TESTVAL=$(md5sum $TMPTSTR | awk '{ print $1 }')

 while $CHECKEDOK; do

  FL=$( tr -dc A-Za-z0-9 </dev/urandom  | head -c 5).TEST

  cp $TMPTSTR $1/tmp/$FL
  TESTTMP=$(md5sum $1/tmp/$FL | awk '{ print $1 }')
  if [ "$TESTVAL" != "$TESTTMP" ]; then   
   echo "Checksum ERROR"
   echo "Original: $TESTVAL Temp File:$TESTTMP"
   CHECKEDOK=false
   df $1 -Ph
   echo 
   echo 
   echo "Removing test files"
   rm $1/tmp -r
   rm $TMPTSTR
   df $1 -Ph
  else
   #echo -n "$FL..."
   clear
   df $1 -Ph
  fi
 done

else
 echo "Error: Directory $1 does not exists."
 echo "Usage: filltest [PATH]"
 echo
 echo "Try the PATH of a mounted USB dongle or SD card to confirm it's capacity"

fi
2
  • 1
    This will possibly give misleading results. Since the OS buffers filesystem writes, you're mostly testing your system's memory, not the SD card.
    – Cerin
    Jun 29, 2016 at 1:56
  • executing "hdparm -W 0 /dev/disk" should resolve a buffered writes issue.
    – Michael
    Jan 23, 2019 at 1:42
1

One can write a sequence of numbers (each row is 16 bytes) and then verify the contents:

dd if=<(seq -w 0 123456789012345) of=/dev/yourSdHere

Then verify skip == output (using a small sample of skip values that are smaller number of records written) e.g. skip=9876:

dd if=/dev/yourSdHere bs=16 count=1 skip=9876
000000000009876
1+0 records in
1+0 records out
16 bytes copied, ...

Or, make a sample of 20 locations with a one liner:

seq -w 000000000000000 NumberOfWrittenRecords | shuf | head -20 | while read i; do [[ $(dd if=/dev/yourSdHere bs=16 count=1 skip=$i) == $i ]] && echo ok || echo bad; done
  • Make sure that you are writing to the SD card
  • Write to a file of=tempFileOnSD, if you want to avoid destroying data stored your card (relevant only if it's not a fake)
  • In the case of 8GB card labeled as 64GB, the chance of passing all 20 tests is (8GB / 64GB) ** 20 < 1e-18
3
  • 1
    I had to read your answer three times before I understood what you were saying: “verify skip == output” is not clear.  And, unless I’m missing something, your approach requires that the user run 123456789012345 commands and manually inspect the output!  Obviously that’s unreasonable.  Why not just do seq -w 0 123456789012345 > /dev/yourSdHere and seq -w 0 123456789012345 | cmp - /dev/yourSdHere? May 6, 2019 at 16:29
  • Thanks for the comment :) I've edited my answer, hope it is better now!
    – karpada
    May 7, 2019 at 11:27
  • Also, 123456789012345 is a 15 digit number to make each number use 16 bytes. one can use the number of 16byte blocks on the SD
    – karpada
    May 7, 2019 at 12:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .