machine text

@hilary was experimenting with selecting a random line from a file. there were many excellent contributions to the discussion. here’s my convoluted approach.

dd if=file.txt skip=$(expr $(date +%N) \% $(stat -c “%s” file.txt)) ibs=1 count=200 2>/dev/null | sed -n ‘2{p;q;}’

it uses “dd” to seek to a random point in the file, read, and print 200 bytes, discards dd’s diagnostic output (normally written to stderr), and uses sed to filter the output.

the random number is generated by taking the nanoseconds portion of the current time modulo the size of the file in bytes. it’s not exactly cryptographically random, but should be good enough. there are lots of other options for generating a random number for this case, but I favored brevity over entropy.

we’re not operating on line numbers but there still needs to be an upper limit for the random number that gets generated. we use the filesize in bytes as the upper limit, which we can get very quickly by using ‘stat’.

why read in 200 bytes? it’s arbitrary but since we are not operating on the basis of line numbers, the intention is to be sure we read at least one entire line from the file and then to use sed to select only the second line from the output (the first line we see is likely to only be part of a line, the second should be a complete line) and print that. this assumes that the file is organized as newline terminated records. the 200 byte read size can be adjusted up or down depending on the typical record size in the file.

could be more portable (works on ubuntu, RHEL, etc). but on a mac, ‘stat’ accepts different format strings and “date” doesn’t support %N for nanoseconds, bleh. on a mac you could instead use something like:

 dd if=file.txt skip=$(jot -r 1 0 $(stat -f “%z” file.txt)) ibs=1 count=200 2>/dev/null  | sed -n ‘2{p;q;}’

it’s fast because 1) “dd” is fast and 2) since it doesn’t use a line number to pull out the random line, we don’t have to count the lines to set the upper limit. in my testing (admittedly on a box with fast (though not ssd) disks), it could consistently pull a random line from a 300GB (yes, GB) file in around 10milliseconds, even from deep in the file.