slopfront
Reddit, read-only. No JavaScript. No cookies. No tracking.
preferences

hashlib md5 is slow

9 points · submitted Jan 2016 by u/StorageNinja · archived

I need md5 checksums in part of a program. I created a small test script just to see how this works. The script is almost twice as slow as calling md5sum directly. Anyway I can tweak this or should I just use subprocess to call md5sum?

#!/usr/bin/python

import sys
import hashlib

filename=sys.argv[1]

class md5_checksum(object):
    def __init___(self):
        pass
    def create_checksum(self, x):
        self.x=x
        self.x=hashlib.md5(open(self.x,'rb').read()).hexdigest()
    def return_checksum(self):
        return self.x

m=md5_checksum()
m.create_checksum(filename)

print "The checksum for file '" + filename + "' is",     m.return_checksum() + '.'

13 comments · 85% upvoted · other discussions

comments

pinano 8 points · Jan 2016

You are reading the entire contents of self.x into memory at one time. How big are these files? What happens if you instead stream the contents in a block at a time?

http://stackoverflow.com/a/3431838/3140

def create_checksum(self, x):
    hash = hashlib.md5()
    with open(fname, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash.update(chunk)
    self.x = hash.hexdigest()

From a code review standpoint, it confuses me when you use self.x to temporarily store the filename and then the resulting hash. Rename the argument from x to filename, and rename self.x to something more descriptive.

LarryPete 6 points · Jan 2016 · Advanced Python 3

I wanted to answer the same way, but you beat me to it.

But for comparison, my results were:

$ time md5sum foo 
2dd26c4d4799ebd29fa31e48d49e8e53  foo

real    0m24.804s
user    0m20.528s
sys 0m4.048s
$ time python3 test.py foo
2dd26c4d4799ebd29fa31e48d49e8e53   foo

real    0m23.204s
user    0m18.076s
sys 0m4.636s

So, pretty much no difference. The python version was even slightly faster (even if I retry).

pinano 1 points · Jan 2016

Is test.py the block-at-a-time version or OP's .read() version?

Also, does the performance differ in python2 vs python3?

LarryPete 2 points · Jan 2016 · Advanced Python 3

No, it's block-at-a-time.

Loading a 10GB file into my memory would probably kill my PC anyway :-)

Edit: And no, no significant difference between python2 and 3.

StorageNinja [OP] 2 points · Jan 2016

thx for the review, I could not agree more. It is already fixed up.

I will try the block at a time next.

[deleted] 1 points · Jan 2016

[deleted]

pinano 1 points · Jan 2016

First, interleaving I/O and computation allows the computer to maximize its utilization of both. On the default Python implementation, file.read(N) calls down to the C stdlib function fread, which is very likely to have its own read-ahead buffer and filesystem cache.

Also, these are 100MB-20GB files. If /u/StorageNinja has less than that much free memory, then the program is guaranteed to hit swap.

Flynn58 -5 points · Jan 2016

Why are you using md5? SHA-2 is built into hash lib.

StorageNinja [OP] 3 points · Jan 2016

I created a 10GB file via "dd if=/dev/zero of=foo bs=1MB count=10000". md5sum against it took almost 19 seconds. Python via md5 took almost 33 seconds. I changed it from md5 to sha256. That took almost 62 seconds.

Sukrim 2 points · Jan 2016

Modern CPUs (will) have special instructions to compute SHA-1 and SHA-256 hashes: https://software.intel.com/en-us/articles/intel-sha-extensions

This means it will be benefitial to use a secure hash function instead of a broken one in some cases. Did you also try with sha256sum or only with Python's hashlib?

LightShadow 1 points · Jan 2016 · 3.13-dev in prod

I'm not exactly sure why the Python version is 2x slower ~ but, if you're doing a lot of MD5 on bigger files, it might be better to execute the command via a sub-shell and just pipe the stdout back into your program.

StorageNinja [OP] 1 points · Jan 2016

The DB dumps are all over the place. They start out at ~100MB and go up to 10GB. We have one DB, which in rare circumstances can be 20GB, but this is very rare.

heptara 2 points · Jan 2016

May I ask the purpose of this? There are specialised programs to test common databases for corruption and they'd likely be better than MD5. If it's for security (tamper protection), MD5 is inappropriate as it's been broken.

StorageNinja [OP] 1 points · Jan 2016

We have been facing silent file system corruption (several different types). If this keeps on happening, we may invest the time into switching over to ZFS.