hashlib md5 is slow

I need md5 checksums in part of a program. I created a small test script just to see how this works. The script is almost twice as slow as calling md5sum directly. Anyway I can tweak this or should I just use subprocess to call md5sum?

#!/usr/bin/python

import sys
import hashlib

filename=sys.argv[1]

class md5_checksum(object):
    def __init___(self):
        pass
    def create_checksum(self, x):
        self.x=x
        self.x=hashlib.md5(open(self.x,'rb').read()).hexdigest()
    def return_checksum(self):
        return self.x

m=md5_checksum()
m.create_checksum(filename)

print "The checksum for file '" + filename + "' is",     m.return_checksum() + '.'

13 comments · 85% upvoted · other discussions

comments

Flynn58

Why are you using md5? SHA-2 is built into hash lib.

StorageNinja [OP]

I created a 10GB file via "dd if=/dev/zero of=foo bs=1MB count=10000". md5sum against it took almost 19 seconds. Python via md5 took almost 33 seconds. I changed it from md5 to sha256. That took almost 62 seconds.

LightShadow

I'm not exactly sure why the Python version is 2x slower ~ but, if you're doing a lot of MD5 on bigger files, it might be better to execute the command via a sub-shell and just pipe the stdout back into your program.

StorageNinja [OP]

The DB dumps are all over the place. They start out at ~100MB and go up to 10GB. We have one DB, which in rare circumstances can be 20GB, but this is very rare.

heptara

May I ask the purpose of this? There are specialised programs to test common databases for corruption and they'd likely be better than MD5. If it's for security (tamper protection), MD5 is inappropriate as it's been broken.

StorageNinja [OP]

We have been facing silent file system corruption (several different types). If this keeps on happening, we may invest the time into switching over to ZFS.