hashlib md5 is slow
I need md5 checksums in part of a program. I created a small test script just to see how this works. The script is almost twice as slow as calling md5sum directly. Anyway I can tweak this or should I just use subprocess to call md5sum?
#!/usr/bin/python
import sys
import hashlib
filename=sys.argv[1]
class md5_checksum(object):
def __init___(self):
pass
def create_checksum(self, x):
self.x=x
self.x=hashlib.md5(open(self.x,'rb').read()).hexdigest()
def return_checksum(self):
return self.x
m=md5_checksum()
m.create_checksum(filename)
print "The checksum for file '" + filename + "' is", m.return_checksum() + '.'
13 comments · 85% upvoted · other discussions
Why are you using md5? SHA-2 is built into hash lib.
StorageNinja [OP] 3 points · Jan 2016
I created a 10GB file via "dd if=/dev/zero of=foo bs=1MB count=10000". md5sum against it took almost 19 seconds. Python via md5 took almost 33 seconds. I changed it from md5 to sha256. That took almost 62 seconds.
LightShadow 1 points · Jan 2016 · 3.13-dev in prod
I'm not exactly sure why the Python version is 2x slower ~ but, if you're doing a lot of MD5 on bigger files, it might be better to execute the command via a sub-shell and just pipe the stdout back into your program.
StorageNinja [OP] 1 points · Jan 2016
The DB dumps are all over the place. They start out at ~100MB and go up to 10GB. We have one DB, which in rare circumstances can be 20GB, but this is very rare.
heptara 2 points · Jan 2016
May I ask the purpose of this? There are specialised programs to test common databases for corruption and they'd likely be better than MD5. If it's for security (tamper protection), MD5 is inappropriate as it's been broken.
StorageNinja [OP] 1 points · Jan 2016
We have been facing silent file system corruption (several different types). If this keeps on happening, we may invest the time into switching over to ZFS.