hashlib md5 is slow

8 points · submitted Jan 2016 by u/StorageNinja · archived

I need md5 checksums in part of a program. I created a small test script just to see how this works. The script is almost twice as slow as calling md5sum directly. Anyway I can tweak this or should I just use subprocess to call md5sum?

#!/usr/bin/python

import sys
import hashlib

filename=sys.argv[1]

class md5_checksum(object):
    def __init___(self):
        pass
    def create_checksum(self, x):
        self.x=x
        self.x=hashlib.md5(open(self.x,'rb').read()).hexdigest()
    def return_checksum(self):
        return self.x

m=md5_checksum()
m.create_checksum(filename)

print "The checksum for file '" + filename + "' is",     m.return_checksum() + '.'

13 comments · 79% upvoted · other discussions

comments

pinano 8 points · Jan 2016

You are reading the entire contents of self.x into memory at one time. How big are these files? What happens if you instead stream the contents in a block at a time?

http://stackoverflow.com/a/3431838/3140

def create_checksum(self, x):
    hash = hashlib.md5()
    with open(fname, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash.update(chunk)
    self.x = hash.hexdigest()

From a code review standpoint, it confuses me when you use self.x to temporarily store the filename and then the resulting hash. Rename the argument from x to filename, and rename self.x to something more descriptive.

permalink

LarryPete 6 points · Jan 2016 · Advanced Python 3

I wanted to answer the same way, but you beat me to it.

But for comparison, my results were:

$ time md5sum foo 
2dd26c4d4799ebd29fa31e48d49e8e53  foo

real    0m24.804s
user    0m20.528s
sys 0m4.048s
$ time python3 test.py foo
2dd26c4d4799ebd29fa31e48d49e8e53   foo

real    0m23.204s
user    0m18.076s
sys 0m4.636s

So, pretty much no difference. The python version was even slightly faster (even if I retry).

permalink

pinano 1 points · Jan 2016

Is test.py the block-at-a-time version or OP's .read() version?

Also, does the performance differ in python2 vs python3?

permalink

LarryPete 2 points · Jan 2016 · Advanced Python 3

No, it's block-at-a-time.

Loading a 10GB file into my memory would probably kill my PC anyway :-)

Edit: And no, no significant difference between python2 and 3.

permalink