Can Go solve this problem?
I have two relatively large (a few million) database tables and want to know if all the rows in table 1 exist in table 2. I have tried using EXCEPT on the two tables but the datasets are just too large.
Is this a problem I could solve using Go? If so how would you go about it? Maybe hashing the rows into a dictionary?
17 comments · 17% upvoted · other discussions
Bloom filter?
dgryski 2 points · Jun 2019
This works but isn't a complete answer (unless having a probabilistic response is sufficient). If you want an exact answer you'll need a little more work. If you're happy with approximate answers, then you can use multiple bloom filters to reduce the odds of an incorrect answer to be arbitrarily small. But it's still not exact.
Another answer if both tables are sorted by PK, you can query them both and do a linear time intersection.
drvd 1 points · Jun 2019
Use bloom and cuckoo filter?