Can Go solve this problem?
I have two relatively large (a few million) database tables and want to know if all the rows in table 1 exist in table 2. I have tried using EXCEPT on the two tables but the datasets are just too large.
Is this a problem I could solve using Go? If so how would you go about it? Maybe hashing the rows into a dictionary?
17 comments · 17% upvoted · other discussions
Bloom filter?
dgryski 2 points · Jun 2019
This works but isn't a complete answer (unless having a probabilistic response is sufficient). If you want an exact answer you'll need a little more work. If you're happy with approximate answers, then you can use multiple bloom filters to reduce the odds of an incorrect answer to be arbitrarily small. But it's still not exact.
Another answer if both tables are sorted by PK, you can query them both and do a linear time intersection.
NetFutility [OP] 1 points · Jun 2019
I'll have to read up on bloom filters but if there's a decent level of confidence then that might be a good enough solution for now.
The two tables don't have a PK or any useful unique columns that can help batch the processing.
ssoroka 1 points · Jun 2019
Bloom filters may be helpful. They could answer the question “is this record in this other data set?” With “definitely no”, or “maybe”. If maybe, you do the more expensive check.
Also a map of ids would also be an O(1) check, and many languages, including Go, could hold a million ids in a map/hash to do quick lookups.