Can Go solve this problem?
I have two relatively large (a few million) database tables and want to know if all the rows in table 1 exist in table 2. I have tried using EXCEPT on the two tables but the datasets are just too large.
Is this a problem I could solve using Go? If so how would you go about it? Maybe hashing the rows into a dictionary?
17 comments · 17% upvoted · other discussions
I don't think Go is relevant, your database is what's relevant.
Starting point: Make a compound index on each table of All The Fields You Care About, which may just be all the fields, but make sure the fields are in the same order. Then it's quite possible that a naive query will work acceptably well. If not, consider exporting them in a predictable format, and there's a well-known trivial solution:
`sort t1 t2 t2 | uniq -u`
This may or may not be fast enough, but "a few million" is pretty trivial. (The results would be any rows in table 1 that weren't found in table 2.)