How to hash to specific partitions?

1 points · submitted May 2022 by u/NetFutility · help

Say I have 10k folders (partitions) and I have requests to persist data with guid IDs. How can I evenly distribute the storage of these incoming data so that each folder themselves won't have more than 10k documents?

1 comments · 56% upvoted · other discussions

comments

Read-only. Use the arrows to collapse threads.

pdffs 2 points · May 2022

On disk? You would typically just take the GUID and pick a depth that gives you the partitioning you need by nesting based on the trailing GUID characters, so with depth 4 you'd do:

GUID: 0001eb2e-385a-4c9a-91fd-c729f061ba82

Path: 2/8/a/b/0001eb2e-385a-4c9a-91fd-c729f061ba82

I'll let you do the math on how much nesting you need to get the right distribution.

permalink