Seems really cool! Might be a silly question, but why would you not want to just load it into a sql table and have a query grabbing a batch of rows instead?
Thanks!
The thing about loading it into a SQL table first is that if you're using Python, you're still going to have to use json.load to load all the data into memory first before you can store it in a DB table.
Also, let's imagine a simple process which is fed some JSON and needs to do something with it. It's a bit extra if we now need a process to also load it into SQL. That would probably end up being slower and more expensive in terms of resources.
Seems really cool! Might be a silly question, but why would you not want to just load it into a sql table and have a query grabbing a batch of rows instead?
Salaah01 [OP] 1 points · Jul 2023
Thanks!
The thing about loading it into a SQL table first is that if you're using Python, you're still going to have to use
json.loadto load all the data into memory first before you can store it in a DB table.Also, let's imagine a simple process which is fed some JSON and needs to do something with it. It's a bit extra if we now need a process to also load it into SQL. That would probably end up being slower and more expensive in terms of resources.
Lost_in_Nullspace 1 points · Jul 2023
Oh cool, so it's really useful for JIT tasks and times where you don't have the luxury of amortising the initial cost of setup over time.
I can imagine that'd be really useful for actually getting large amounts of data into tables to begin with.
Salaah01 [OP] 1 points · Jul 2023
Exactly, ETL stuff is something that comes to mind here (the extract part at least).
It supports async processing that might even be faster than working with each JSON object 1 at a time the traditional way.