Data structure for versioned records
I'm trying to make a database structure for versioned data for a really simple CRUD application. The versioning is per record, but the normal ways you data version in SQL don't apply because
1) It's not *audited* data. I.e. it's not one current version, and multiple historical versions, where the business logic only runs on the current version. The different versions are all "hot" and can be queried with the same queries.
2) It's not temporal data. I.e. the versions are v1.0, v2.0 and so on. Not date ranges. So existing solutions that solve temporal queries don't apply.
One valid structure, for example looks like so:
| id | rev_id | name | version |
|---|---|---|---|
| 1 | 1 | A | v1 |
| 1 | 2 | A | v2 |
| 2 | 3 | B | v2 |
| 3 | 4 | C | v1 |
Querying for v2 data now gives the A,B,C records with rev_id {2, 3, 4} while querying for v1 only gives A, C records with rev_ids {1, 4}. This works well, but the problem is the massive complexity that comes with it. Queries have to make a max-by-version thing. Even though the database has just 2 main table, joins between the tables become really complex. The ORM (EF core in this case) gets really confused. Performance suffers.
In a legacy version of this application, the whole database was cloned instead. So v1 and v2 databases were separate. This came with its own problems of course, but it did solve the CRUD complexity within each database.
Is there some compromise solution between these two solutions (between row versions and full db forking)?
This will be EF Core/SQL server but I do think the question is mostly db agnostic.
16 comments · 100% upvoted · other discussions
Can you give a concrete example of what kind of data it is and what kind of questions need be answered with queries? It's hard to understand what kind of data would be required to be stored like this and what you could do differently.
afops [OP] 1 points · Oct 2024
It's resource data for translation of applications. So so it's a simple key/value store like any resource lookup for translation. But the complexity arises from supporting multiple different applications in different versions.
So, simplified to non-versioned data it looks like this:
resource keys;
resource values;
The typical query is to then generate the set of resources for a given application, for N different languages like
fr = { "YES": "oui", "NO": "non" }, and so on. And this must be possible to do for any version. For example: after v1 is shipped for ACME app, then changes is made to the data for v2. But it must be possible to regenerate the v1 data again, for version v1.1, without seeing any of the changes made for v2.0.
This would be almost trivial to make if the data store was a git repository with resource files in different branches, rather than a relational DB.
Added to the above structure is 2 complexities: the resource keys must be able to "inherit" another, thus forming a resource hierarchy. I can restrict this to 1 level however. Then it's the support for app_version being "1.0" or "2.0" and so on. This becomes especially tricky when combined: a key can inherit from a different key in v1 than in v2, for example.