u/codeinred

this is really cool!

Performance differences between zones should be negligible, at least for vtz! Benchmarks were run across randomly generated timestamps over a 200 year period between 1900 and 2100, with America/New_York being the default timezone for all benchmarks. This zone requires correct handling of daylight savings time, and of historical rule changes, which is why it was chosen.

I have a section of the README that explains the underlying conversion algorithm in depth!

https://github.com/voladynamics/vtz?tab=readme-ov-file#how-it-works-vtzs-algorithm-for-timezone-conversions

Direct support for date/time: Right now vtz has support for types in std::chrono, and it has support for formatting and parsing timestamps.

I would like to add more complete support for dates and times, and the machinery for doing so already exists inside include/impl/vtz/civil.h, however this is still on my TODO list.

Compatibility with Hinnant tz: vtz does it's best to match std::chrono and date/tz.h for the API of vtz::time_zone. vtz::local_info and vtz::sys_info both match std::chrono::local_info and sys_info respectively, and of course vtz has vtz::choose to match std::chrono::choose.

vtz::time_zone provides some additional functions on top of those dictated by the standard, but these are there for user convenience, and should not have any impact on compatibility.

I need to spend some time fleshing out other parts of the std::chrono API (eg, adding a polyfill for std::chrono::zoned_time, adding support for calendar types, etc).

Any contributions on this front would be most welcome.

Support for IANA tz database: vtz supports reading a downloaded IANA timezone database, just the same as date does!

If tests and benchmarks are enabled, the build system will download a copy of the tz database, but vtz itself does not attempt to perform any sort of download at runtime.

Instead, you have two options for running vtz:

• On Unix platforms, vtz will default to using the compiled tzif files shipped with the system.
• However, if you provide the VTZ_TZDATA_PATH environment variable (or you call set_install(), vtz will check the given path for a copy of the tz database.

So vtz comes with out-of-the-box support for both sources, and you don't need to recompile to change which source you're using.

That being said, vtz does it's best to be helpful on this front:

• You can override the name of the environment variable at compile time, eg -DVTZ_TZDATA_PATH_VARS=MY_APP_TZDATA_PATH will compile vtz such that it uses MY_APP_TZDATA_PATH instead of VTZ_TZDATA_PATH
• You can provide multiple env vars which vtz will check, in order
• vtz supports set_install() if you would rather just set the path manually
• vtz does it's best to provide helpful error messages when it's unable to load the tz database (either because the environment variable was bad, or the path provided by set_install() was bad, or something else).

Example error message with bad path:

$ env VTZ_TZDATA_PATH='bad_path' build/examples/vtz_tldr
libc++abi: terminating due to uncaught exception of type std::runtime_error: Unable to load the tz database: Error when opening "bad_path/version". What: No such file or directory (OS Error 2)

Checked the following locations:

- getenv("VTZ_TZDATA_PATH") -> "bad_path"
- get_install() -> "bad_path"

Please configure one of the above (or call vtz::set_install()) so that your application can find the tz database.

The timezone database may be downloaded at https://www.iana.org/time-zones

To use the timezone database, unpack one of these source files, and configure the environment to point to that directory.

Note: This application checks for tzdata source files in the directory given by getenv(...)

I would also like to add the option to embed the tz database within vtz (so that having a separate copy elsewhere on the system is unnecessary), however this feature is also on the TODO list.

I've put a ton of time and effort into the implementation, and it would be nice for it to see wider use. I would definitely be open to this!

As an aside - I greatly appreciate the work that you've done on this front.

Your library is what first brought me in contact with timezones, and chrono-Compatible Low-Level Date Algorithms was an incredibly useful resource

They're signed. And this isn't incorrect per se, but this problem only manifests for inputs after December 3rd, 292,277,026,596, which is approaching the end of the stelliferous era, and I didn't expect to see it come up in typical workflows.

Edit: I will update the library to handle overflow in time zone conversions due to very large input times

The only place I can think of this being a source of security vulnerabilities is if the security mechanism were checking the expiry of a certificate against a timestamp from an untrusted source, but such timestamps are typically in UTC (so no overflow), and if an untrusted source can directly provide an arbitrary timestamp they could simply date it prior to the certificate expiry.

That being said, if it's a concern I could implement saturating arithmetic without affecting performance on the happy path, which already does a bounds check

vtz uses 64 bit ints to represent timestamps by default. If a timestamp (eg, in seconds) is so large that adding a zone offset to it results in a value larger than INT64_MAX, this will result in integer overflow. Because the result is also represented as a 64-bit int under the hood, the result will be incorrect.

There are certain workflows that require either (1) processing large numbers of timestamps in a zone-aware manner, or which (2) care about events which occur in a particular timezone across a wide range of dates.

Timezone conversions should not to be a bottleneck. Doing something in a zone-aware manner ought to have a negligible impact on whatever calculation you're running.

But as things are implemented now, handling timezones correctly tends to make certain kinds of operations significantly slower.

• Elastisearch: time_zone option makes date histograms much slower #28727
• STL: <chrono>: time_zone::get_info code is excessively slow #5304
• STL: <chrono>: Major performance issues when using zoned_time or time_zone #2842
• Hinnant date: Possibly big performance improvement for time_zone::to_sys() #817
• SQL Server: AT TIME ZONE Performance Issues

Timezones are often seen as big, and complex, and scary, so people mostly accept that timezone conversions are slow, and then they maybe try to put in some clever caching logic to reduce the frequency of zone lookups. But that requires a lot of testing and validation in and of itself, so it's not uncommon for issues like that to simply go unfixed.

vtz tries to be (1) fast by default, (2) fast for all use cases, and (3) to provide an implementation so close to being truly optimal (at least at zone conversions) that it becomes the definitive library for handling timezones.

vtz doesn't care if your timestamps are sorted, or unsorted. vtz doesn't care if your timestamps contain times absurdly far in the future, or in the past. vtz doesn't care if timezone lookups are batched, or performed one at a time. vtz will deliver performance beyond any other implementation, and my hope is that one day no one will have to think about the performance of a timezone conversion, ever again.

(Edit: typo)

Tbh my experience with swig has been much worse than with pybind11. I’ve used both for the same library (pybind11 for python, and swig for C#), and maintaining the swig bindings is much more of a headache, especially with any even slightly unusual types, whereas pybind11 is basically trivial

That makes sense. Good luck on your project!

Sorry, I was on my phone so I wasn't able to test it.

If you have C++23, you can use deducing this to use overload sets, and there is no need to have an intermediary function like 'getVisitor'.

Given the following definitions:

using json_value = rva::variant<
    std::nullptr_t,
    bool,
    double,
    std::string,
    std::vector<rva::self_t>,
    std::map<std::string, rva::self_t>>;

using json_list = std::vector<json_value>;
using json_object = std::map<std::string, json_value>;

We can define our overload set, where the functions that need to use `std::visit` take a `self` parameter. Here, I am using fmtlib to handle the conversion to string.

    auto visitor = Overload {
        [](std::nullptr_t) -> std::string { return "null"; },
        [](std::string const& s) -> std::string { return fmt::format("\"{}\"", s); },
        [](this auto&& self, json_list const& xx) -> std::string {

            std::vector<std::string> ss;
            for(auto const& x : xx) {
                ss.push_back(std::visit(self, x));
            }
            return fmt::format( "[{}]", fmt::join(ss, ", "));
        },
        [](this auto&& self, json_object const& xx) -> std::string {
            std::vector<std::string> ss;

            for(auto const& [k, v] : xx) {
                ss.push_back(fmt::format("\"{}\": {}", k, std::visit(self, v)));
            }
            return fmt::format( "{{ {} }}", fmt::join(ss, ", "));
        },
        // Fallback - just print the value as-is
        [](auto const& x) -> std::string { return fmt::format("{}", x); },
    };

Note the parameter `this auto&& self` - this first parameter is the overload set, and we can pass it back to `std::visit`!

Finally, we can construct a value, and then print it:

    json_value v = json_object {
        {"a", "Hello"},
        {"b", 10.0},
        {"c", json_list{1.0, 2.0, 3.0, nullptr, false}},
        {"point", json_object {
            {"x", 0.1},
            {"y", 0.2},
            {"z", 0.3},
        }}
    };

    fmt::println( "v = {}", std::visit(visitor, v) );

The output is:

v = { "a": "Hello", "b": 10, "c": [1, 2, 3, null, false], "point": { "x": 0.1, "y": 0.2, "z": 0.3 } }

You can run this example with godbolt.

All of this is very nifty, but if you're working with json in production, I recommend using an off-the-shelf json library.

Option 1: If you want to do that you can create a struct and overload operator(). Then you can call visit recursively by just passing *this:

struct MyVisitor { book doubleOk = false; void operator()( double X ) { … } void operator()( std::string const& s ) { … } void operator()( std::map<std::string_view, json_value> const& m ) { for ( auto const& [k, v] : m ) { std::visit( *this, v ); } } }

Option 2, using overload sets: have a function that returns the visitor. You can just do recursive visiting by calling getVisitor again inside the lambdas of the overload set

auto getVisitor( bool& doubleOk ) { return Overload{ [&] ( double X ) { doubleOk = true; }, [&] ( std::map<std::string, json_value> const& m ) { for ( auto const& [k, v] : m ) { std::visit( getVisitor(doubleOk), v ); } }, }; }

Doing type erasure with std::function breaks the visitor pattern. The std::function only accepts json_values, so when you call std::visit, it converts all the inputs right back into a json_value

This would also be the case with a normal std::variant.

Don’t assign the overload set to a std::function, just use auto here

Is it possible you accidentally added some very large files?

Yes

Regarding context - a 32-bit number won’t have more than 32 factors, and a 64-bit number won’t have more than 64 factors. In either case it’d be straightforward to just have an array on the stack, and construct a vector at the end.

Rather than trying to force the compiler not to inline push_back, I would instead just avoid using push_back in performance-critical paths. There’s usually an alternative

The idea you’re describing here is basically the same as a micro service architecture (assuming the modules do different things), where the services communicate over an API. Unless you have very stringent performance requirements, using a language with nicer async communication capabilities might make your life easier (eg, JavaScript, python, go, or Rust)

/tmp is often stored in RAM. If you want to be absolutely sure it doesn’t touch the disk, the most straight forward way that I can think of is spawning a daemon and then setting up shared memory with the daemon. That way if the main program crashes, the daemon persists.

The /tmp directory on Linux is designed exactly for that use case, and windows has an equivilant

Any time a function could fail, std::optional can be used for the return value. It’s an alternative to throwing an exception that allows better performance in the case that failures are expected/common.

Smart pointers? If you use any degree of polymorphism (virtual classes, members), you should use unique_ptr or shared_ptr rather than manually allocating or freeing memory.

I would choose a simple project that’s at least semi-relevant to the company’s field.

You want to break up the program info hpp and cpp files because no one likes a giant code file that’s thousands of lines long, and a clean interface (provided in the header) is helpful.