u/kenfar

Mostly, but not entirely - riding a bike recklessly can get you severely injured or killed without cars.

But given that cars on the roads aren't going away, they're part of the environment that all cyclists and motorcycle riders must adapt to.

He apparently thinks so.

Imagines that wearing helmets protects kids on motorcycles from getting killed in traffic when hit by an oncoming car due to reckless driving.

I never liked weapon specialization, but I probably never gave it a fair chance since my group created their own specialization rules years before TSR released them. Ours were simply:

• +1/+1 for every level of specialization you take
• doesn't matter what class you are
• you could specialize in multiple weapons

This was simple & elegant - no complicated rules about number of attacks (especially with dual weapon fighters), no arbitrary rules that you had to be a pure-class fighter, no arbitrary rules that only someone dedicated to fighting gets a benefit from focusing their efforts on fewer weapons, etc, etc, etc.

Back to the original: I don't think the official specialization rule is necessarily OP, but it is extremely powerful, and doesn't make much sense.

An improved Wand of Wonder

https://docs.google.com/document/d/1XPaCLp6rJ0rQ0agNt-zTgKv0NEFpKR711g0TOrw_qyk/edit?usp=sharing

That's a difference without a distinction - depending on how their budget works that might mean they can't afford what was intended to save labor.

Note: I'm not defending them, maybe they are merely after revenue. I'm just pointing out that this is not evidence of that.

Hmm, that's a leap: using the vans might have been considered as a cost-savings measure compared to hiring more cops. Getting rid of the vans because they aren't saving money may be just that and have nothing to do with a "revenue stream".

Mistakes will always be made, upstream systems will make unannounced changes, software will fail, and recovery will only be partially successful - losing or mangling data. It happens.

As data becomes more critical the acceptance of incorrect data drops. And multiple solutions are required.

One that I always do is to reconcile major counts against upstream systems. Row counts by day, or row counts by customer by day. Or by sensor by day. Etc.

Another reconciliation I'll do is to compare a small random sampling of entire rows against their source.

The other thing is to to collect frequency distributions of columns (maybe binned for high-cardinality) every hour or day, then look for deviations over time.

Sometimes those checks get really specific - like looking for anything beyond 3 stddev for a given day of week and hour. It's amazing how this will spot problems upstream that the folks supporting those systems aren't even aware of.

Can you provide more detail?

I've been very happy with event-driven data pipelines in which each step is stored on s3 and triggers the next through a s3 event notification.

The latency depends on the config - if the next step is ready on say Kubernetes then the messaging & wait time is negligible, any unnecessary latency is just due to the redundancy of writing the data and then reading it back. I feel this is typically worth a few seconds for large files given the flexibility and manageability it provides.

But if I needed to routinely keep my latency at under a second I would probably consider streaming solutions.

Being able to test each step and see why a record was rejected can save a lot of debugging later.

Yep - and help find the problems while they're small and before the customer notices.

Many of the warehouses I've built had extremely strict data quality requirements. The first one had to balance $14 billion in expenses against the company's financial systems - and the penalty for screwing up was that they would shut down the system and fire the team.

What are you thinking about when you refer to logic the team needs to maintain outside the warehouse?

Hmm, in my experience almost all users are satisfied if you proactively handle DQ issue Response extremely well.

The exceptions are if the issues prevent them from doing their jobs, if it causes them a bunch of extra work, or if they simply hate you.

If it's any of these latter cases it seems like the only solution is a focus on the Prevention - which is sounds like you're doing with replacing the cause of frequent problems.

Good luck!

Would detecting the missing records, reporting these to customers along with expected resolution targets calm them down?

Makes sense. How does your CDC solution work?

Do you have really good reconciliation between the source & warehouse/data lake - or are you getting surprised by the customer's identification of missing records?

And do you have good idempotency to support these backfills?

ETL can keep source data around just as easily as ELT - that's no difference.

The top reasons I'll go for ETL:

• very readable code: when the transforms get complex python simply beats SQL and regex. I want people to actually review code, and if it's a nightmare to look at - nobody really does.
• high-quality unit tests: each field transform is a separate function, each gets a dedicated unit test class, the inputs are typically just a field value or two.
• high-quality logging - I like my logs to get into the details: for any given micro-batch, for each field, how often did the source value pass validation and how often was it thrown away? How many rows were rejected?

For really simple stuff, SQL is great. But when on-call and committed to delivering high-quality, complete and accurate data, I never reach for ELT and SQL.

Sure - I typically write these as event-driven data pipelines:

• Files get written to s3, that event generates a SNS/SQS set of events to all consumers.
• Consumers could be lambdas, but at high frequency I mostly make them ECS or kubernetes.
• aws files on s3 are the preferred output of these jobs - which then allows the easy construction of a chain of multiple event-driven processes.

This pattern allows me to write the process in python (or whatever). What I like about a language like python is that the logging can be very effective, all file formats are supported, etc.

I will sometimes include file diffs to support the creation & maintenance of versioned dimension tables.

Agreed - there are those cases where seconds matter.

But I have scaled this approach up to running micro-batch files getting dropped off on S3 every second - to support 40+ billion rows a day and it worked great.

So, it's rare that I need the kind of speed that demands kafka, flink, etc.

I typically go with micro-batches - processing data every 5-15 minutes. That's usually fast enough that people don't have long wait times to see new data, and failures don't impact people much.

What I prefer about either of these over streaming is the ease with which you can reprocess a single batch. In my world these batches (or microbatches) are just files. And they represent a discrete unit of work. They're easy to query, examine, modify if necessary, copy, etc. Streaming makes all of this harder.

I think this approach is fine, but a few thoughts:

1. Data Warehousing is a process not a place: it's a process of curation in which you organize data around subject and version it. So, if you wanted to create a data warehouse holding the last 10 years of your bowling league - you could easily do that in a spreadsheet.

2. Data Warehousing was originally implemented using relational databases for about the first 20-30 years. Databases like db2, informix, and teradata was especially effective since you could split the data across multiple servers and run queries in parallel. Oracle and SQL Server were also popular, Oracle because of marketing.

3. Using a relational database for a data warehouse is still a reasonable solution: you might miss out on columnar storage and load speed, but you still have parallelism within an instance as well as across instances, indexing (occasionally very helpful), a very smart optimizer, data validated at load time rather than query time, may reuse the same engineering skills you already have in-house, etc.

4. The original justifications for the data-lake - have mostly evaporated: storing sound, images, and video has turned out to be a rare edge case. Storing xml and json and querying it - is supported just fine in a relational database, and is not inconsistent with data warehousing. Separating compute from storage was common with data warehousing 15 years before hadoop then lake houses emerged. Loading data as compressed files and then reading them later as tables for improved cost & load speed is consistent with data warehousing, and possible with relational databases (via external wrappers). And the vast majority of queries are written in SQL - not map reduce.

5. The data lakehouse then (IMHO) is more of an effort by a vendor (data bricks) to create a standard for assembling products into something that would be competitive with classic relational database data warehousing, but use focus on object storage, columnar storage, etc - all aspects that a traditional relational database also could do, if the vendors wanted to spend the effort on them.

So, then what's actual the difference between a warehouse & data lake house? Here's a suggestion:

• Formal: warehousing is a process, data lakehouse is a product
• Informal: They're both OLAP products, warehouse uses relational database tech, data lakehouses don't.

e-motos driven by kids without the maturity to make good decisions and getting seriously hurt and killed is becoming an epidemic.

Rather than sweeping this under the carpet we need to share it and talk about it - if we're going to prevent more tragedies.

As others have said - it depends on your route.

I really liked my class III ebike when I occasionally had to use streets - it was safer since I could match the speed of cars more closely. It was also great when I was tired or when I was wearing a lot of clothes, or when it was really hot.

But otherwise, I preferred my pre-e-bike.

It also got me to fall completely in love with commuting in the winter with great studded tires on a shitty old mountain bike.

Got it, so:

• No initial training
• No splitting of xp
• No rules about not getting any xp for new class if you used your prior classes

Sounds like the strengths are that it's very simple, easy to understand and follow.

The downsides are that it's subject to abuse in a game with xp=gp rather than milestones, and you don't even have to use the new class's features to go up a level in it.

But those issues could probably be addressed by the DM with some simple houserules.

Looks great!

I've had a few dual-classed characters over time. It worked ok for them. But, it's not for most characters.

Personally, I don't enforce racial restrictions on either multi-classing or dual-classing. None of that make any sense. Just unnecessary and clunky rules.

Can you provide more info on what you mean by modern multiclassing? Not familiar with it.

Also, as a matter of world-building, any sensible world ends up dominated by demi-humans who live for centuries and don't hit a level cap.

This is only true if you assume that the player characters represent typical non-humans. But they don't have to - in which case the world-building is unaffected.

It creates a situation where there is no advantage to playing a human.

Level limits are a crude tool to achieve that. You could just give a human a +1 to put anywhere they want, or a +10% XP bonus, etc, etc.

The other issues with racial limits is that it's a huge drag for the play if they hit that limit or think they're going to hit it.

Sure, here's a few:

1. More completely random results is less valuable than adjusting the kinds of results to make it less dangerous to the user to use. The original is harmful to the user about 16% of the time - so it's almost immediately sold off or put in storage.

2. The user has a chance to choose the result it produces, or at least the category of result. Maybe this is adjusted by their class, level, and number of times they've used it.

3. I like when they have themes. I have one written up that has a bit of a mushroom/gelatinous cube/flying squirrel/chicken theme. This theme mostly surfaces on the summoning and broken-reality results.

4. This is a good candidate for a slowly self-charging wand.

By "near" you mean in the city of Boulder?

No, it's far easier to just point out the problems to city council.