u/MobileChipmunk25

Ik werk zelf remote als software engineer voor een Amerikaans AI Lab en heb toevallig ook geen relevante diploma’s. Grappig genoeg is dat wel iets waar ze erg op gefocust zijn tijdens het proces.

Qua salaris klinkt dit niet heel vreemd. Ik zou vooral nog goed kijken naar opties/aandelen in het aanbod, zeker met het oog op aankomende box 3 heffingen kan dit wel een dingetje zijn.

Houd er ook rekening mee dat die Amerikanen vrij fanatiek zijn als het op werk ethos aankomt. Er wordt wel een tikkie harder gewerkt dan bij de meeste bedrijven hier

I didn’t know Moshi, looks very nice! I’m gonna give it a try!

I’m running CC in a cheap server and control it remotely using Omnara. Reason for it is that I travel quite a bit and this way I can control it from my MacBook, iPad or phone.

Tried using the Claude managed remote servers first, but missed granular control over system dependencies and got weird connection timeouts all the time.

Then moved on to renting a cheap server from Hetzner and installed CC. Started out controlling it via SSH and a Zellij session, which works fine with your laptop or iPad. Also wanted to steer it from my phone, so looked into options for that. Tried Happy at first, but that didn’t really work for me. Found Omnera after that, which has been working great!

I really love the fact that I can go out, close my laptop and it will just continue working. That’s why I’ve taken this approach.

My work consists for 80% of streaming use cases, or at the very least micro batching (looking at you, Spark Structured Streaming).

Most of my projects are customer facing, like on-site personalisation. Low latency is key for such use cases.

I’ve primarily used Apache Flink (DataStream API) over the past couple of years, I think it’s awesome :) Also used Spark Structured Streaming, but I prefer true stream processing over micro-batching. The concept of batches in a streaming environment just doesn’t make sense to me.

Exactly the setup we use; Flink Operator on kubernetes with ArgoCD

I’ve been doing DE for roughly ten years. I’ve always loved exploring all the tools in the Apache ecosystem, something about those open source tools that sparked my interest.

Since a couple of years I’ve found out that I like the technical challenge most. I’ve basically stopped scripting and spend most of my time writing stream processing applications in Java with Apache Flink. It’s the maturity of that ecosystem and the level of technicality required that I’ve grown to love.

Got a bit bored from doing Terraform, the endless Spark DF manipulations and the SQL scripts to be honest. I try to leave these to others now.

I spoke with one of their sales reps about a month ago about their roadmap. He told me that they aim to grow into the full stack data platform area, so in line with your expectations. They already have quite a few of the components in place, the only thing missing is indeed a compute engine and storage. I'm wondering if they will be integrating an open source compute engine into the product to tick that box as well. Storage will most likely stay Iceberg on top of S3, GCS etc.

You can run the necessary components in Docker (mainly Kafka). You can generate fake clickstream data into a Kafka topic using something like the Faker library in a simple Python script.

Kafka itself is “just” the mechanism to get your data from A to B. So it sounds like you’ve been using it right.

To me, it became interesting when applying stream processing frameworks on top of that data, such as Apache Flink (personal favourite with the DataStream API) or Kafka Streams.

A use case could be processing clickstream data in real-time to calculate aggregated metrics for user profiles (number of product views in last 30 minutes in a rolling window etc). These are valuable for marketing related projects, such as personalisation.

If your main reason for looking into n8n is building AI agents, RAG etc, I would suggest to dive into the LangChain library instead. It’s what powers the AI features of n8n under the hood. This way you get the best of both worlds.

I think Heroku would be just fine. I’m not familiar with Render, but I guess it will be just fine as well.

One thing I would recommend is that you host your database in the same provider as where you decide to host your application. This way you can properly secure your database from public internet access. Otherwise you would have to connect your API to the database over public internet or setup some sort of private gateway between the two platforms.

You have quite a few options to get going, like Confluent, Aiven, but also GCP Managed Service for Kafka. You can get a 3-node startup-tier cluster with Aiven for $200 USD per month, hosted in GCP. You probably would want to use Private Service Connect to link it to your cloud run instances over a private network (its called privatelink in Aiven).

Also wanted to look for a quick pricing calculation for Confluent, but their pricing calculator is nowadays locked behind a quote form 😅

It still feels like there is quite a big gap between developing ML models and the deployment of models (without relying on a proprietary platform).

There are open source projects that promise to deliver the full MLOps cycle, but I haven’t come across an option that’s mature on all parts of the MLOps cycle. Most implementations I’ve seen still feel quite hacky and lack important features, like traceability and explainability.

It might be overkill in terms of what it could technically achieve, but since they have a consumption based pay as you go model I actually think it’s a good option.

Just had a quick peak at their pricing page. You’re looking at ~50 dollars per month for pure storage costs (could be even less due to compression when moving to columnar storage). The rest is compute, so that’s hard to estimate, but I don’t think your use cases will be too heavy. I could imagine you end up with a bill of 100-200 dollars per month.

For that amount of money, you should save yourself the troubles of self-hosting an open source DWH. Yes, you could end up with a cheaper bill. But you should keep the time you need to invest in deployment, maintenance, security etc into account as well.

You could load the three topics into Flink SQL tables using the Kafka connector and create a fourth table for the flattened results using the upsert-kafka connector. You can specify the output topic and JSON format in there as well.

You can then run an INSERT statement to query for the flattened results and store them to your output table/topic.

It depends on quite a few different aspects, such as the type of analytical questions you want to answer, if you need preprocessing of the data, where all of this will be running (cloud provider, k8s, etc) and the experience of you and your team.

Generally speaking:

If generic preprocessing of the data is beneficial, I would go with Flink. It depends a bit on the type of processing whether I would prefer the SQL/Table API or the DataStream API. I have worked with Flink, Spark Structured Streaming and a little bit of ksqlDB. In my experience Flink was the nicest to work with from a developer experience, it is stable and can achieve the low latency you mentioned. The k8s operator works very well for self-managed deployment, but you could also look into managed offerings like Ververica or Confluent. Keep in mind I primarily use the DataStream API with Java. I'm not sure how mature the Python API's are.

Spark Structured Streaming has been very resource intensive when I worked with it. Latency wasn't great. I also experienced stability issues when my processing required state. Plus side of Spark is that it has an excellent Python API.

I'm not so sure about kslDB nowadays. To me, it always felt like Confluent's alternative to Flink. Nowadays, Confluent fully promotes the use of Flink themselves (it's even mentioned on their page about ksql). But perhaps I'm wrong here :)

After processing (or if processing is not required), I would load the data into an OLAP store. Which one will depend on the options that are available to you. I've seen Apache Druid, Clickhouse and Apache Pinot being mentioned. You could add Materialize to that list as well.

You could create a single spark structured streaming application for all topics by using a wildcard in your topic subscription. The raw data would then be stored to a single bronze table, partitioned by topic key. You could then branch out to the specific flattening logic per topic within that same application and write to separate silver tables.

Wouldn’t recommend this for complex and very large volume applications, but for your scenario it sounds like a simpler and more efficient approach with less overhead.

Databricks indeed launched the Free Edition, which would be an easy way to get going. Another easy way to try out PySpark interactively is to use the Jupyter Docker Stack: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-pyspark-notebook

• ⁠Country: the Netherlands
• ⁠Position (e.g. DE, Analytics Engineer, BI Engineer): Senior DE at a consultancy firm
• ⁠Primary tech stack: Java, Apache Flink, Apache Kafka, AWS, Kubernetes, Python, Databricks, Terraform, SQL
• ⁠Experience (years): 8 years
• ⁠Education: irrelevant bachelor
• ⁠Salary yearly (before taxes in USD): 97k