u/unplannedmaintenance

I have experience with Cloud Functions and local development is indeed not a very smooth experience. I'd recommend to switch to Cloud Run (Cloud Functions is also being transitioned to Cloud Run). Yes it will require a bit of upskill as you need to learn how to create Docker images. But you only need the basics, half a day to a day should be enough to get up and running.

Why don't you just use an older version of Pandas?

Yes, I resolved it by switching to Avro (see comment above).

Why would you want to use a load balancer for a bucket?

It's not about non-compete, it's about copyright infringement/stealing code.

I notice this as well, but in the end I don't give a shit. I need some services, this company is offering it, they look legit -> buy.

Wondering the same thing

When feeding it to an LLM, for example.

Ask Arnout if what they are doing is allowed, he'll probably write a blog about it: https://blog.iusmentis.com/

None of these points are even remotely important for me, or for a lot of other people.

You can also set a limit per query, and billing alerts when the cost exceeds a certain treshold.

I'm always a bit surprised when reading about how many people (mainly) use the web console. I work with BigQuery on a daily basis and I always use my IDE (DataGrip). Although it's not feature complete for BigQuery, but it doesn't miss anything major (for me). The only things I use the web console for are checking amount of bytes processed, and getting more detailed error messages (sometimes the JDBC driver simply returns "API Error", but the console gives more info in those cases).

I'd be scared to death honestly, trusting a web interface to maintain the integrity of what you're working on.

Currently developing a solution using (managed) Qdrant + Haystack, which has good integration and supports hybrid search. You can use your own embedding models as well, it's agnostic in that regard.

I haven't tried PG, but from most things I've read it's at least a factor 5 slower than dedicated vector DB's, and doesn't scale well.

edit, there's a nice example here: https://medium.com/@nickprock/qdrant-hybrid-search-under-the-hood-using-haystack-355841225ac6

Thanks for the clarification, I see now that they also added some clarifying text here: https://cloud.google.com/vertex-ai/generative-ai/docs/rag-overview

No. This happened when I was testing the experimental models for long-form audio transcription. The regular models (Flash and Pro 1.5) worked pretty reliably, but the experimental ones would more often deviate from the instructions, even with the temperature set to very low or to zero. I thought it would be a cheap/free way to transcribe a massive amount of audio files, but I gave up on it.

You should develop your own scraper and not rely on Apify. Apify is for people who can't or don't want to code. For example when you're creating a quick PoC or something. If you need to do high volume scraping just write a script an deploy it somewhere. Either to an on prem server, or on a cloud platform (using a VM or FaaS).

But I can already tell you that there will be no one size fits all solution. If the site uses Javascript to render/generate links and there's no sitemap, you're already in for some trouble.

Because they are experimental. I've gotten quite strange and inconsistent results with the experimental models (flash and pro).

Agreed. It should be tab, or right arrow.

I agree with this.

Another option would be to just append the rows to your table and create a view on top of that only selects the latest version of all the rows, based on an insert timestamp, for example. You do need one or more fields to uniquely identify rows, though. But since you're doing updates, you're probably set.

You can do it very easily like this:

```

select *

from your_dataset.your_table

qualify row_number() over (partition by some_unique_key_column order by insert_timestamp desc) = 1

```

https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token

The most expensive model is $2.50 per 1 million input tokens. And the maximum output tokens is 8192. So I'm wondering how you get to $10-20 per query?

Yes. You can either add the different projects as different connections, or you can a comma-separated list to the 'AdditionalProjects' parameter in the Advanced tab of the connection. Here is an example: https://i.sstatic.net/7zxH6.png

It's not in the Jetbrains docs, but in the Simba docs: https://storage.googleapis.com/simba-bq-release/jdbc/Simba%20Google%20BigQuery%20JDBC%20Connector%20Install%20and%20Configuration%20Guide_1.6.1.1002.pdf

I'd be very happy if Jetbrains and Simba would put a bit more effort into BigQuery compatibility. At least Simba does not seem to take it that seriously.

Next time try reading the docs first?

Good initiative. For those who didn't know yet: Jetbrains DataGrip also has support for BigQuery, which works pretty well, generally. The intellisense is also okay (not great for complex fields and more uncommon constructs). The only big thing I miss is cost evaluation.

According to EyeLevel research EyeLevel is the best?