u/RocketOneMan

This is still getting updated, https://github.com/awslabs/aws-solutions-constructs, which is surprising to me I thought it was deprecated.

I wish it was at least usable as an example but there's too many layers of indirection that it's hard to easily read. Like there's no 'real' code in here https://github.com/awslabs/aws-solutions-constructs/blob/b33dfb66df4fdc985f35c9acb4a8362be461ac30/source/patterns/%40aws-solutions-constructs/aws-alb-fargate/lib/index.ts

For whoever downvoted your comment, yes there are ways to get out of it and escape hatches and stuff but those are like the opposite end of the spectrum from the 'ease' that the L3's are designed to provide you. If you knew how to use the logical id overrides or manually setting properties on the generated L1's you're likely not the target audience for the L3's to begin with.

AWS has a service called Lightsail which offers what you're probably used to from a provider offering VPS's. But AWS also has 200, maybe 300 at this point, other services for other uses. Database, queues, load balancing, container orchestration, streams, metrics, alarms, AI inference.

Given you're asking this question I wouldn't jump straight into EC2, a more let's say "feature rich" virtual machine service. AWS is a maze as far as billing is concerned and you might find yourself spending more money than you thought quick.

Should I consider myself blessed to have not needed to know this is a thing?

I haven't used SAM for a while but if memory serves me right you're writing the cloudformation templates themself with addition of some special types that make certain serverless resources "easier" (or harder depending on if the abstraction works for you or not). And it has its own CLI for deploying these.

CDK lets you define your AWS resources, all* types of resources, in a turing complete language like typescript (not writing the YAML/JSON directly). It's still very declarative where you say what you want not how you want it but you can use the usual functions, variables, loops etc to make certain things easier.

I don't think SAM has seen much growth since the CDK was released (I could be wrong). You'll probably find more examples online or if someone else were to join you they'd more likely be familiar with CDK. And since you can define all* resources that way, I would recommend it.

I would also give terraform a look. CDK is built on top of Cloud Formation which has odd quirks that you have to learn. If you don't know how cloud formation works already then debugging some problems you think are CDK related will just be that much harder. Terraform does its own thing to call the AWS APIs directly.

But if this is your first time with AWS in general, do some workshops, click around a bit. If you're worried about running up the bill because you're unfamiliar with things the only thing that IaC does to help with that is deleting everything is easier. But you can also just delete your account after a week of learning things if you want.

You call this a startup but it's your first time with AWS and maybe are implying it's just you wanting to deploy something in the cloud. That's fine and all but it can be more trouble than it's worth when you're just starting out. AWS also has Lightsail with some basic VPS/Container services with more predictable pricing than DIYing it in the rest of the services AWS has to offer.

Can tweak the garbage collector / heap settings. Idk if that’s cheating or not.

https://youtu.be/Gqo0oCfZSjg

The opensearch integration can help here

And it comes with WAF

Do you have a link to a video of this?

-m Print optimization decisions. Higher values or repetition produce more detail.

https://pkg.go.dev/cmd/compile

Cool!

Are you sure you haven't been hacked? What does cloudtrail say for those tables? Or AWS Config resource timeline. Rotate your iam user keys.

So that you know they're all working and ready to be switched to as the primary if needed. Also forces you to deal with the added latency always instead of finding that out the first time the passive region becomes active.

We have an active passive system with DynamoDB global tables that automatically rotates between which region is active throughout the day. Oncall didn't know that we can remove a region from the options if we wanted to so had some down time because of that when it rotated to us-east-1. The set of regions is in a global DynamoDB table. Not sure if they had trouble accessing it or not when they finally decided to do it.

We have another system that's active/active that worked without any issues out of us-west-2 during the incident but that's also its default region, users didn't have to change which endpoint to hit.

Hard to say if active/passive is better than nothing or not. You have to be able to coordinate the switch while things are broken, and do that correctly which is likely unpracticed, and the passive region has to be all setup and ready to go. If it's never the active region you may not know if it works at all, hence the first service i mentioned automatically rotating between them.

AWS docs talk a lot about control plane and data plane separation and AZ fault tolerance but then we have these issues where it's all broken.

You can get the SentTimestamp in the attributes on ReceiveMessage to help debug how long between put in the queue and made it to your code.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html#SQS-ReceiveMessage-request-MessageSystemAttributeNames

You don't mention your visibility timeout but if it were 10 seconds and you failed processing something without resetting its visibility timeout to zero, or maybe successfully processed but the delete message call failed, then it may be processed again around the 12 second mark.

You can also request ApproximateReceiveCount and log if it's every more than 1, or see if you ever log the same message id more than once.

There are some other nuances with the ApproximateAgeOfOldestMessage metric, like how it doesn't count poison pills / messages received several times, but I don't know if that fits your scenario. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html

Do you have MaximumBatchingWindowInSeconds set to something besides zero? Can you share your event source mapping configuration?

https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

I cannot be sure if this is the case for us-west-1, but in us-east-1, everyone’s 1a isn’t the same. The idea is if you’re going to launch something and just pick the first thing then it doesn’t disproportionally overload one AZ. The mapping is set when the account gets made. You can request multiple accounts get aligned if your workload depends on it. So my guess is some people’s us-west-1a maps to this third AZ, for example.

Interesting, I wonder why

I think this is what data translations / 'parameter mapping' is for with API Gateway, although I haven't done it myself, the example looks similar to your ask.

For HTTP:

See: Change the response from an integration

https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-parameter-mapping.html#http-api-mapping-response-parameters

For REST:

https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-override-request-response-examples.html

Do you have any experience with kinesis on demand scaling? Last I used it it was slow to react to changes in load but that was 2-3 years ago. I’m still unsure about the maybe ‘constant’ merging or splitting of shards and handling that right as a consumer seems more challenging than provisioned capacity.

If on demand works well then it’s more hands off like SNS is.

If your server serves concurrent API requests from a thread pool, the percent of threads (avg or max between your hosts) that are available to serve new requests. You may only be at 25% CPU but if you’re out of threads to serve new requests they’ll either get queued or rejected.

And your tile and grout. Will look like a new home.

database connection

Is it a relational database? Are you using RDS proxy? Not sure if this makes things faster or not or just more consistent.

+1 for brushing up on writing. I find reading good writing the best way to learn how to write better. Or at least be able to judge whether what you just wrote is good or not. Less is more. Always include data (but not too much, can have an appendix). In your onboarding you should be given your team’s OP1/OP2 and/or 3 year plan. Can also ask for the latest QBR (quarterly business review). These docs are not always the best in terms of writing but they should give you an idea of the writing style that is likely different than what you’re used to.

Some other general advice which you may not have heard already:

Try to get a mentor that is not in the same org as you. Someone you can get advice from that won’t be influenced by any prior knowledge they have about the situation you’re in on your team. Your manager can help set this up.

Keep a note of everything you’ve done every day or every week. Check back in on changes you’ve made and capture any impact they’ve had. This will make end of year reviews and promo docs much easier to write.

Take an interest in doing “boring” operational excellence work. Your team will appreciate it and it’s the best way to learn your team’s services. Each team’s oncall burden is different, some AWS teams can be rough. Doing things you’re not asked to but which will either fix small recurring bugs or make them easier to debug next time goes a long way.

I think it’s more likely done for marketing/business reasons than tech reasons. Smaller brands running Shopify sites do this too to build hype.

The reasons you listed aren’t invalid but maybe unlikely. Ideally want everything scaled up, configured and tested at the scale you expect it to run at during peak before the actual event so you can iron out any bugs or misconfigurations. Systems which horizontally scale are usually safer to do this last minute scaling than ones that vertically scale.

Online games sometimes have downtime to do major updates. I imagine this is to more easily ensure everyone is on the same version of the game and harder to do a rolling update during active or long running sessions. Can also care less about backward compatibility by doing this. I’m not sure these apply to e-commerce sites but every business is run differently.

If they’re making substantial changes to the site that they can’t just enable with feature flags or only to a subset of internal users then maybe they want to deploy and test things before going live.

What’s the ‘added security of secrets manager’ compared to a DynamoDB table with a customer managed kms key for encryption at rest? I would think if it’s implemented that way then there might be less quirks with the limits so maybe there’s something special about it? I imagine this is in the docs but the discussion could be interesting.

Are you proposing OP does this through LakeFormation or just on the IAM user they're using to access Athena?

I haven't had a good experience setting up LakeFormation, maybe you have?