u/RecordingForward2690

I had trouble transferring some domains from other registrars to AWS a while ago. The error messages are very unhelpful. In the end I decided to let them "hop" through my private Versio account, so that I could call the helpdesk and get things sorted via them. I then transferred them to AWS without issue. That was 80 dollars well spent, as I'd been busting my head on that issue for hours already - and I am billed by the hour.

I would not be so sure about that. As far as the bank is concerned, you entered into a legit agreement with AWS, and that agreement is still in place. So the AWS charge is legit. There's no fraud or anything, so if I were the bank I would not touch it. The best you can do, probably, is to instruct the bank to block all future charges, and hope the bank goes along with that.

But there's a big caveat here: I'm pretty sure AWS keeps track of non-paying customers, somehow. If you start blocking/withholding payments for legitimate bills, then you might be ending up on that list, and that could cause problems in the future, if ever you need to setup a new AWS account.

Have you tried going back to the college, explaining the situation and asking them if they can re-instate your email address for a few days (or temporarily setup an email forward to your current address), so you can sort this out? It can't be the first time this has happened: We started using email addresses as our identification tokens (instead of a free-format username) decades ago, so losing access to an email address means losing access to a whole bunch of other stuff. So maybe the college already has procedures in place to give former students a forwarder rule of some sort for a while.

We are running a complex network with a TGW, DX to on-prem and a hundred or so attached VPCs. DNS needs to be transparent across AWS and on-prem, so we have a dedicated VPC with inbound and outbound Route53 endpoints. This VPC happens to be 10.0.0.0/22 - the reason for that particular CIDR block is lost in the mist of time.

When teams need to build something in AWS, we give them broad access to their own account(s) - we just pre-provision those accounts with some resources, including a TGW-connected VPC. If nothing indicates that more IP addresses are needed, those VPCs are provisioned as /24 by default.

A team decided that that /24 VPC that they were given was not big enough so they unilaterally decided to add an additional IP range to their VPC. They picked the first IP range they could think of - 10.0.0.0/24. That was duly propagated into the TGW routing tables. And since it was more specific than 10.0.0.0/22, all inbound DNS traffic now ended up in the wrong VPC.

That effectively halted all communication from on-prem to AWS: Dozens of workloads, literally thousands of customers impacted.

Obviously the whole AWS team was scrambling to figure out what had happened. I was the one who eventually found out what the cause of the outage was. A year on, the PM of a different project that happened to be sitting across from me still reminds me sometimes that he had never, ever, heard so many different profanities come out of my mouth in such a rapid succession.

We now have an SCP that prevents the addition of CIDR blocks to VPCs.

We have a slightly smaller organisation with just about 90 accounts and maybe around 200 or so WAF-protected resources. When we implemented Firewall Manager, we first made the decision to keep it simple: Just two WAF definitions: Private and Public. With the notion that if an application really needed something custom, they could always do their own WAF but were then also responsible for it.

Public WAF is the following rules: IP whitelist (for testing/webscraping that should not be impacted by rate limits); Rate limits; OWASP top-10, Geoblocking, just-in-case-but-empty-for-now Blacklist, default Allow. This WAF is obviously used for public resources.

Private WAF is the following rules: Manual IP whitelist, Self-service IP whitelist (*), Secret Header, default Deny. We use this WAF for resources that should be internal-only (think internal documentation) and for resources that are still in the Dev/Test/Accept stages.

Our implementation is tag-based, where only resources tagged with a WAFPolicy tag get the appropriate WAF. This allows developers to implement exceptions. It also allowed a gradual switch-over. Only once the migration was nearly complete, did we implement an org-wide policy that resources are supposed to have a WAFPolicy tag (even if it's set to None). This is enforced with AWS Config. If no tag is set and no other WAF policy is associated, the Config rule sets the tag automatically.

Initially we thought that this approach would be too simple and not offer everything we thought we needed. But at the end of the day we realised that - apart from DoS/DoW attacks - your WAF is always your *secondary* line of defence, not the primary. Apps with sensitive content should be behind an authentication layer. Apps should do input checking so XSS/SQL injection attacks are not possible. Etc. So even with this relatively simple approach we still find we have everything we need. Oh, and we use standard AWS rulesets, no Marketplace stuff.

(*) The self-service whitelist is intended for developers who work from home, need to access resources, but have a daily-changing IP address, for instance because they use a 5G Wifi router. So we rigged an authenticated portal (Cognito to Entra ID) where they can enter their IP address-of-the-day (autodetected). This is then added to the whitelist, with a lifetime of something like 24 hours.

If you are new to the cloud and your intention is to learn as much as you can, do not start with "wrapper" layers such as Elastic Beanstalk, Amplify, Serverless and others. These layers are great to build application environments quickly, but will not teach you how the individual components work. In fact (looking at you, Amplify) they may go out of their way to hide components from you.

Instead, first learn how the individual components work (EC2, RDS, ...), then learn how they work together and how to do scaling (ALB, ASG) and then learn how you manage them as a single solution (CloudFormation, CDK). Only then consider whether a wrapper layer such as Beanstalk makes sense for you.

Personally I have found that these wrapper layers do not expose all functionality of the underlying components, some of which are critical to my needs/environments. So I don't use them.

The most recent frustration was that AWS Firewall Manager can simply apply a WAF to any CloudFront distribution that has a certain tag. Great. But this doesn't work when the CloudFront distribution is managed by Amplify - since Amplify does not allow you to add tags to the resources it creates.

Not necessary lift-and-shift. That's a term that applies when doing migrations from on-prem to AWS. But it could also be completely new applications.

We recently did a project with a software vendor that delivered a new (for us) solution. Their software needs to run in a Windows environment and given the size of the workload, required no less than ~65 EC2 instances. We would have loved to use a more modern design (containers, Lambda) for that, but the vendors software was simply not capable of running in those environments.

Their software application is not a decades-old monolith BTW, but it's dozens of relatively small services running on Microsoft Service Fabric.

Also bring a comfortable backpack and each morning load it up with the stuff you need to get you through the day. Laptop, phone, headphones, charger, power bank, water bottle, sunglasses, sweater, whatever. Don't count on getting back to your hotel room until well after dinner.

Re:Invent is a multi-venue affair and getting from one venue to another can easily take half an hour to an hour. Plan your day so that you can stay at a single venue for the full morning or the full afternoon, or even the whole day. Otherwise you're just wasting time.

Make a shortlist of sessions you want to attend. As soon as the session registration opens, register for the sessions you want to go to. Things fill up fast and the standby lines are not always getting you in. Some sessions have an overflow opportunity, sometimes in a different venue. Use that if you don't want to miss out on sessions. Also plan for an hour or two at the expo.

If you want to attend the keynotes live, be there very early. That means standing in the line at least an hour before the room opens - which could be another hour before the session starts.

If you've never been to Vegas, give yourself half a day off just to walk the Strip. It's a unique place, both in the positive and in the negative sense, and you don't want to miss out on the experience.

This is not so much an AI mistake, but rather the default setting of AWS. When you don't explicitly set a DeletionPolicy for a resource, the DeletionPolicy = Delete, *except for certain RDS resources*, where the default is Snapshot.

So even if you were to review the AI generated stuff (which you should), there probably would not have been an explicit DeletionPolicy in the templates and you would not have caught this, unless you know this.

https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-attribute-deletionpolicy.html

If Microsoft supports multiple levels of OU, and the regular AWS API calls support it, then so should the SSM document. If it doesn't, you should file a bug report.

The regex really should be something like:

^$|^OU=[a-zA-Z0-9]+(,OU=[a-zA-Z0-9]+)*(,DC=[a-zA-Z0-9]+)+$

(and even then I'm not sure if that one is entirely correct)

Having said that, we're not using the pre-canned SSM document for this. We created a custom domain join SSM document that first changes the computer name into the right one, then performs a reboot, then joins to the domain. This saves a change of computer name after the join, which is an issue in our environment due to various automations. Input for this SSM document is taken from a variety of sources: SSM Parameter Store, Secrets Manager and the tags of the EC2 instance. You may need to adapt this to your own environment.

Our full SSM doc is like this (company/app name pruned):

  JoinDomain: 
    Type: "AWS::SSM::Document"
    Condition: Deploy
    Properties:
      DocumentType: Command
      Name: !Sub "${AppName}-Join-Domain"
      UpdateMethod: NewVersion
      Content:
        schemaVersion: '2.2'
        description: |+
          *Join domain*  

          This document performs a Domain Join to the domain, for ${AppName} servers. It also sets the hostname to the right value, if needed, based on the EC2 "ADName" tag.
          After the domain join and the subsequent reboot, it also adds various AD admin groups to the local Administrators group. It will perform
          intermediate reboots as and when required.

          Prerequisites: The EC2 instance profile needs to have access to the Domain Join secret in Secrets Manager. This access needs to be revoked afterwards.
          Furthermore, the right parameters need to have been configured in the Parameter Store.
        mainSteps:
          - name: JoinDomain
            precondition:
              StringEquals:
                - platformType
                - Windows
            action: aws:runPowerShellScript
            inputs:
              timeoutSeconds: '120'
              runCommand: 
              - |
                # We need to make various IMDSv2 calls. For this we need to get a token
                Write-Host "Retrieving IMDS data"
                $token = Invoke-RestMethod -Method Put -Uri http://169.254.169.254/latest/api/token -Headers @{ "X-aws-ec2-metadata-token-ttl-seconds" = "3600" }
                $region = Invoke-RestMethod -Method Get -Uri http://169.254.169.254/latest/meta-data/placement/availability-zone -Headers @{ "X-aws-ec2-metadata-token" = $token } | %{ $_.Substring(0, $_.Length - 1) }
                $instanceID = Invoke-RestMethod -Method Get -Uri "http://169.254.169.254/latest/meta-data/instance-id" -Headers @{ "X-aws-ec2-metadata-token" = $token }
                Write-Host "Got IMDSv2 Token: $token Region: $region InstanceId: $instanceID"


                # Get other domain join info
                Write-Host "Getting ADName, Target OU and Target Domain"
                $ADName = aws ec2 describe-tags --filters "Name=key,Values=ADName" "Name=resource-id,Values=$instanceId" --query "Tags[*].Value" --output text
                $InstanceName = aws ec2 describe-tags --filters "Name=key,Values=Name" "Name=resource-id,Values=$instanceId" --query "Tags[*].Value" --output text
                $targetOU = Get-SSMParameter -Name "${AppName}-DomainOU" -Select Parameter.Value
                $targetDomain = Get-SSMParameter -Name "${AppName}-DomainName" -Select Parameter.Value

                # Rename machine if needed
                $currentName = $env:COMPUTERNAME
                if ( $currentName -ne $ADName ) {
                  Write-Host "Renaming computer to $ADName"
                  Rename-Computer -NewName $ADName -Force -Confirm:$False

                  # The below forces a Windows reboot then restarts this script. See https://docs.aws.amazon.com/systems-manager/latest/userguide/send-commands-reboot.html
                  Exit 3010
                } else {
                  Write-Host "This system already has the right name: ADName"
                }

                # Perform domain join
                if ((gwmi win32_computersystem).partofdomain -eq $false ) {
                  # Get domain join credentials
                  Write-Host "Getting join credentials"
                  $domainJoinSecretArn = Get-SSMParameter -Name "${AppName}-DomainJoinSecretArn" -Select Parameter.Value
                  $domainJoinSecret = Get-SECSecretValue -SecretId $domainJoinSecretArn -Select SecretString | ConvertFrom-Json
                  $domainJoinUserName = "${AppName}\" + $domainJoinSecret.CUSTOMER_MANAGED_ACTIVE_DIRECTORY_USERNAME
                  $domainJoinPassword = $domainJoinSecret.CUSTOMER_MANAGED_ACTIVE_DIRECTORY_PASSWORD | ConvertTo-SecureString -AsPlainText -Force
                  $domainJoinCredential = New-Object System.Management.Automation.PSCredential ($domainJoinUserName, $domainJoinPassword)

                  Write-Host "Going to join to domain $targetDomain as $ADName with OUPath $targetOU"
                  Add-Computer -DomainName $targetDomain -Credential $domainJoinCredential -OUPath $targetOU -Passthru -Verbose -Force -Confirm:$False -Options accountcreate

                  Exit 3010
                } else {
                  $currentDomain = (gwmi win32_computersystem).Domain
                  Write-Host "This system is already joined to the domain $currentDomain"
                }

                # Configure AD Administrative access for the G-RG-AWS-ADMIN AD Group
                $ADGroup = "${AppName}\G-RG-AWS-ADMIN"
                $LocalGroup = "Administrators"
                $user = Get-LocalGroupMember -Group $LocalGroup -Member $ADGroup
                if( !$user )
                {
                  Write-Host "Adding the AD group $ADGroup to local group $LocalGroup"
                  Add-LocalGroupMember -Group $LocalGroup -Member $ADGroup       
                } else {
                  Write-Host "$ADGroup is already member of $LocalGroup"
                }

Given your requirements, I don't think S3 on its own can cut it. You'll need some sort of DB (relational or NoSQL) that holds your business rules and the index to your data, and some logic that can generate pre-signed URLs into S3 for any legitimate request.

Trying to meet all these requirements, using native S3 features only might, just might, work today, but what if you get even one additional requirement?

For resiliency, what distance is "good enough"? Does two AZs 30-50 km apart really offer less real-world resiliency than London-Cardiff, which is 200 km apart?

What I do know is that distance increases latency linearly, and that slows down synchronous commits to my hot-standby database. And I don't know about Azure, but in AWS cross-region traffic is more expensive (in $$$ terms) than cross-AZ traffic.

I'm not saying that your customers should have gone to AWS or Azure. I'm sure the discussions were highly technical and detailed, taking all those factors (and probably a lot more - the Region/AZ model is not the only differentiator between them) into account. All I'm saying is that more regions is not necessarily better.

Azure has a different region/AZ model compared to AWS. In Azure in the UK, only the UK South (London) region has three AZs. The UK West (Cardiff) Region is a non-AZ region. And there are plenty more non-AZ regions like that around the globe.

Due to this, historically, Azure placed a lot more emphasis on Multi-Region resiliency, while AWS focused on Single-Region, Multi-AZ resiliency.

To me, AWS' model of Regions and AZs is more consistent and predictable, and therefore easier to design for and operate in.

Even things hidden deep in services like SQS and DDB do things like "one in three" or "two in three" because it automatically gives non-quorum and quorum results. For instance the difference between an eventually and a strongly consistent read. With a fourth AZ all this becomes harder. Not just to implement, but also to understand how the implementation works, especially in the light of failures.

"...as I am configuring my EC2 instance..." You're on the wrong track for a startup about to launch an MVP.

Try (really try) if serverless (S3, CloudFront, API Gateway, Lambda, DynamoDB, Serverless RDS, ...) works for you first. Not only are you building a much more scalable, secure and resilient solution that way, but it will also stretch your $100 much further. And will require less maintenance in the long run.

If serverless doesn't work, try a containerized solution (ECS/EKS/Fargate). Only when that doesn't work either, consider an EC2 based solution. And then don't configure your EC2 instance, but ensure your EC2 instance configures itself (or build an AMI through the Image Builder). Then add a Load Balancer, Auto Scaling group, CloudWatch Metrics/Alarms, ACM, Route53 and other pieces into the mix to complete the solution.

We create an Environment.json file that contains global and account-specific settings. We also created an Environment class (that lives in our generic tools library) that's imported in every project and pulls this file in. When referring to Environment('bla') it pulls the most specific setting from the file, so an account-specific setting overrides the global setting.

Focusing beyond the money/credits here.

What are you going to build that requires 100-150 dollars per month with no users to speak of? What does your architecture look like? Did an actual AWS architect review that design? How did you arrive at that cost estimate? Did you consider any alternatives?

Any startup in AWS that I know of, started out with serverless architectures and the beauty of that is that the cost will be near-zero (*) until your users start coming in. And by then those users should also bring in money, directly or indirectly (think advertising here) to offset the increase in cost.

(*) There will be some cost for permanent storage, domain registration and a few others, but I'd be surprised if those combined would be more than 10-15 dollars per month.

You forgot to mention the Custom Authorizer on the API Gateway, which allows you to do even more exotic things.

IAM Access Analyzer and similar tools: Sounds great in theory, doesn't reall work in practice.

First issue is "exception" code paths. Something goes terribly wrong, a dump of some sort is created in an S3 bucket. That type of code is virtually impossible to test, especially if it is only triggered by corner-case behaviour of external API calls. So it won't be in your CloudTrail.

Second issue is parameterization. In the dev environment my bucket will be called something-dev-something, and that's what the tool will put in the policy. But when I then move to prod, the bucket will be something-prod-something and that's not in the auto generated policy. I need to make sure that my policies are properly parameterized so that the policy uses !Sub "arn:aws:s3:::something-${env}-something" in the policy.

Third issue is that CloudTrail in Dev also contains the traces of when I was still building my code, and maybe abandoning certain code paths in favour of something else, leading to a different set of API calls being made.

There is a case for CloudTrail though: Indirect permissions. For instance, you need s3:ListBucket in addition to s3:GetObject if you want a meaningful 403/404 error distinction. Or, for resources that rely on KMS encryption with a Customer Managed Key, you need kms:Decrypt and a few others in addition to your s3:GetObject. Those indirect permissions are a nightmare to figure out. Looking at CloudTrail is often the easiest - but it does assume you are trailing the data plane.

I've read a few of the other responses, and while good, there's a little thing you need to know about a WAF in general before you make your decision. WAF is not just straightforward IP filtering, but it also does things like detecting SQL injection attacks, XSS, OWASP top-10 and so forth. For this, the WAF needs to have access to the *unencrypted* traffic.

The internet today is all about end-to-end encryption. So you cannot simply put a WAF "in front of your existing solution" and expect that it works without further ado. It doesn't work that way. In order for a WAF to work, the encryption needs to terminate at the WAF itself. Or rather, the way AWS implements it, is that the WAF product is not standalone but is integrated in the products where the encryption termination happens anyway: CloudFront, API Gateway and a few others. This also means that that particular location is where your public certificates and the associated private key need to live. (And if you use non-AWS issued certificates, this is also where your cert rotation needs to take place.)

And that's the recommendation you read in most of the posts: Use CloudFront + WAF. CloudFront is where your certs+private key are going to live, and thus where the encryption termination happens. The unencrypted traffic is fed to the WAF for checking/rate limiting and whatnot, and once it passes the WAF the traffic is forward to the origin - your existing server.

Once the traffic is past the WAF you can re-encrypt it for the path to the origin - essentially a requirement if your origin lives outside of AWS. This means that the origin also needs to have a cert+privkey deployed, but that's probably already the case. And you need to come up with a way that ensures that CloudFront+WAF cannot be bypassed.

Another thing to keep in mind in investigating these types of issues is that it could also be DNS.

If your pings, based on IP address, give you sub-second response times then the routing path is not the issue. But you've got to look very carefully about how hostnames are resolved. Which DNS server(s) are pushed to the client upon VPN configuration? Do the clients use these exclusively, or are they merged with the set that the client already has, or do clients have overrides in place? Or maybe, even fancier, do clients have routing rules for DNS domains/servers? So does the client end up with a list of DNS servers that's pure-internal, pure-external or a mix of both? Do you have a split-DNS setup of some sort where an internal server could give a different response to a query than an external server for the same domain?

And we've also had to deal with a corner case issue where the Active Directory registration of a Windows system was too late and cached, and a particular server implemented a whitelist of allowed Windows systems that could connect to it. If you connected the VPN you had to manually perform an ipconfig /registerdns to make sure your system was properly registered in AD, and only then connect to the server. This was a specific corner case where "it didn't work". But if you have a server that (maybe for logging purposes) performs a reverse DNS lookup and the VPN client IP address cannot be looked up, then this can lead to a large timeout as well.

Fully agree with the above, in the sense that AWS networking is completely different from on-prem networking. For years I have taught both the basic and the advanced architecting course, and I always warned my students at the beginning of the networking chapter, that those with extensive on-prem networking experience would have a harder time following my story than those without. With extensive on-prem networking experience you are trying to map AWS technology onto your existing knowledge, and it simply doesn't work that way.

If you want to start your conversion from extensive on-prem experience to AWS, then this video is the one I always recommend: https://www.youtube.com/watch?v=Zd5hsL-JNY4 It's 12 years old and only talks about the basics of how VPC tech is actually implemented, but it really clarifies a lot of architectural choices that AWS made, for somebody who really knows networking. From this you can then extend your knowledge to newer AWS tech, such as the Transit Gateway.

The only real place where your extensive on-prem experience can still be a major plus, is in setting up complex Direct Connect configurations. That's where the AWS abstractions hit the fiber optic cable, so all of a sudden you're dealing with physical characteristics like light levels again. Extensive experience with BGP4 also helps tremendously in this area, in particular when you need to setup Public VIFs.

This. And add a WAF with, at least, rate limiting.

True. We forwarded that info to the developers as well. However, our setup is significantly more complex, including our email bounce response, that we're not sure we can use this out of the box. We're looking at solutions such as mailosaur to implement in our DTA environments.

We had this a few weeks ago. We have a prod SES setup that sends tens of thousands of emails per day, with a bounce and complaint rates well under 0.1%. But we also have a test environment that only sends 10 mails per day to known email addresses, one of which bounces. This one is specifically to test our bounce handling setup (auto adding to blacklists etc.)
Due to statistical variations our bounce rate was 9.99% and jumped to 10.01% or something, which triggered an AWS response. We explained the situation to them and also upped the amount of legitimate mails to 20, and that brought things in-line. AWS was happy.

In any case, what we found is that the first email that AWS sends you is just a warning, with a notification that if you don't fix this within the next 130K emails, they would block our access. So there's quite a bit of leeway. I don't know if the 130K is a fixed number or based on historical sends though.

And you'd better start planning this NOW for Node.js 12.x. All Node.js runtimes up to 16.x had the AWS-SDK v2 in it by default. Starting with Node.js 18.x, AWS shipped the AWS-SDK v3 in the runtime. So an upgrade from 16.x to 18.x requires either a layer or something where you manually bring in the AWS-SDK v2, or significant code changes: SDK v2 and SDK v3 are really, absolutely, completely different. Yes, there are tools that help you make the transition but it's not completely seamless.

This is a project that you need to plan. If you ignore it and all of a sudden find yourself in the situation where you do need to change your Node.js code in a hurry, it's not going to be a fun day.

For most other runtimes upgrades, including Node.js 18x. to the current runtimes, and from Python 3.6 to the current runtimes, 95% of your code will simply run without any changes in the new runtime. For us, a quick syntax and sanity check is usually enough to deploy to prod with confidence.