Another year, another AWS re:Invent has come and gone. I’ve been following the announcements closely, and there are some genuinely interesting developments worth discussing. The Big Picture This year’s re:Invent felt a bit different. The pre:Invent announcements started later than usual (mid-November instead of early October), and the keynote felt more focused on GenAI than infrastructure improvements. That said, there are still plenty of practical enhancements that can make our lives easier....

It’s 2024, and AWS finally built a browser-based S3 viewer. Twenty years after S3 launched, you can now browse your buckets directly in the browser. It’s still in alpha, but hey, better late than never, right? You might be thinking - wait, weren’t there options before? Like S3Fox? Yeah, there were some third-party tools, but AWS itself never had an official browser interface. So what took so long? Real Problem: Access Control Turns out, building a simple file browser wasn’t the hard part....

Disclaimer: here you might not find something new if you know 12 factors app. The 12-factor app methodology 12factor.net, is a set of principles designed to enable applications to be built with portability and resilience when deployed to the web. These principles focus on declarative formats for automation, clean contracts with the operating system, and suitability for deployment on modern cloud platforms, thus minimizing divergence between development and production, enabling continuous deployment for maximum agility....

Recent elasticsearch licensing change ensures that the Beats modules are sending data to an officially supported versions of Elasticsearch and Kibana where Elastic can attest to the quality and scale of the products. Does AWS have any plans to fork a version filebeat? https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-7.13.html https://www.reddit.com/r/aws/comments/nn95aq/elastic_has_broken_filebeat_as_of_713_it_no/ What are the alternatives? Host elasticsearch on EC2 instances, why not? CloudWatch, slow… https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html Kubernetes options like ECK or helm chart deployments…tricky for production usage Loki?...

Reasons to upgrade to k8s 1.30 Container resource based autoscaling Container resource based autoscaling is now promoted to stable https://github.com/kubernetes/enhancements/issues/1610 Horizontal Pod Autoscaler examines the total resource usage of the entire pod i.e. sum of all containers and scale pods based on average CPU or memory usage. Container resource based autoscaling feature allows HPA to scale workloads based on the resource usage of individual containers within a pod, instead of the aggregated usage of all containers in the pod...

Kubernetes InPlacePodVerticalScaling feature Kubernetes v1.27 introduces InPlacePodVerticalScaling, allowing seamless pod resource resizing without restarts Enhanced Continuity: Eliminates the downtime and potential data loss caused by pod restart Cost Savings: Avoid overprovisioning and optimizing resource usage. InPlacePodVerticalScaling lets you allocate resources precisely as needed In this example for pod memory resources configuration, the resizePolicy indicates that changes to the memory allocation require a restart of the container, and for CPU resources the restart is not necessary during resizing....

Collaboration Use remote state and state locking For certain backends like AWS S3, enable versioning to make it easier to recover your state if needed Agree on naming convention Use meaningful tags to easily identify resources: environment, owner, project keys are must You can also add cloud-custdodian for components which are out of terrarfom/IaC tools, which could automatically tag your manually created resources with Owner Creator based on CloudTrail events Don’t reinvent the wheel Use existing shared and community modules....

Userdata is compatible with the standard AWS EKS Terraform module, with the sole recommendation being the utilization of a custom AMI. In order to use instance-store you also need to install local-static-provisioner - https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner Terraform example: eks-dev-instance-store = { instance_types = ["r6id.large"] min_size = 1 max_size = 3 desired_size = 1 block_device_mappings = {# Root volume xvda = { device_name = "/dev/xvda" ebs = { volume_size = 24 volume_type = "gp3" iops = 3000 encrypted = false delete_on_termination = true } } } ami_id = data....

There are numerous solutions for accessing private RDS instances, many of which require thoughtful design. The solution I use sometimes is straightforward: I deploy it as a Helm chart within a k8s cluster. In this setup, access to the RDS is contingent on having access to the k8s cluster with the appropriate RBAC configurations. While it may not be perfect, it’s secure, quick to implement, and requires minimal maintenance. The following command demonstrates the basic principle:...

After upgrading Kubernetes (k8s), you might encounter errors such as no matches for kind "Deployment" in version "apps/v1beta1". These errors typically indicate that certain resources have become deprecated. To resolve these issues without the need to delete your Helm chart, you can follow this simple solution: helm plugin install https://github.com/helm/helm-mapkubeapis helm mapkubeapis <releasename> helm upgrade <releasename> It’s important to note that you may still need to update your Helm chart templates, especially if there have been structural changes between versions....