The idea came from some of our most pressing problems like getting the EC2 instance name, finding whether a ssm parameter or lambda exists etc. If you’ve worked with AWS,  you may be aware that it’s nearly impossible to find if an ssm parameter/lambda exists, especially if you don’t remember the name of that parameter. At the same time, when finding the name of the EC2 instances (as we name all our instances with a private DNS), we often need the name of the machine to ssh.  

Using AWS console (slowest) or even aws command line (slower) to discover this information required a lot of time. In the past, we  would manage aliases to quickly gather information. This later got converted into a full-fledged tool, including local caching, thereby allowing us to quickly discover information for EC2 servers and Load balancers, IP addresses, SSM params, dns records, S3 buckets, Cloudfronts, Lambdas etc. 

Lets see below some use cases for this: 

• ali api: This command can simply help us list any EC2 host whose name includes api
  • Along with this we see some other information like private/public ip, instance id, running state, and instance type
• all api: can help us list any lambda that has the  name api 
• alp param: can help us list any ssm param that has param in its name
• allb <instance id> can list the load balancer along with its configuration 
• agd api.example.com: can help us learn the name of load balancers, EC2 instances, ports config, etc.
  • Think about a time if someone asks you to find out which load balancer is routing traffic to this Ec2 instance?

All of the above and many more such cases can simply consume too much time. Opening the AWS console itself can take 5-10 seconds wherein our team requires less than 5 seconds to find out any information required at hand. 

This is a lifesaver for Titan’s DevOps to quickly find various resource information without spending too much time in AWS GUI

For authentication and authorization, the tool relies on boto3, which is recommended to have read access to all the AWS resources (eg: ReadonlyAccess) to utilize most of the features.

Please note that confidential information like ssm parameters are not being stored to handle security issues.

We have options to reconfigure the shortcut commands according to our convenience This help us customize the shortcuts with our preference. 

How it works

The cli is a multi-threaded application written in Python. To install this(on Linux/mac), please ensure that you have an aws cli already configured with you and just run the below command.

curl -s https://raw.githubusercontent.com/talk-to/aws_shortcuts/master/awss.sh -o awss.sh && . ./awss.sh && rm awss.sh

The installation will automatically create a cron in your local machine which will frequently fetch all the latest AWS configurations for popular resources in multiple threads and caches in the local machine. Local caching avoids repeated AWS api calls and improves the response time.

Few commands supported by the tool

>> awss 

---------------------------------------------------------------------

List ec2 instances                  -	ali
List s3 buckets                     -	alb
List lambdas                        -	all
List ssm_parameters                 -	alp
Get ssm_parameters                  -	agp
List route53 domains                -	ald
Get route53 domains                 -	agd
List lb                             -	allb
List cloudfront distributions       -	alcf

List commands                       -	awss
Rename commands                     -	awss configure

Fetch latest data from AWS          -	awss update
Update project to latest version    -	awss upgrade


---------------------------------------------------------------------

To find out more examples and see the source code head over to Github https://github.com/talk-to/aws_shortcuts .

For the curious ones some details below for cron and caching directory

>> Crontab -l
0 */6 * * * /bin/bash -l  /<user-home>/.aws_shortcuts/aws_shortcuts/scripts/cron.sh


## Local cache Directory:  
## /<user-home>/.aws_shortcuts

The post How to increase productivity in AWS cloud operations by using shortcuts appeared first on Email Service Suite for Professionals and Businesses: Try Titan Email.

Team Devops @Titan

The DevOps team at Titan, apart from normal development operations,  has not only significantly reduced operational toil but also has significantly contributed to the reliability of Titan as a service provider. 

As it often goes, with the growth of the product and the team comes the growth of technical challenges. Some of the challenges the team faces are very common across organizations.  But others take some serious problem solving and teamwork to find solutions. Unlike other operation teams, our focus is not only on making the infrastructure resilient, but actively contributing to production-facing solutions that handle large-scale traffic. Let’s look at some of the work undertaken by the team. 

Work highlights from the team

All below work is specific to the DevOps team. The tools/services/automation mentioned are all maintained in-house and developed by Titan’s DevOps team

Programming/Scripting/Automation

• The team spends 40-60% of their time on programming. We have automated most repetitive tasks and we take pride in writing our own services.
  • For frontend, we use React/Jquery/Bootstrap etc. 
  • As a backend framework, we use Flask.
  • Our critical software services are written in Golang while scripting is done using Python/Lua.

• We have written custom milters, policy services ,ratelimiters and more internal extensions and services. 
• We have migrated TBs of data using an  internal tool, “IMF” (Internal Migration Framework)
• We value speed and efficiency—as a result, when our infrastructure started to grow, one of our team members wrote a small tool called aws_shortcuts, which quickly became a crucial part of our day-to-day operations.  (Stay tuned for more info on this in the near future!)
• The team has also modified a few open source software to be used internally to suit our needs with correct license attributions.

Deployments and Migrations

• Naah! At Titan, DevOps doesn’t do deployments.
• Our deployment pipelines are enabled in such a way that gives our Developers the  power to deploy 24×7 without the help of DevOps. Similar is the case with database migrations.
• We have enough to say about this topic to create another blog post—so we will! Stay tuned.

Dashboards

• Be it an analytics dashboard, managing access to servers and applications, or finding important information, we wrote all of the dashboard. A few examples are:
  • A dashboard for support teams so they can find a needle in the haystack and quickly resolve any issues
  • Abuse management for email infrastructure
  • Add developers’s access to 50 servers? Our automation got it covered without the team’s involvement.

Abuse management

• Abuse, or fighting spam, is one of the most complicated problems in the email industry.
• Our DevOps team has built a lot of solutions and automation around this issue. For example:
  • We automatically detect, analyze, and block abusers based on their usage patterns.  
• We built a system to block phishing and reduce bounces.
• We created dashboards for the abuse team to get better visibility into spam complaints and the customer usage patterns to aid in  better decision making.
• We built tools around managing Titan’s Outgoing IP management.
• We proactively detect spam accounts before they cause any damage. 

Monitoring and On-call management

Monitoring and on-call management are some of the most important aspects for any operations team, and we have this covered as well. 

• We use Prometheus for monitoring ,Consul for service discovery and Victoriametrics as a time series database, Grafana and Kibana for visualizations
• We are also using ELK and Loki as our logging solutions at appropriate places.
• We have written an in-house tool called “Sentinel” which notifies respective Dev/Devops based on the role via SMS/call (This is similar to what we know as PagerDuty today) 
• We have also written the oncall calendar (a tool used to manage on-call schedule/rotations)

Containers

• We hear you… and like everyone else, we leverage containerised applications and environments (ecs,eks) for our microservices.
• We are actively moving our production services to Kubernetes

Did we miss anything? Of course, if we were to mention everything the team contributes, you’d be reading this blog post all day. So check back as we continue to share more about Titan’s DevOps team and the processes we’ve created to ensure efficiency, accuracy, high availability and reliability. 

By the way—Titan is always looking for exceptional talent to join our DevOps team. Maybe you’re the hire we’ve been looking for :). Interested in talking about this with us more? Apply here.

The post Join us for an exciting ride of DevOps @Titan appeared first on Email Service Suite for Professionals and Businesses: Try Titan Email.