What is AWS and Case Study on how companies like Netflix got benefitted from it?
AWS:
AWS (Amazon Web Services) is a comprehensive, evolving cloud computing platform provided by Amazon that includes a mixture of infrastructure as a service (IaaS), platform as a service (PaaS), and packaged software as a service (SaaS) offerings. AWS services can offer Organization tools such as compute power, database storage, and content delivery services.
AWS was launched in 2002. The company wanted to sell its unused infrastructure as a service, or as an offering to customers. The idea was met with enthusiasm. Amazon launched its first AWS product in 2006. Four years later, in 2012, Amazon hosted a huge event focused on collecting customer input about AWS. The company still holds similar events, such as Reinvent, which allows customers to share feedback about AWS.
In 2015, Amazon announced that its AWS revenue had reached $7.8 billion. Between then and 2016, AWS launched measures that helped customers migrate their services to AWS. Those measures, along with the public’s growing appreciation of AWS’s features, induced further economic growth. Amazon’s revenue increased to $12.2 billion in 2016. Today, AWS offers customers 160 products and services. That number will likely increase, given the rate at which Amazon builds upon and tweaks AWS.
AWS Services:
Let's take a look at some of the services that are provided by AWS:
1. Amazon S3
This tool is used for internet back up, and it’s the cheapest storage option in the object-storage category. The best part: you can retrieve stored data from almost anywhere whenever you need it.
2. AWS Data Transfer Products
As the name suggests, these are migration, data collection, and data transfer products that help you collect data seamlessly. They also enable you to monitor and analyze data in real-time.
3. Amazon EC2 (Elastic Compute Cloud)
This provides a secure and resizable computing capacity based on your needs. The service is designed to make web-scale cloud computing more accessible.
4. Amazon SNS (Simple Notification Services)
This is a notification tool that delivers messages to a large number of subscribers through email or SMS. You can send alarms, service notifications, and other messages intended to call attention to important information.
5. Amazon KMS (Key Management System)
This is a security tool that uses 256-bit encryption for your data. It also safeguards it from hackers and cyber attacks.
6. Amazon Lambda
This service runs your code depending on specific events and manages the dependent resources. You need neither managing nor provisioning servers, and how much you pay depends on how long it takes to execute your code. It saves a lot of money compared with services that charge hourly rates.
7. Route 53
This is a DNS service in the cloud that doesn’t require you to maintain a separate DNS account. It’s designed to provide businesses with a reliable and cost-effective method to route users to internet applications
Case Study on how Netflix uses AWS:
Netflix was originally a DVD shipping business where they would send out DVDs of your chosen programs to you. This was going well until 2008 where they experienced a major database loss and for 3 days could not ship out any DVDs to their customers. That was when the senior management at Netflix realized that they had to shift from continuous vertical scaling which leads to single points of failure to a more reliable and scalable horizontal scaling system. They chose Amazon Web Services despite having Amazon as a competitor (Amazon has their own streaming service known as Amazon Prime) because AWS provided them with the greatest scaling capabilities and the biggest set of available features. It took 7 years of migration for Netflix to shut down their last remaining data centers and move completely to the cloud.
Netflix is the world’s leading internet television network, with more than 100 million members in more than 190 countries enjoying 125 million hours of TV shows and movies each day. Netflix uses AWS for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more — hundreds of functions that in total use more than 100,000 server instances on AWS.
Application Monitoring:
Netflix uses Amazon Web Services (AWS) for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more — hundreds of functions that in total use more than 100,000 server instances on AWS. This results in an extremely complex and dynamic networking environment where applications are constantly communicating inside AWS and across the Internet. Monitoring and optimizing its network is critical for Netflix to continue improving customer experience, increasing efficiency, and reducing costs. In particular, Netflix needed a solution for ingesting, augmenting, and analyzing the multiple terabytes of data its network generates daily in the form of virtual private cloud (VPC) flow logs. This would enable Netflix to identify performance-improvement opportunities, such as identifying apps that are communicating across regions and collocating them. The company would also be able to increase uptime by quickly detecting and mitigating application downtime.
Each log record carries information about the communications between two IP addresses. However, in a dynamic environment like the one at Netflix, where an IP address can float between applications from day to day or even minute to minute, IP addresses alone don’t have much meaning. “The data sources we had before we took on this initiative were one-sided,” says John Bennett, a senior software engineer at Netflix. “We’d know an application was connecting to others, but we didn’t know both sides of the conversation and how to optimize those communications or the placement of the applications on the network.”Netflix set out to establish a new data source that could give it more insight into communication among applications and regions by combining VPC flow logs with application metadata.
Centralizing Flow Logs Using Amazon Kinesis Data Streams:
From the outset, AWS enabled Netflix to experiment with different approaches to analyzing its network data. “Early in the design process, the flexibility to try different ways of processing the data was important,” says Bennett. “We experimented with multiple designs and used many AWS products to get here.”The solution Netflix ultimately deployed — known internally as Dredge — centralizes flow logs using Amazon Kinesis Data Streams. The application reads the data from Amazon Kinesis Data Streams in real-time and enriches IP addresses with application metadata to provide a full picture of the networking environment. “Usually, we would put the data into a database, which would build an index to enable faster querying,” says Bennett. “Dredge joins the flow logs with application metadata as it streams and indexes it without using a database, which eliminates a lot of the complexity.”
The enriched data lands in an open-source analytics application called Druid. Netflix uses the OLAP querying functionality of Druid to quickly slice data into regions, availability zones, and time windows to visualize it and gain insight into how the network is behaving and performing.AWS was the logical choice for Dredge in part because the data was already resident in the AWS Cloud. “It would have been daunting to publish, stream, and consume that much information from an external system such as Kafka,” says Bennett. “It took just a few API calls to centralize multiple terabytes of flow logs into Amazon Kinesis Data Streams. Now we can focus on getting insights from the data rather than simply getting access to it.”
The scalability of Amazon Kinesis Data Streams was a good fit for the Dredge application because of the cyclical and elastic nature of network usage at Netflix. “When it comes to our networking data, it’s more cost-efficient to be able to scale up and down, which is not as easy to do with alternatives to Amazon Kinesis Data Streams,” says Bennett.
How does Netflix Improve Customer Experience?
Netflix’s Amazon Kinesis Data Streams-based solution has proven to be highly scalable, each day processing billions of traffic flows. Typically, about 1,000 Amazon Kinesis shards work in parallel to process the data stream. “Amazon Kinesis Data Streams processes multiple terabytes of log data each day, yet events show up in our analytics in seconds,” says Bennett. “We can discover and respond to issues in real-time, ensuring high availability, and great customer experience.”
Netflix is now able to identify new ways to optimize its applications, whether that means moving an application from one region to another or changing to a more appropriate network protocol for a specific type of traffic. “Our solution built on Amazon Kinesis enables us to identify ways to increase efficiency, reduce costs, and improve resiliency for the best customer experience,” says Bennett. Although a streaming data solution is not new to the IT industry, it is an innovation in the networking space. “Netflix is heavily invested in AWS in part because it abstracts the underlying network, so we don’t have to deal with switches and routers,” says Bennett. “We’re monitoring, analyzing, and optimizing at a higher level of the stack — in ways we would never even consider if we were running our own data centers.”