AWS VPC Flow Logs

Monitoring network traffic is essential for ensuring security compliance, investigating incidents, tracking metrics, and setting up automated alerts. AWS VPC Flow Logs provide valuable insights into the traffic flowing through your VPC's network interfaces. In our upcoming tech blog, we'll explore how to leverage VPC Flow Logs in Amazon CloudWatch. By creating custom metrics and alerts based on these logs, you can gain valuable insights into trends and receive timely notifications for potential security threats. Additionally, we'll demonstrate how to use Amazon Athena to query and analyze VPC Flow Logs stored in S3. Join us as we delve into these powerful tools for enhancing network security and performance.

Benefits of AWS VPC Flow Logs:

Enhanced Visibility: VPC Flow Logs provide a comprehensive view of the traffic traversing your VPC, including details such as source and destination IP addresses, ports, protocols, and packet counts. This enhanced visibility allows organizations to gain insights into network traffic patterns and identify potential security threats or performance issues.

Security Monitoring: By analyzing VPC Flow Logs, organizations can detect and investigate security incidents such as unauthorized access attempts, malware infections, or data exfiltration attempts. With real-time alerts and notifications based on predefined criteria, security teams can promptly respond to security threats and mitigate risks.

Compliance Assurance: VPC Flow Logs play a crucial role in meeting compliance requirements by providing detailed records of network traffic. Organizations can use VPC Flow Logs to demonstrate compliance with regulatory standards such as PCI DSS, HIPAA, GDPR, and more by maintaining an audit trail of network activity.

Troubleshooting and Performance Optimization: VPC Flow Logs can be invaluable for troubleshooting network issues and optimizing performance. By analyzing traffic patterns and identifying bottlenecks or latency issues, organizations can take proactive measures to optimize network performance and ensure smooth operation of their applications and services.

How to Leverage AWS VPC Flow Logs:

Enabling VPC Flow Logs: The first step is to enable VPC Flow Logs for your VPC. This can be done through the AWS Management Console, CLI, or API. You can specify the type of traffic to log (e.g., accepted, rejected), the destination for the logs (e.g., CloudWatch Logs, S3), and the log format.

A. Create an S3 Bucket for VPC Flow Logs and VPC Flow Log to S3

Analyzing VPC Flow Logs with Amazon CloudWatch: Once VPC Flow Logs are enabled, you can analyze them using Amazon CloudWatch. CloudWatch provides a centralized location for storing, monitoring, and analyzing log data. You can create custom metrics, dashboards, and alarms based on VPC Flow Logs to monitor network activity and detect anomalies.

Automated Alerts and Notifications: By configuring CloudWatch Alarms based on predefined thresholds or patterns, organizations can receive automated alerts and notifications for security incidents or performance issues detected in VPC Flow Logs. These alerts enable security teams to respond promptly and mitigate risks.

A. Create CloudWatch Filters and Alerts

Create filter pattern

Enter the following Filter Pattern to track failed SSH attempts on port 22:

[version, account, eni, source, destination, srcport, destport="22", protocol="6", packets, bytes, windowstart, windowend, action="REJECT", flowlogstatus]

Advanced Analysis with Amazon Athena: For more in-depth analysis, organizations can use Amazon Athena to query and analyze VPC Flow Logs stored in Amazon S3. Athena allows you to run standard SQL queries against your log data, making it easy to extract insights, perform forensic analysis, and generate reports.

B. Create CloudWatch Log Group and VPC Flow Log to CloudWatch

Create metric filter & Create Alarm

Configure VPC Flow Logs Table and Partition in Athena

CREATE EXTERNAL TABLE IF NOT EXISTS default.vpc_flow_logs (

version int,

account string,

interfaceid string,

sourceaddress string,

destinationaddress string,

sourceport int,

destinationport int,

protocol int,

numpackets int,

numbytes bigint,

starttime int,

endtime int,

action string,

logstatus string

)

PARTITIONED BY (dt string)

ROW FORMAT DELIMITED

FIELDS TERMINATED BY ' '

LOCATION 's3://lab-vpc-flow-log-01/AWSLogs/520594340035/vpcflowlogs/us-east-1/'

TBLPROPERTIES ("skip.header.line.count"="1");

ALTER TABLE default.vpc_flow_logs

ADD PARTITION (dt='2024-03-29')

location 's3://lab-vpc-flow-log-01/AWSLogs/520594340035/vpcflowlogs/us-east-1/2024/03/29';

SELECT day_of_week(from_iso8601_timestamp(dt)) AS day, dt, interfaceid, sourceaddress, destinationport, action, protocol FROM vpc_flow_logs WHERE action = 'REJECT' AND protocol = 6 order by sourceaddress LIMIT 100;

AWS VPC Flow Logs are a powerful tool for enhancing network monitoring, security, and compliance in AWS environments. By capturing detailed information about network traffic, organizations can gain valuable insights into their VPC's activity, detect security threats, troubleshoot issues, and ensure compliance with regulatory requirements. Leveraging AWS VPC Flow Logs, organizations can strengthen their security posture, optimize network performance, and safeguard their critical assets in the cloud.