CAIDA Spoofer Project Improves Routing Security by Publicizing Spoofed Source Address Packets

This week, the Center for Applied Internet Data Analysis (CAIDA) announced that:

“In response to feedback from operational security communities, CAIDA’s source address validation measurement project (https://spoofer.caida.org) is automatically generating monthly reports of ASes originating prefixes in BGP for systems from which we received packets with a spoofed source address. We are publishing these reports to network and security operations lists in order to ensure this information reaches operational contacts in these ASes.”

We see this as a positive step forward for routing security. Anti-spoofing is one of the major MANRS Actions for network operators, and in fact we’ve been asking prospective MANRS participants to run Spoofer for some time now.

IP source address spoofing is when fake source addresses hide a sender’s identity or impersonate someone else. This can be exploited in various ways, most notably to execute Denial of Service (DoS) attacks that send traffic to the spoofed address. To combat this, MANRS calls for anti-spoofing — enabling source address validation to prevent spoofed packets from entering or leaving your network.

You can check the anti-spoofing capabilities of your own network by downloading the software here. And of course, you can read all about the Anti-spoofing component of MANRS here.

Together, we can make the Internet a better place and Protect The Core!

MANRS, IXPs, and Routing Security in the News

It’s been a busy couple of weeks for routing security. First, on 23 April, we introduced a new category of MANRS participants aimed specifically at Internet Exchange Points. Then, just two days later, we learned about yet another BGP hijacking wreaking havoc, which could have been avoided if more networks across the globe implemented the MANRS for Network Operators Actions.

All of this has resulted in several news articles recently. Here’s a roundup of the coverage.

Related to the IXP Programme:

And articles relating MANRS as a solution for BGP hijacking:

It’s great to see MANRS and routing security getting the media coverage it deserves. We encourage all network operators and IXPs to join MANRS as we work together to improve the security and resilience of the Internet’s routing system.

A Deeper Dive into the Amazon Route 53 BGP Hijack

Yesterday, we reported some initial news and details about the Amazon Route 53 BGP hijack that resulted in a loss to some cryptocurrency users.

Today on the Internet Society blog, Aftab Siddiqui presents a more technical dive into what exactly happened, using data from Isolario, RIPE Stats, and various reports from organizations who monitor Internet routing and health.

It’s an interesting read, and once again points out how MANRS can help alleviate future incidents. In fact, the Actions called for in MANRS for Network Operators were in place by a few network operators involved in this incident, which helped mitigate some of the damage. From Aftab’s post:

This problem could have been easily avoided if Hurricane Electric (AS6939), 1&1 Internet SE (AS8560), Shaw Communications Inc. (AS6327), and BroadbandOne/WV Fibre (AS19151) had prefix filtering in place. Thankfully, Level3 (AS3356), Cogentco (AS174), and NTT (AS2914) are all MANRS members and had prefix filters in place, or the damage would have been much bigger. As per Dyn they recorded only 15% of their nodes received malicious specific advertisement originated from AS10297, while NLNOG-RING (AS199036) were getting 87 unique paths to 205.251.192.0/23 (one of the Route53 prefix) originating from Amazon (AS16509) at 10am UTC. But when the attack started at 11:05 UTC they installed 15 new paths for 205.251.192.0/24 (one of the malicious more specific prefix) originated from eNET (AS10297). Out of those 15 unique paths, 12 of them were coming via Hurricane Electric (AS6939).

We encourage you to read the whole deep dive, and of course to implement the MANRS Actions (for network operators or for IXPs) and to join the MANRS community of security-minded organizations.

Together, we can protect the core!

Another BGP Hijacking Event Highlights the Importance of MANRS and Routing Security

Another BGP hijacking event is in the news today. This time, the event is affecting the Ethereum cryptocurrency. (Read more about it here, or here.) Users were faced with an insecure SSL certificate. Clicking through that, like so many users do without reading, they were redirected to a server in Russia, which proceeded to empty the user’s wallet. DNSSEC is important to us, so please check out the Deploy360 DNSSEC resources to make sure your domain names are protected. In this post, though, we’ll focus on the BGP hijacking part of this attack.

What happened?

First, here’s a rundown of routing attacks on cryptocurrency in general – https://btc-hijack.ethz.ch/.

In this case specifically, the culprit re-routed DNS traffic using a man in the middle attack using a server at an Equinix data center in Chicago. Cloudflare has put up a blog post that explains the technical details. From that post:

“This [hijacked] IP space is allocated to Amazon(AS16509). But the ASN that announced it was eNet Inc(AS10297) to their peers and forwarded to Hurricane Electric(AS6939).

“Those IPs are for Route53 Amazon DNS servers. When you query for one of their client zones, those servers will reply.

“During the two hours leak the servers on the IP range only responded to queries for myetherwallet.com. As some people noticed SERVFAIL.

“Any DNS resolver that was asked for names handled by Route53 would ask the authoritative servers that had been taken over via the BGP leak. This poisoned DNS resolvers whose routers had accepted the route.

“This included Cloudflare DNS resolver 1.1.1.1. We were affected in Chicago, Sydney, Melbourne, Perth, Brisbane, Cebu, Bangkok, Auckland, Muscat, Djibouti and Manilla. In the rest of the world, 1.1.1.1 worked normally.”

What does this have to do with MANRS and routing security?

Mutually Agreed Norms for Routing Security (MANRS) calls for four simple, but concrete actions ALL network operators should take to reduce the most common routing threats. The first is filtering, which prevents the propagation of incorrect routing information. (The others are anti-spoofing, address validation, and global coordination.) If all the operators along the path had implemented the MANRS actions – especially filtering – this would not have propagated across the Internet like it did.

For example, eNET also peers with Level3 (AS3356) and NTT (AS2914), but those operators didn’t forward the wrong information because they are MANRS compliant. (Other MANRS participants are listed here.)

Security Boulevard has also published a short recap of this BGP hijack event and called out MANRS as a potential solution.

What are you waiting for?

This can still happen at any time. Network operators have a responsibility to ensure a globally robust and secure routing infrastructure. Your network’s safety depends on a routing infrastructure that weeds out bad actors and accidental misconfigurations that wreak havoc on the Internet. The more network operators work together, the fewer incidents there will be, and the less damage they can do.

Learn more about MANRS here. Implement the four actions for network operators and join the communityof security-minded operators working together to make the Internet safer for everyone.

Introducing a New MANRS IXP Programme for Routing Security

Today, we are pleased to announce that MANRS is getting a new category of members – IXPs. The MANRS IXP Programme introduces a separate membership category for IXPs with a set of security actions to address the unique needs and concerns of IXPs.

The ten founding participants are Asteroid (International), CABASE (Argentina), CRIX (Costa Rica), DE-CIX (Germany), INEX (Ireland), MSK-IX (Russia), Netnod (Sweden), RINEX (Rwanda), TorIX (Canada), and YYCIX (Canada).

Programme participation provides an opportunity for an IXP to demonstrate its attention to the security and sustainability of the Internet ecosystem and, therefore, its dedication to providing high-quality services.

The IXP Action set was developed by a group of IXPs from all around the world and was presented at multiple IXP fora for discussion and feedback. We hope that with IXPs as partners, their ISP members will join the Network Operator category of MANRS.

Participation in the MANRS IXP Programme requires an IXP to implement and document a majority of the IXP Programme Actions (at least three out of five). Actions 1 and 2 are mandatory, and the IXP must implement at least one additional Action. Here are the five Actions:

  1. Facilitate prevention of propagation of incorrect routing information
  2. Promote MANRS in the IXP’s membership
  3. Protect the peering platform
  4. Facilitate global operational communication and coordination between network operators
  5. Provide monitoring and debugging tools to members

The full set of Actions for IXPs can be found here: https://www.manrs.org/participants/ixp/

The IXP Programme founding participants have taken these actions, which establish a security baseline that many IXPs may already surpass and from which others can build.

All IXPs are invited to join this new Programme! Read more about the Actions here, sign up to join here, or see the list of participants here. You can also read the full press release about the new Programme.

Supporting Quotes from Initial Participants

“Asteroid has a strong focus on security, so we are proud to be a founding participant in MANRS. IXPs are often at the heart of the local operators’ community, and we believe it’s our responsibility to lead by example and to promote routing security. MANRS is an opportunity for us all to lift the standard of our shared security responsibility,” said Andy Davidson, CTO, Asteroid.

“Our community has matured to formalize “good-order rules.” Usually, the participants of the IX have already observed all the stipulated norms, but now we will have a formal document that the interested persons will be able to sign. It was important for our participants (over 95% use the Route Server service) that MSK-IX take on the role of arbiter,” says Alexander Ilin, Chief Technical Officer, MSK-IX.

“We at DE-CIX are proud to support the MANRS IXP Programme as a founding participant with our knowledge and experience. It is time for IXPs to take responsibility to make the Internet a more secure and resilient place,” said Christoph Dietzel, Head of Research & Development at DE-CIX.

“We are very proud to be involved with the MANRS IXP Programme. Securing Internet routing has clear benefits not just for IXP members and their customers but for end users and the global internet community as a whole”, says Mattias Karlsson, Head of Engineering, Netnod.

Routing Security and MANRS at EURO-IX This Week

This week, Andrei Robachevsky will be talking about routing security in general and MANRS in particular at Euro-IX in Galway, Ireland. The European Internet Exchange Association (Euro-IX) gathers 83 Internet Exchange Points (IXPs) from around the world. It was formed in May 2001 with the intention to develop, strengthen and improve the IXP community.

The MANRS Actions were initially designed for network operators, but IXPs also play an active role in protecting the Internet. IXPs represent active communities with common operational objectives and already contribute to a more resilient and secure Internet infrastructure. Euro-IX is an opportunity to highlight the many ways IXPs can contribute to the overall health and resilience of the routing system by joining MANRS.

MANRS can help IXPs build safe neighborhoods, leveraging the MANRS security baseline. It also demonstrates an IXP’s commitment to security and sustainability of the Internet ecosystem, and dedication to providing high quality services.

IXPs are important partners in the MANRS community

IXPs can be a collaborative focal point to discuss and promote the importance of routing security. To address the unique needs and concerns of IXPs, the community is creating a related but separate set of MANRS actions for IXP members. We’ll explain more about the upcoming MANRS IXP Programme and invite IXPs to join once the program launches.

If you’ll be in Galway, please let us know!

New MANRS Logo & Your Feedback Needed

We have a lot of exciting things happening in the MANRS world lately, and I’d like to take a moment to share a few of them with you and also ask for your feedback. 

New Logo

We have a new MANRS logo! This is now incorporated in a few places on the website, and we’ll be using it in presentations, on stickers and t-shirts, on our Twitter graphics, etc. You can download the logo in a few formats at https://www.manrs.org/downloads/. It’s available in vertical and horizontal, color and black & white versions. If you need another size/format, please let us know and we’d be happy to send it. 

Updated Website

As we roll out some new programs and content, we’ll be taking a look at the website as a whole, and we’d love your feedback. What would you like to see on the site in the future? What do you like? What do you hate? We will try to incorporate as much of your feedback as possible. Please feel free to drop me a line at manrs@isoc.org to share your thoughts. 

New Content

The community has developed several new initiatives that we’ll be rolling out in the coming months, so stay tuned for more information on the IXP Partnerships, a MANRS Ambassador program, additional tutorials and training, and more. What else do you think we need? Again, I invite you to email us to share your ideas. 

Routing Security is a Serious Problem – and MANRS Can Help. A Report from APRICOT 2018.

Last week, at APRICOT 2018 in Kathmandu, Nepal, there were a lot of talks and discussions focused on routing security and the Mutually Agreed Norms for Routing Security (MANRS).

First, there was a Routing Security BoF, attended by about 150 people, where we talked about what it takes to implement routing security practices, how CDNs and other players can help, and why it is so difficult to make progress in this area. The BoF included an interactive poll at the end, and it showed some interesting results:

  • Participants almost unanimously see lack of routing security as a serious problem.
  • Slow progress in this area is largely seen as due to a lack of incentives
  • Participants see community initiatives (like MANRS) as the main driving forces for improvement, followed by CDNs and cloud providers. They doubt that governments or end-customers can effectively drive change.

My colleague Aftab Siddiqui is writing a separate blog post just about that BoF, so watch the blog in the next day or two.

Later, in the security track of the main APRICOT programme, Andrei Robachevsky, ISOC’s Technology Programme Manager, presented statistics on routing incidents and suggested a way forward based on the MANRS approach. In his presentation, “Routing Security in 2017 – We can do better! And how MANRS can help”, he provided a detailed overview of simple steps a network operator should take to improve routing hygiene and overall security of the routing system we all depend on so much.

His slides are available here:

An interactive poll that followed offered interesting insights into the challenges and state of securing routing:

  • More than 50% of the operators polled experienced routing incidents with varying impact, and only a lucky <20% were not terribly affected by them
  • There were remarkable differences regarding the security posture of networks. More than half of respondents have no resources to implement even such simple measures as MANRS. At the same time 1/3 of network operators already implement those measures and actively promote them in the community

It was very encouraging to see that a majority of the participants valued MANRS and wanted to join. At least when they become ready to implement the actions.

I’ll leave you with a quote Aftab shared at the beginning of the Routing BoF, from Nobel Peace Prize Winner Jane Addams: “The good we secure for ourselves is precarious and uncertain until it is secured for all of us and incorporated into our common life.”

Are you ready to look into the four MANRS Actions and start moving your network in the right direction? We have an Implementation Guide and Training Modules available! Or perhaps you are ready to join MANRS? Sign up here!

Improving Routing Security: Introducing Six New MANRS Tutorials

Routing outages or attacks – such as hijacking, leaks, and spoofing – can lead to stolen data, lost revenue, reputational damage and more, all on a global scale. Routing security is therefore vital to the future and stability of the Internet, and the Mutually Agreed Norms for Routing Security (MANRS) initiative implements crucial fixes. Today, we are pleased to announce a series of six new MANRS tutorials that will help network operators improve both the Internet’s routing security and their own network’s operational efficiency.

These tutorials are intended for network administrators, network engineers, and others with a working knowledge of routing and security who are looking for steps to improve their network’s routing security and to join the growing list of MANRS participants.

About the Tutorials

Module 1: Introduction to MANRS

What is MANRS, and why should you join? MANRS is a global initiative to implement crucial fixes needed to eliminate the most common routing threats. In this module you will learn about vulnerabilities of the Internet routing system and how four simple steps, called MANRS Actions, can help dramatically improve Internet security and reliability.

Module 2: IRRs, RPKI, and PeeringDB

This module helps you understand the databases and repositories MANRS participants should use to document routing policy and maintain contact information. You’ll learn what database objects to use to document routing information related to your network and how to register information in the RPKI system. Finally, you will learn how to use the Peering DB and other databases to publish your contact information.

Module 3: Global Validation: Facilitating validation of routing information on a global scale

In this module, you will learn how to prevent incorrect routing announcements from your customers and your own network. The module explains how filters can be built, including the tools used to build them. It also shows how to signal to other networks which announcements from the network are correct.

Module 4: Filtering: Preventing propagation of incorrect routing information

This module will help you apply anti-spoofing measures within your network. After this module you will be able to identify points/devices in the network topology where anti-spoofing measures should be applied, identify adequate techniques to be used (for example, uRPF, or ACL filtering), configure your devices to prevent IP spoofing, and verify that the protection works.

Module 5: Anti-Spoofing: Preventing traffic with spoofed source IP addresses

This module is to understand how to create and maintain contact information in publicly accessible places. It explains why it is important to publish and maintain contact information, how to publish contact information to Regional Internet Registries (RIRs), Internet Routing Registries (IRRs), and PeeringDB, and what contact information you should publish to a company website.

Module 6: Coordination: Global communication between network operators

This module helps you understand how to enable others to validate route announcements originating from your network by documenting a Network Routing Policy. You’ll learn what a Network Routing Policy is, how to document your organization’s Network Routing Policy and make it publicly available in order to signal to other networks which announcements from your network are correct.

Please go through all six new MANRS tutorials, and get your network ready to join MANRS!

14,000 Incidents: a 2017 Routing Security Year in Review

How was the state of the Internet’s routing system in 2017? Let’s take a look back using data from BGPStream. Some highlights:

  • 13,935 total incidents (either outages or attacks like route leaks and hijacks)
  • Over 10% of all Autonomous Systems on the Internet were affected
  • 3,106 Autonomous Systems were a victim of at least one routing incident
  • 1,546 networks caused at least one incident

An ‘incident’ is a suspicious change in the state of the routing system that can be attributed to an outage or a routing attack, like a route leak or hijack (either intentional or due to a configuration mistake).[i] Let’s look at just a few examples of incidents picked up by the media.

March 2017. SECW Telecom in Brazil hijacked prefixes of Cloudflare, Google, and BancoBrazil causing some outage for these services in the region.

April 2017. Large chunks of network traffic belonging to MasterCard, Visa, and more than two dozen other financial services companies were briefly routed through a Russian telecom. For several minutes, Rostelecom was originating 50 prefixes for numerous other Autonomous Systems, hijacking their traffic.

August 2017. Google accidentally leaked BGP prefixes it learned from peering relationships, essentially becoming a transit provider instead of simply exchanging traffic between two networks and their customers, causing large-scale internet disruption. It hit Japanese users the hardest, slowing or blocking access to websites and online services for dozens of Japanese companies.

October 2017. Another BGP mishap caused reachability and performance problems for networks such as Twitter, Google, and others. For almost 20 minutes, traffic for many large CDNs was rerouted through Brazil, caused by a BGP leak.BGP mishap caused reachability and performance problems for networks such as Twitter, Google, and others. For almost 20 minutes, traffic for many large CDNs was rerouted through Brazil, caused by a BGP leak.

November 2017. Level 3 BGP routing issues causing large scale network service degradation in North America for slightly more than 90 minutes. Another route leak.

December 2017. Several high-profile sites (Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and Riot Games) were rerouted to a previously unused Russian AS. Two BGP routing incidents only lasted about three minutes each.

Not a single day passed without an incident. While none of the incidents was catastrophic, all of them continue to demonstrate the lack of routing controls like those called for in MANRS that could have prevented them from happening.

This is just a small fraction of what happened in the routing system in 2017. Rather than measure routing security by anecdotal evidence, let’s look at the data.

Routing Incidents

Of the 13,935 total incidents, 62% were classified as outages and 38% were considered routing attacks like route leaks and hijacks.

6,128 Autonomous Systems were involved, which is more that 10% of all announced ASNs on the Internet. If we look at the outages, almost half of them happened to Brazilian operators.[ii]

Let us look to incidents that represent a potential attack, be it malice or a configuration mistake. It is interesting to analyze such routing incidents by the roles a network played – whether it was a victim, a culprit, or an accomplice.

The U.S. ranks first among countries where networks became a victim of an incident, for example when a network’s prefix is hijacked. Last year, that happened 1,193 times in the U.S. It is followed by Brazil (450), India (299), and Russia (242).

Unsurprisingly, the majority of the networks victimized by the most incidents are based in the U.S. In total 3,106 Autonomous Systems were victims of at least one routing incident in 2017.

U.S. and Brazil, followed by Russia and China, lead the list of countries in which networks caused incidents. They are responsible for more that 75% of all incidents. Overall, 1,546 networks caused at least one incident during 2017.

The ranking is different when it comes to the top 10 guilty networks. An interesting case is AS198949 – SecurityDAM, responsible for 54 incidents, mostly prefix hijacks. This is a security provider, offering DDoS attack mitigation among other services. Most probably these incidents were part of attack mitigation actions. Since the BGPStream only registers suspicious routing changes, without knowing intent in some cases it is impossible to distinguish an attack from a legitimate (or consented) routing change.

The U.S. also leads the list of countries with networks that could have prevented an attack, but didn’t, such as not filtering false routing announcements from their customers (one of MANRS Actions). The usual suspects – Russia, Brazil, and China – follow.

In the end, I’d like to note that absolute numbers tell only part of the story. They need to be put into perspective. Countries and networks differ significantly in terms of connected users, announced prefixes, etc. The numbers in this report are not normalized by any of these metrics, but to give an idea, look at a possible one – number of active networks in a country. For example, perhaps the U.S. leads many of these lists simply because there are more networks where incidents could happen. (The “AS’s advertised” chart below comes from data available at http://resources.potaroo.net/iso3166/regiontablecc.html.)

Another point is that it is hard to say whether these numbers are OK, or really bad. Is the system improving or getting worse? The statistics in this report will be a good basis for a trend analysis in years to come.

What You Can Do – Join MANRS

MANRS is a community-driven initiative coordinated by the Internet Society that provides a minimum set of low-cost and low-risk actions that, taken together, can help improve the resilience and security of the routing infrastructure. The more service providers apply these minimum actions, the fewer incidents there will be, and the less damage they can do.

There are four MANRS Actions:

  • Filtering – Ensure the correctness of your own announcements and of announcements from your customers to adjacent networks with prefix and AS-path granularity
  • Anti-spoofing – Enable source address validation for at least single-homed stub customer networks, your own end-users, and infrastructure
  • Coordination – Maintain globally accessible up-to-date contact information
  • Global Validation – Publish your data, so others can validate routing information on a global scale

Maintaining up-to-date filters for customer announcements could mitigate many route leaks. Preventing address squatting could help ward off things like spam and malware. Keeping complete and accurate routing policy data in Internet Routing Registry (IRR) or Resource Public Key Infrastructure (RPKI) repositories is essential for global validation that helps prevent BGP prefix hijacking. Having updated contact information is vital to solving network emergencies quickly.

Let us hope we will see more network operators joining MANRS, and improvements in routing security in 2018. Happy New Year!


[i] BGPStream is an operational tool that tries to minimize false positives, so this number of total incidents may be on the low side.

[ii] This is only counting the number of incidents and not factoring in duration or number of prefixes affected, which may indicate the impact of these incidents.