Routing Security is a Serious Problem – and MANRS Can Help. A Report from APRICOT 2018.

Last week, at APRICOT 2018 in Kathmandu, Nepal, there were a lot of talks and discussions focused on routing security and the Mutually Agreed Norms for Routing Security (MANRS).

First, there was a Routing Security BoF, attended by about 150 people, where we talked about what it takes to implement routing security practices, how CDNs and other players can help, and why it is so difficult to make progress in this area. The BoF included an interactive poll at the end, and it showed some interesting results:

  • Participants almost unanimously see lack of routing security as a serious problem.
  • Slow progress in this area is largely seen as due to a lack of incentives
  • Participants see community initiatives (like MANRS) as the main driving forces for improvement, followed by CDNs and cloud providers. They doubt that governments or end-customers can effectively drive change.

My colleague Aftab Siddiqui is writing a separate blog post just about that BoF, so watch the blog in the next day or two.

Later, in the security track of the main APRICOT programme, Andrei Robachevsky, ISOC’s Technology Programme Manager, presented statistics on routing incidents and suggested a way forward based on the MANRS approach. In his presentation, “Routing Security in 2017 – We can do better! And how MANRS can help”, he provided a detailed overview of simple steps a network operator should take to improve routing hygiene and overall security of the routing system we all depend on so much.

His slides are available here:

An interactive poll that followed offered interesting insights into the challenges and state of securing routing:

  • More than 50% of the operators polled experienced routing incidents with varying impact, and only a lucky <20% were not terribly affected by them
  • There were remarkable differences regarding the security posture of networks. More than half of respondents have no resources to implement even such simple measures as MANRS. At the same time 1/3 of network operators already implement those measures and actively promote them in the community

It was very encouraging to see that a majority of the participants valued MANRS and wanted to join. At least when they become ready to implement the actions.

I’ll leave you with a quote Aftab shared at the beginning of the Routing BoF, from Nobel Peace Prize Winner Jane Addams: “The good we secure for ourselves is precarious and uncertain until it is secured for all of us and incorporated into our common life.”

Are you ready to look into the four MANRS Actions and start moving your network in the right direction? We have an Implementation Guide and Training Modules available! Or perhaps you are ready to join MANRS? Sign up here!

Improving Routing Security: Introducing Six New MANRS Tutorials

Routing outages or attacks – such as hijacking, leaks, and spoofing – can lead to stolen data, lost revenue, reputational damage and more, all on a global scale. Routing security is therefore vital to the future and stability of the Internet, and the Mutually Agreed Norms for Routing Security (MANRS) initiative implements crucial fixes. Today, we are pleased to announce a series of six new MANRS tutorials that will help network operators improve both the Internet’s routing security and their own network’s operational efficiency.

These tutorials are intended for network administrators, network engineers, and others with a working knowledge of routing and security who are looking for steps to improve their network’s routing security and to join the growing list of MANRS participants.

About the Tutorials

Module 1: Introduction to MANRS

What is MANRS, and why should you join? MANRS is a global initiative to implement crucial fixes needed to eliminate the most common routing threats. In this module you will learn about vulnerabilities of the Internet routing system and how four simple steps, called MANRS Actions, can help dramatically improve Internet security and reliability.

Module 2: IRRs, RPKI, and PeeringDB

This module helps you understand the databases and repositories MANRS participants should use to document routing policy and maintain contact information. You’ll learn what database objects to use to document routing information related to your network and how to register information in the RPKI system. Finally, you will learn how to use the Peering DB and other databases to publish your contact information.

Module 3: Global Validation: Facilitating validation of routing information on a global scale

In this module, you will learn how to prevent incorrect routing announcements from your customers and your own network. The module explains how filters can be built, including the tools used to build them. It also shows how to signal to other networks which announcements from the network are correct.

Module 4: Filtering: Preventing propagation of incorrect routing information

This module will help you apply anti-spoofing measures within your network. After this module you will be able to identify points/devices in the network topology where anti-spoofing measures should be applied, identify adequate techniques to be used (for example, uRPF, or ACL filtering), configure your devices to prevent IP spoofing, and verify that the protection works.

Module 5: Anti-Spoofing: Preventing traffic with spoofed source IP addresses

This module is to understand how to create and maintain contact information in publicly accessible places. It explains why it is important to publish and maintain contact information, how to publish contact information to Regional Internet Registries (RIRs), Internet Routing Registries (IRRs), and PeeringDB, and what contact information you should publish to a company website.

Module 6: Coordination: Global communication between network operators

This module helps you understand how to enable others to validate route announcements originating from your network by documenting a Network Routing Policy. You’ll learn what a Network Routing Policy is, how to document your organization’s Network Routing Policy and make it publicly available in order to signal to other networks which announcements from your network are correct.

Please go through all six new MANRS tutorials, and get your network ready to join MANRS!

14,000 Incidents: a 2017 Routing Security Year in Review

How was the state of the Internet’s routing system in 2017? Let’s take a look back using data from BGPStream. Some highlights:

  • 13,935 total incidents (either outages or attacks like route leaks and hijacks)
  • Over 10% of all Autonomous Systems on the Internet were affected
  • 3,106 Autonomous Systems were a victim of at least one routing incident
  • 1,546 networks caused at least one incident

An ‘incident’ is a suspicious change in the state of the routing system that can be attributed to an outage or a routing attack, like a route leak or hijack (either intentional or due to a configuration mistake).[i] Let’s look at just a few examples of incidents picked up by the media.

March 2017. SECW Telecom in Brazil hijacked prefixes of Cloudflare, Google, and BancoBrazil causing some outage for these services in the region.

April 2017. Large chunks of network traffic belonging to MasterCard, Visa, and more than two dozen other financial services companies were briefly routed through a Russian telecom. For several minutes, Rostelecom was originating 50 prefixes for numerous other Autonomous Systems, hijacking their traffic.

August 2017. Google accidentally leaked BGP prefixes it learned from peering relationships, essentially becoming a transit provider instead of simply exchanging traffic between two networks and their customers, causing large-scale internet disruption. It hit Japanese users the hardest, slowing or blocking access to websites and online services for dozens of Japanese companies.

October 2017. Another BGP mishap caused reachability and performance problems for networks such as Twitter, Google, and others. For almost 20 minutes, traffic for many large CDNs was rerouted through Brazil, caused by a BGP leak.BGP mishap caused reachability and performance problems for networks such as Twitter, Google, and others. For almost 20 minutes, traffic for many large CDNs was rerouted through Brazil, caused by a BGP leak.

November 2017. Level 3 BGP routing issues causing large scale network service degradation in North America for slightly more than 90 minutes. Another route leak.

December 2017. Several high-profile sites (Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and Riot Games) were rerouted to a previously unused Russian AS. Two BGP routing incidents only lasted about three minutes each.

Not a single day passed without an incident. While none of the incidents was catastrophic, all of them continue to demonstrate the lack of routing controls like those called for in MANRS that could have prevented them from happening.

This is just a small fraction of what happened in the routing system in 2017. Rather than measure routing security by anecdotal evidence, let’s look at the data.

Routing Incidents

Of the 13,935 total incidents, 62% were classified as outages and 38% were considered routing attacks like route leaks and hijacks.

6,128 Autonomous Systems were involved, which is more that 10% of all announced ASNs on the Internet. If we look at the outages, almost half of them happened to Brazilian operators.[ii]

Let us look to incidents that represent a potential attack, be it malice or a configuration mistake. It is interesting to analyze such routing incidents by the roles a network played – whether it was a victim, a culprit, or an accomplice.

The U.S. ranks first among countries where networks became a victim of an incident, for example when a network’s prefix is hijacked. Last year, that happened 1,193 times in the U.S. It is followed by Brazil (450), India (299), and Russia (242).

Unsurprisingly, the majority of the networks victimized by the most incidents are based in the U.S. In total 3,106 Autonomous Systems were victims of at least one routing incident in 2017.

U.S. and Brazil, followed by Russia and China, lead the list of countries in which networks caused incidents. They are responsible for more that 75% of all incidents. Overall, 1,546 networks caused at least one incident during 2017.

The ranking is different when it comes to the top 10 guilty networks. An interesting case is AS198949 – SecurityDAM, responsible for 54 incidents, mostly prefix hijacks. This is a security provider, offering DDoS attack mitigation among other services. Most probably these incidents were part of attack mitigation actions. Since the BGPStream only registers suspicious routing changes, without knowing intent in some cases it is impossible to distinguish an attack from a legitimate (or consented) routing change.

The U.S. also leads the list of countries with networks that could have prevented an attack, but didn’t, such as not filtering false routing announcements from their customers (one of MANRS Actions). The usual suspects – Russia, Brazil, and China – follow.

In the end, I’d like to note that absolute numbers tell only part of the story. They need to be put into perspective. Countries and networks differ significantly in terms of connected users, announced prefixes, etc. The numbers in this report are not normalized by any of these metrics, but to give an idea, look at a possible one – number of active networks in a country. For example, perhaps the U.S. leads many of these lists simply because there are more networks where incidents could happen. (The “AS’s advertised” chart below comes from data available at

Another point is that it is hard to say whether these numbers are OK, or really bad. Is the system improving or getting worse? The statistics in this report will be a good basis for a trend analysis in years to come.

What You Can Do – Join MANRS

MANRS is a community-driven initiative coordinated by the Internet Society that provides a minimum set of low-cost and low-risk actions that, taken together, can help improve the resilience and security of the routing infrastructure. The more service providers apply these minimum actions, the fewer incidents there will be, and the less damage they can do.

There are four MANRS Actions:

  • Filtering – Ensure the correctness of your own announcements and of announcements from your customers to adjacent networks with prefix and AS-path granularity
  • Anti-spoofing – Enable source address validation for at least single-homed stub customer networks, your own end-users, and infrastructure
  • Coordination – Maintain globally accessible up-to-date contact information
  • Global Validation – Publish your data, so others can validate routing information on a global scale

Maintaining up-to-date filters for customer announcements could mitigate many route leaks. Preventing address squatting could help ward off things like spam and malware. Keeping complete and accurate routing policy data in Internet Routing Registry (IRR) or Resource Public Key Infrastructure (RPKI) repositories is essential for global validation that helps prevent BGP prefix hijacking. Having updated contact information is vital to solving network emergencies quickly.

Let us hope we will see more network operators joining MANRS, and improvements in routing security in 2018. Happy New Year!

[i] BGPStream is an operational tool that tries to minimize false positives, so this number of total incidents may be on the low side.

[ii] This is only counting the number of incidents and not factoring in duration or number of prefixes affected, which may indicate the impact of these incidents.

Another BGP Routing Incident Highlights an Internet Without Checkpoints

Yesterday, there were two BGP routing incidents in which several high-profile sites (Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and Riot Games) were rerouted to a previously unused Russian AS. The incidents only lasted about three minutes each, but demonstrated once again the lack of routing controls like those called for in MANRS that could have prevented this from happening.

As reported in BGPmon’s blog post on 12 December 12, “our systems detected a suspicious event where many prefixes for high profile destinations were being announced by an unused Russian Autonomous System.

Starting at 04:43 (UTC) 80 prefixes normally announced by organizations such Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and Riot Games were now detected in the global BGP routing tables with an Origin AS of 39523 (DV-LINK-AS), out of Russia.”

Either a configuration mistake or a malicious attack, it propagated quickly through the Internet without visible obstacles. This was one of almost 5000 route leaks and hijacks in 11 months of 2017. For comparison, network outages during the same period caused almost 8000 incidents (source:

Routing incidents

In practice, the efficacy of corrective actions strongly depends on the reliability and completeness of information related to expected routing announcements. And these qualities quickly deteriorate with every routing hop on the path. Meaning that the easiest and most effective way to prevent such incidents from a customer is by its direct transit provider. In the case of AS39523 – that is AS31133 (Megafon).The Internet is an interconnected system and its security is only as strong as its weakest link – the least secure network operator. But the concept of “defense in depth” is more applicable here: If a network emits a false routing announcement, there should be many chances to correct it.

Deploying the simple, low-cost, low-risk measures promoted by MANRS is vitally important for all network operators. Had Megafon implemented Action 1 “Prevent propagation of incorrect routing information,” the false announcements yesterday would have been stopped at the first hop. Had reliable data been available about what prefixes DV-LINK-AS is authorised to advertise, others could have prevented that too.

Is your network doing all it can to prevent incidents like this? Read the MANRS document, follow the Implementation Guide, and Join MANRS!

MANRS, Routing Security, and the Brazilian ISP Community

Last week, I presented MANRS to the IX.BR community. My presentation was part of a bigger theme – the launch of an ambitious program in Brazil to make the Internet safer.

While there are many threats to the Internet that must be mitigated, one common point and a challenge for many of them is that the efficacy of the approaches relies on collaboration between independent and sometimes competing parties. And, therefore, finding ways to incentivize and reward such collaboration is at the core of the solutions.

MANRS tries to do that by increasing the transparency of a network operator’s security posture and its commitment to a more secure and resilient Internet. Subsequently, the operator can leverage its increased security posture, signaling it to potential customers and thus differentiating from their competitors.

MANRS also helps build a community of security-minded operators with a common purpose – an important factor that improves accountability, facilitates better peering relationships, and improves coordination in preventing and mitigating incidents.

So, what does the Brazilian ISP community think about routing security and MANRS?

I ran an interactive poll with four questions to provide a more quantitative answer. More than 100 people participated, which makes the results fairly representative.

A sort summary is that while routing incidents are not perceived as the most painful area, the Brazilian ISP community is willing to embrace the collaborative security approach and work on improving Internet infrastructure.

In the past three months, according to BGPSetream, Brazilian ISPs experienced about 1,000 routing events that likely represent incidents. About a quarter of them were route leaks and hijacks; the rest were outages.

From operational experience, 20% of operators dealt with routing security incidents with impact. For the majority, however, such incidents were either infrequent or had little impact. That says something about the perceived risk.

At the same time, improving routing security is important to the vast majority of operators. Almost half are willing to play an active role in promoting best practices.

Almost one-third of respondents already implement the majority of the MANRS Actions and could join the effort.

When it comes to joining the effort, two-thirds feel they would become active adopters of MANRS, once their network has appropriate controls in place.

We look forward to seeing many Brazilian ISPs officially join MANRS, given these survey results! If you’re interested, please let us know. A MANRS Implementation Guide is also available to help you get your network ready.

You can watch Andrei’s full presentation on YouTube in the video below, or at this link.

From ITProPortal: Contributed Post on MANRS and Routing Security

Andrei Robachevsky wrote this contributed blog post for ITProPortal, which was published today at:

It outlines some of the problems with routing security, explains the MANRS actions, discusses the new Research Study we recently completed, and describes how to Join MANRS and get involved.

Please read it and let us know what you think!

Press Release: New Internet Society Research Reveals Disconnect between Enterprises and Service Providers on Crucial Internet Security Fixes

For Immediate Release

Study Indicates that Enterprises Value Internet and Routing Security More than Service Providers Realize

Washington, D.C. – 16 October 2017 – The Internet Society today announced the results of its recent survey conducted through 451 Research, which points to a disconnect between how much enterprises care about Internet security and what service providers think these customers value. These results indicate an unrealized opportunity for service providers to leverage Mutually Agreed Norms for Routing Security (MANRS), the Internet Society-coordinated routing security initiative, to improve their competitive positioning and generate increased revenue. The study shows that although the MANRS initiative is closely aligned with the goals and security expectations of enterprise respondents, some service providers are failing to recognize that congruence and as a result are underserving their customers and missing additional business opportunities.

Undertaken to better understand the attitudes and perceptions of Internet Service Providers and the broader enterprise community around the MANRS initiative, the MANRS Project Study Report revealed a divide between these two groups and potential ways to bridge it. It showed a large number of enterprise respondents (71 percent) stating that security was a core value for their organization. Once introduced to MANRS, almost all enterprise respondents expressed confidence that MANRS actions over time would be either very effective (34 percent) or somewhat effective (64 percent). Most importantly, enterprises showed a willingness to pay a 15 percent premium to support MANRS compliance.

On the other hand, service providers seem to underestimate the value of MANRS. For instance, service providers were asked what they would do if a MANRS requirement arrived as part of an RFP. Only 12 percent said they would plan for implementation, and 16 percent said it would have no impact. The remaining (72 percent) who said such a requirement would spur consideration of MANRS, however, indicate that practical incentives may yet drive greater adoption.

“There is a gap between enterprises and service providers, to be sure, but also an opportunity to engage,” said Andrei Robachevsky, Technology Programme Manager for the Internet Society. “As they seek out security-minded providers, enterprises could also put MANRS compliance into their RFPs, and for their part, service providers can market compliance with MANRS as a business differentiator. By committing to being held accountable by the Internet community and doing good, they can also align with customer concerns, capture a premium and do well.”

Behind the large number of enterprises who see security as a core value is the growing prominence of the Internet side of business and media coverage of security breaches. Asked about specific threats, enterprise respondents ranked traffic routing, interception, and hijacking at the top of the list (at 74 percent), with DDoS and address spoofing tied for second place (at 57 percent) and concerns over 24×7 Internet service availability and blacklisting following thereafter. While MANRS is not a one-stop solution to all of the Internet’s routing challenges, many enterprises appear to agree that its recommended actions in route filtering, anti-spoofing, coordination, and global validation are important steps in the right direction toward a globally robust and secure routing infrastructure. In addition to revealing a willingness to support MANRS compliance with a 15 percent (median value) price increase, the survey showed that 13 percent of enterprise respondents would only select a provider that was MANRS-compliant in a competitive situation.

“The bottom line impact is real,” said 451 Research Chief Analyst Eric Hanselman and report author. “Our expectation is that MANRS compliance could translate into additional value, just in the procurement process, for instance, through minimization of the discounting required to win contracts, with as much as a 7 percent long-term revenue increase for providers who are able to leverage the MANRS branding as part of the selling process.”

In looking to the future, the MANRS Project Study Report identifies more possibilities. Already trusted by enterprise customers who are lacking cybersecurity resources, service providers could gain additional revenue by adding MANRS-derived services to their portfolio. Anti-spoofing controls that log activity, for instance, can be used to generate periodic reports for customers. These reports can be part of an intelligence feed that alerts customers to misconfigurations or potential attacks. Appropriately automated, this type of service can provide additional customer binding, in additional to generating revenue.

Given all the potential additional revenue, service providers can realize a strong return on a relatively small investment in the four MANRS actions, which represent a lowest common denominator of security measures to increase overall routing security. While the survey indicated that some service provider respondents think that implementation could be disruptive, compared to general routing security practices, all MANRS actions are intended to have low risk and low cost. More details on becoming MANRS compliant can be found in the MANRS Implementation Guide. Service providers who are already compliant can join the MANRS effort here and may download the MANRS badge for their sales and marketing materials here.

For more information, read the full MANRS Project Study Report.

About the Internet Society
Founded by Internet pioneers, the Internet Society (ISOC) is a non-profit organization dedicated to ensuring the open development, evolution, and use of the Internet. Working with a global community of chapters and members, the Internet Society collaborates with a broad range of groups to promote the technologies that keep the Internet safe and secure, and advocates for policies that enable universal access. The Internet Society is also the organizational home of the Internet Engineering Task Force (IETF).


Internet Society Contact:
Megan Kruse
Manager, Technology Outreach and Strategic Planning
Internet Society

Media Contact:
Andrea Maclean
Wireside Communications®
For the Internet Society

New Study: Understanding MANRS’ Potential for Enterprises and Service Providers

MANRS was founded with the ambitious goal of improving the security and reliability of the global Internet routing system, based on collaboration among participants and shared responsibility for Internet infrastructure. These are undoubtedly essential pillars supporting the Internet’s tremendous growth and success, but we must better articulate the incentives of contributing to global security and resilience to grow MANRS participation and reach our goals.

To do so, we engaged 451 Research to understand the attitudes and perceptions of Internet service providers and the broader enterprise community around MANRS and how it relates to their organizations. The results of the study are documented in the report:

The study results demonstrate considerable unrealized potential for MANRS, showing that enterprises are interested in security and their interest should be a strong incentive for more service providers to participate. Market education could be particularly effective in overcoming the operational inertia that many providers face.

The key points from the study are:

  • While MANRS itself is not well known by enterprises, its attributes are highly valued.
  • Enterprises have high expectations for MANRS efforts.
  • Enterprise perceptions of MANRS can translate into increased revenue for service providers.
  • Existing MANRS actions cover a reasonable set of controls.
  • There are options to extend the MANRS actions for some providers.

While there have been challenges in creating a dramatic increase in MANRS adoption, the study shows there is solid alignment between the motivations of service providers and the aspirations of enterprises.

We encourage you to read the entire report and let us know what you think! We hope that with additional effort, bringing these two together could create a bright future for MANRS.

Verisign joins MANRS to further security, stability and resiliency of the internet routing system

Verisign, a renowned security solutions provider and a DNS registry and root server operator, demonstrated its commitment to ensuring that the global routing system becomes more secure by joining Mutually Agreed Norms for Routing Security (MANRS) today.

To create a sustainable technical and business environment, organizations must work together to address the challenges of the Internet’s routing system. Deploying small measures, like those defined in the MANRS Actions, can make a big difference. MANRS provides added value for the network operator and its customers: better protection against traffic anomalies caused by misconfigurations; cleaner setups resulting in easier troubleshooting and lower time-to-resolution (TTR); improved peering conditions; and opportunities for valuable collaboration with other operators through a discussion forum and professional network. And many MANRS participants go beyond these baseline actions, leading the group of participants and encouraging further collaboration.

“As the registry operator for .com and .net, root server operator for the A and J roots, and root zone maintainer, Verisign is deeply committed to ensuring the security, stability and resiliency of the internet. Routing security is of the utmost importance, and we are pleased to support MANRS, as we have since its inception, in its goal toward promoting a culture of collective responsibility, collaboration and coordination among our peers in the global internet routing system,” said Frank Scalzo, Director, Security Strategy.

We are looking for more security leaders – networks that have already implemented the MANRS recommendations and much more – to sign up, support this effort, and encourage others! A new MANRS Implementation Guide is also available to help organizations deploy the Actions and get started.

Are you a network expert? Please participate in a MANRS configuration survey and share your knowledge!

MANRS defines 4 Actions, which are really the building blocks for simple use cases. We believe that even if these measures are implemented widely, security and resilience of the Internet routing system will significantly improve. The minimum baseline that MANRS defines also allows networks to build on, implementing routing security in more complex network topologies.

But it is easier to say then to implement. To facilitate implementation of MANRS Actions the community have developed a BCOP document – a MANRS implementation guide.

In order to make it more practical and useful we need actual configuration examples from most commonly used vendors and equipment models.

We created a simple survey to collect this information. It should take only 10-15 min to complete if you know how to configure these things. For some of the questions there are example, to give you an idea of what is expected.

Please contribute to the survey and share knowledge:

Your participation is highly appreciated!