Lesson Learned: Twitter Shored Up Its Routing Security

With so many eyes on networks and traffic flows around Ukraine and Russia, any misleading announcement is quickly investigated. That’s what happened yesterday, when AS8342 (RTComm) started announcing 104.244.42.0/24, a prefix assigned to Twitter Inc.

Doug Madory from Kentik highlighted it in this post:

From 12:05-12:50 UTC, RU telecom RTComm (AS8342) hijacked a prefix (104.244.42.0/24) belonging to Twitter.The hijack didn't propagate far due to a RPKI ROA which asserted AS13414 was the rightful origin.This is the same prefix hijacked during the coup in Myanmar last year.

The same incident was also picked up by the Georgia Tech BGP Observatory:

Georgia Tech BGP Observatory incident report

A nearly identical incident happened on 5 February 2021, when an ISP in Myanmar tried to hijack the exact same prefix. Here is our detailed analysis of that incident. Why this same prefix again? Twitter’s DNS A records point to this prefix, so disrupting traffic toward it would potentially disrupt traffic toward Twitter.

This time, though, it went differently, because Twitter made some key changes.

These are all the DNS A records for twitter.com:

twitter.com.                 776      IN        A         104.244.42.193

twitter.com.                 776      IN        A         104.244.42.65

twitter.com.                 763      IN        A         104.244.42.129

twitter.com.                 1400    IN        A         104.244.42.1

Per RIPE RIS data, AS8342 started originating 104.244.42.0/24 on Monday, 28 March 2022 at 12:06:11 (UTC).

1648469171.000000 104.244.42.0/24 199524 8359 8342
1648469171.000000 104.244.42.0/24 28917 8359 8342
1648469171.000000 104.244.42.0/24 31027 8359 8342
1648469171.000000 104.244.42.0/24 57695 60068 8359 8342
1648469171.000000 104.244.42.0/24 8359 8342
1648469174.000000 104.244.42.0/24 199524 8359 8342
1648469174.000000 104.244.42.0/24 8660 1267 8359 8342
1648469176.000000 104.244.42.0/24 25091 8359 8342
1648469178.000000 104.244.42.0/24 56665 8359 8342
1648469184.000000 104.244.42.0/24 59414 13030 8359 8342
1648469184.000000 104.244.42.0/24 8359 8342
1648469185.000000 104.244.42.0/24 21232 13030 8359 8342
1648469185.000000 104.244.42.0/24 25091 8359 8342
1648469191.000000 104.244.42.0/24 59605 8359 8342
1648469193.000000 104.244.42.0/24 41095 8359 8342
1648469195.000000 104.244.42.0/24 15435 8359 8342
1648469196.000000 104.244.42.0/24 8607 8359 8342
1648469199.000000 104.244.42.0/24 137409 8359 8342
1648469199.000000 104.244.42.0/24 25091 8359 8342
1648469199.000000 104.244.42.0/24 9304 8359 8342
1648469200.000000 104.244.42.0/24 22652 8359 8342
1648469201.000000 104.244.42.0/24 137409 8359 8342
1648469202.000000 104.244.42.0/24 19151 8359 8342
1648469207.000000 104.244.42.0/24 6720 8447 8359 8342
1648469207.000000 104.244.42.0/24 8660 1267 8359 8342
1648469210.000000 104.244.42.0/24 51185 8359 8342
1648471814.000000 104.244.42.0/24 31027 8359 8342
1648471824.000000 104.244.42.0/24 199524 8359 8342
1648471844.000000 104.244.42.0/24 19151 8359 8342

AS8342 is connected to DataIX Internet Exchange, and its direct upstreams as per bgp paths are AS12389 (Rostelecom), AS8920 (RTComm), AS8359 (MTS), AS25091 (IP-Max), AS6939 (Hurricane Electric) etc. The above bgp update messages show that the propagation happened through AS8359 (MTS) only, while others filtered those erroneous/malicious BGP announcements successfully.

MTS (Mobile TeleSystems PJSC) is a very well-connected network, and it propagated this route across the global routing table. Interestingly, this MTS looking glass shows that routers in Moscow have the individual IPs related to Twitter’s DNS A records in the routing table with a /32 mask originated by a private ASN, meaning they are actively redirecting Twitter traffic internally.

MTS looking glass shows that routers in Moscow have the individual IPs related to Twitter’s DNS A records in the routing table with a /32 mask originated by a private ASN

RIPE RIS data suggest that this hijacked announcement was accepted, via MTS, by many networks and then further propagated to their neighbours, but it failed to propagate globally and didn’t create a global outage.

The reason behind this is Twitter embracing RPKI.

A Route Origin Authorisation (ROA) is a cryptographically signed object that states which AS is authorized to originate a particular IP address prefix or set of prefixes. ROAs are important protection from mis-origination.

Below are some of the routes originated by Twitter that are covered by a valid ROA.

ASN	Route		 RPKI Status
13414   69.195.190.0/24 Valid
13414   69.195.190.0/24 Valid
13414   69.195.190.0/24 Valid
13414   69.195.191.0/24 Valid
13414   69.195.191.0/24 Valid
13414   104.244.40.0/24 Valid
13414   104.244.47.0/24 Valid
13414   104.244.44.0/24 Valid
13414   104.244.41.0/24 Valid
13414   104.244.42.0/24 Valid
13414   69.195.190.0/24 Valid
13414   69.195.190.0/24 Valid
13414   69.195.190.0/24 Valid
13414   104.244.47.0/24 Valid
13414   104.244.44.0/24 Valid
13414   199.59.148.0/22 Valid
13414   104.244.45.0/24 Valid
13414   104.244.42.0/24 Valid
13414   69.195.191.0/24 Valid
13414   69.195.191.0/24 Valid
13414   104.244.40.0/24 Valid
13414   199.16.156.0/23 Valid
13414   199.16.156.0/22 Valid
13414   104.244.46.0/24 Valid
13414   199.59.148.0/22 Valid
13414   104.244.45.0/24 Valid
13414   199.16.156.0/23 Valid
13414   199.16.156.0/22 Valid
13414   104.244.46.0/24 Valid
13414   104.244.41.0/24 Valid

Many network operators have implemented Route Origin Validation (ROV) and will not accept any routes with an “Invalid” ROA status. The network 104.244.42.0/24 originated by AS8342 was considered invalid since its data didn’t match the Twitter-created ROA. The route was not accepted by many operators because they filtered it and prevented it from propagating further to the whole routing table.

Another encouraging aspect is that Twitter has created ROAs for most (but not all) of its routes. The table below shows when they started creating ROAs.

ASN	Prefix		Max Length	Date Created
13414   69.195.190.0/24 24      2021-08-10T04:00:00
13414   69.195.191.0/24 24      2021-08-10T04:00:00
13414   69.195.190.0/23 24      2021-08-10T04:00:00
13414   69.195.190.0/23 23      2021-08-10T04:00:00
13414   69.195.191.0/24 24      2021-08-10T04:00:00
13414   69.195.190.0/24 24      2021-09-01T04:00:00
13414   69.195.190.0/23 24      2021-09-01T04:00:00
13414   104.244.42.0/24 24      2021-11-25T05:00:00
13414   104.244.44.0/24 24      2021-11-25T05:00:00
13414   199.16.159.0/24 24      2021-11-25T05:00:00
13414   69.195.190.0/24 24      2021-11-25T05:00:00
13414   199.59.148.0/24 24      2021-11-25T05:00:00
13414   199.59.151.0/24 24      2021-11-25T05:00:00
13414   199.16.156.0/22 24      2021-11-25T05:00:00
13414   104.244.40.0/24 24      2021-11-25T05:00:00
13414   199.16.157.0/24 24      2021-11-25T05:00:00
13414   69.195.191.0/24 24      2021-11-25T05:00:00
13414   69.195.190.0/23 24      2021-11-25T05:00:00
13414   199.59.149.0/24 24      2021-11-25T05:00:00
13414   104.244.47.0/24 24      2021-11-25T05:00:00
13414   199.16.156.0/23 24      2021-11-25T05:00:00
13414   104.244.45.0/24 24      2021-11-25T05:00:00
13414   199.16.156.0/24 24      2021-11-25T05:00:00
13414   104.244.41.0/24 24      2021-11-25T05:00:00
13414   104.244.46.0/24 24      2021-11-25T05:00:00
13414   199.59.150.0/24 24      2021-11-25T05:00:00
13414   199.59.148.0/22 24      2021-11-25T05:00:00
13414   199.16.158.0/24 24      2021-11-25T05:00:00
13414   69.195.190.0/24 24      2021-12-14T05:00:00
13414   104.244.43.0/24 24      2022-01-06T05:00:00

Just like the February 2021 incident, the incident yesterday was likely a failed attempt to block a service in the country or for their own network, and it lead to accidentally leaking those routes to the global routing table. Thankfully, the industry has learned some valuable lessons from the past and is taking strong steps to fix problems like these. Last year, Twitter had not created ROAs for any of its IP resources. This year it has, and it is now much more difficult for a bogus announcement to propagate.

It is extremely important for network operators to implement effective route filtering based on verifiable information about which networks are legitimately authorised to originate which number resources (AS numbers and IP prefixes). This is what RPKI was designed for.

Having a valid Route/Route6 Object is important, but these days when many major network operators are doing ROV, it is even more important to have a valid ROA for all your resources.

This is what MANRS has been promoting. MANRS is an industry-supported initiative that builds on well-established best practices by bringing together actions that can address the most common threats in the global routing system. With more and more operators joining MANRS, incidents like these will have smaller and smaller impact.

Join us.

Leave a Comment