Taken as a basis TOP 1M domains of Alexa: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip The test machine: Virtual machine Intel(R) Core(TM) i7 CPU @ 3.4 GHz with 2 dedicated cores, 3 Gb RAM and bandwidth 100 mbit/s. OS Windows 7 x64. Task: Spoiler: Preset Code: eyJwcmVzZXQiOiJcdTA0MWZcdTA0NDBcdTA0M2VcdTA0MzJcdTA0MzVcdTA0NDBc dTA0M2FcdTA0MzAgXHUwNDIyXHUwNDFlXHUwNDFmIDFcdTA0MWMgXHUwNDEwXHUw NDNiXHUwNDM1XHUwNDNhXHUwNDQxXHUwNDRiIFx1MDQzZFx1MDQzMCBcdTA0M2Rc dTA0MzBcdTA0M2JcdTA0MzhcdTA0NDdcdTA0MzhcdTA0MzUgXHUwNDNjXHUwNDNl XHUwNDMxXHUwNDM4XHUwNDNiXHUwNDRjXHUwNDNkXHUwNDNlXHUwNDM5IFx1MDQz Mlx1MDQzNVx1MDQ0MFx1MDQ0MVx1MDQzOFx1MDQzOCIsInZhbHVlIjp7InByZXNl dCI6Ilx1MDQxZlx1MDQ0MFx1MDQzZVx1MDQzMlx1MDQzNVx1MDQ0MFx1MDQzYVx1 MDQzMCBcdTA0MjJcdTA0MWVcdTA0MWYgMVx1MDQxYyBcdTA0MTBcdTA0M2JcdTA0 MzVcdTA0M2FcdTA0NDFcdTA0NGIgXHUwNDNkXHUwNDMwIFx1MDQzZFx1MDQzMFx1 MDQzYlx1MDQzOFx1MDQ0N1x1MDQzOFx1MDQzNSBcdTA0M2NcdTA0M2VcdTA0MzFc dTA0MzhcdTA0M2JcdTA0NGNcdTA0M2RcdTA0M2VcdTA0MzkgXHUwNDMyXHUwNDM1 XHUwNDQwXHUwNDQxXHUwNDM4XHUwNDM4IiwicGFyc2VycyI6W1siTmV0OjpIVFRQ IiwiZGVmYXVsdCIseyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InF1ZXJ5Zm9ybWF0 IiwidmFsdWUiOiJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9wYWdlc3BlZWRv bmxpbmUvdjNiZXRhMS9tb2JpbGVSZWFkeT9rZXk9QUl6YVN5RGtFWC1mMUpOTFFM QzE2NFNaYW9iQUxxRnY0UEhWLWtBJnNjcmVlbnNob3Q9ZmFsc2Umc25hcHNob3Rz PWZhbHNlJmxvY2FsZT1lbl9VUyZ1cmw9aHR0cCUzQSUyRiUyRiRxdWVyeS5kb21h aW4lMkYmc3RyYXRlZ3k9bW9iaWxlJmZpbHRlcl90aGlyZF9wYXJ0eV9yZXNvdXJj ZXM9ZmFsc2UmY2FsbGJhY2s9X2NhbGxiYWNrc18uX2RjdXZiQnZuSVhWQiJ9LHsi dHlwZSI6ImZpbHRlciIsInJlc3VsdCI6ImRhdGEiLCJmaWx0ZXJUeXBlIjoiY29u dGFpbiIsInZhbHVlIjoiXCJwYXNzXCI6IGZhbHNlIiwib3B0aW9uIjoic2VucyJ9 XV0sInJlc3VsdHNGb3JtYXQiOiIkcXVlcnkuZG9tYWluXFxuIiwicmVzdWx0c1Nh dmVUbyI6ImZpbGUiLCJyZXN1bHRzRmlsZU5hbWUiOiJ0b3AtMW0td2l0aG91dC1t b2JpbGUtdmVyLnR4dCIsImFkZGl0aW9uYWxGb3JtYXRzIjpbXSwicmVzdWx0c1Vu aXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiJHF1ZXJ5Il0sInVuaXF1ZVF1ZXJp ZXMiOmZhbHNlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0 aW9ucyI6eyJvbkFsbExldmVscyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0 ZXJhdG9yIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7Im92ZXJ3cml0ZSI6ZmFs c2V9LCJkb0xvZyI6Im5vIiwia2VlcFVuaXF1ZSI6Ik5vIiwibW9yZU9wdGlvbnMi OmZhbHNlLCJyZXN1bHRzUHJlcGVuZCI6IiIsInJlc3VsdHNBcHBlbmQiOiIiLCJx dWVyeUJ1aWxkZXJzIjpbeyJzb3VyY2UiOiJxdWVyeSIsInR5cGUiOiJzdHJpbmdT cGxpdCIsInNlcGFyYXRvciI6IiwiLCJ0byI6WyJyYW5rIiwiZG9tYWluIl19XSwi cmVzdWx0c0J1aWxkZXJzIjpbXSwiY29uZmlnT3ZlcnJpZGVzIjpbXX19 The source file with requests contains data in the format <alexa-rank>, <domain>, a million entries, each domain on a new line. Using the Query Builder separate the domain and its rank. Used parser Net::HTTP, to which is input the link with inserted domain to check for the mobile version. Mobile version check here: https://www.google.com/webmasters/tools/mobile-friendly/ Using a proxy is enabled. The result, which in this case is all domains that do not have a mobile version, are saved to the file top-1m-without-mobile-ver.txt Working task: Statistics: The speed of parsing was about 2000 domains per minute Failed requests just 26 of 1000000 Parsing passed in 1000 threads Elapsed time: 8 hours 7 minutes Result: 40.98% of the million most visited sites do not have a mobile version! File with the list of websites without mobile versions: http://files.a-parser.com/top-1m-without-mobile-ver.txt
