Monday, November 18, 2013

Weird Attack

My AWS bill is usually $5/month (for a reserved micro). This time it was $10. That extra $5 was data transfer charges. For about a month I was serving an extra 14 GB per week which was almost exclusively GETs for holtstrom.com/michael/sites.php (which is 301 redirected from holtstrom.com/michael). That's 5 million unexpected GETs a week. Note that the images and css files referenced from that page weren't fetched. Also scripts on that page weren't executed or the hits would have appeared in my google analytics (which they didn't).

So, what was going on? Presumably a bot was hammering that single page. Why? It wasn't sufficient load to noticeably impact service from a micro, so it's not DOS. It was just a single page so it wasn't crawling. Strangely almost all the attack hits listed urls from egycivilization.com as their referrer. That site is obviously a fake blog built by a robot presumably attempting to generate ad revenue. Can we believe the referrer? If you were going to attack a site wouldn't you leave the referrer blank?

IP-limiting won't help here because of the 14GB/week, 5million GETs/week were distributed across 20,000 distinct IPs, no single IP making more than 4k GETs/week. I myself sometimes generate 7k GETs/week. I could IP rate-limit with exclusions for certain IPs, 4k GETs/week doesn't seem unreasonable for a single IP.

A few million hits doesn't seem like enough to brute force a session cookie. Maybe the were trying to find a server exploit? I'll rebuild my instance just to be safe, but at this point I think either a bot got stuck in a loop or someone was practicing a DDOS attack on a random target.

Long Story

What happens when AWS says you're transferring much more data then you expect?

Zip up the logs:

cd /
sudo tar -zcf /jail/tmp/logs.tar.gz jail/var/log/lighttpd

Use WinSCP to copy /jail/tmp/logs.tar.gz to C:\Temp

From the options at cyberciti, I chose GNU Utilities For Win32, although that link is from 2003 so I fetched it from sourceforge instead. I just unziped to C:\Program Files (x86)\UnxUtils\ then from a regular DOS console you can run commands like C:\Program Files (x86)\UnxUtils\usr\local\wbin\tar

Extract logs:

cd C:\Temp
"C:\Program Files (x86)\UnxUtils\usr\local\wbin\gzip" -d logs.tar.gz
"C:\Program Files (x86)\UnxUtils\usr\local\wbin\tar" -xf logs.tar

To convert to windows line breaks create C:\Temp\jail\var\log\lighttpd\fix.bat

for %%z in (*.log) do (for /f "delims=" %%i in (%%z) do @echo %%i)>%%z.fixed

Then run it.

cd /d C:\Temp\jail\var\log\lighttpd
fix

That takes a long time. Move those fixed files to C:\Temp to make the following easier.

Download and install Microsoft's LogParser.

cd "C:\Program Files (x86)\Log Parser 2.2"

First prove we can look at a simple log.

logparser "select count(logfilename) from C:\Temp\access.log.fixed" -i:NCSA -q:ON

2773

logparser "select * from C:\Temp\access.log.fixed" -i:NCSA

LogFilename              LogRow RemoteHostName RemoteLogName             UserName DateTime                     Request                                              StatusCode BytesSent Referer User-Agent                                                                                           Cookie
------------------------ ------ -------------- ------------------------- -------- ---------------------------- ---------------------------------------------------- ---------- --------- ------- ---------------------------------------------------------------------------------------------------- ------
C:\Temp\access.log.fixed 1      66.249.75.89   holtstrom.com             -        [17/Nov/2013:04:51:30 +0000] GET / HTTP/1.1                                       200        1625      -       DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) -
C:\Temp\access.log.fixed 2      65.55.24.244   glendeningsporthorses.com -        [17/Nov/2013:04:52:41 +0000] GET / HTTP/1.1                                       200        7163      -       Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)                              -
C:\Temp\access.log.fixed 3      178.255.215.66 www.tribalworker.com      -        [17/Nov/2013:04:53:19 +0000] GET /robots.txt HTTP/1.1                             302        0         -       Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)                                -
C:\Temp\access.log.fixed 4      178.255.215.66 www.tribalworker.com      -        [17/Nov/2013:04:53:19 +0000] GET / HTTP/1.1                                       200        15300     -       Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)                                -
C:\Temp\access.log.fixed 5      178.255.215.66 www.tribalworker.com      -        [17/Nov/2013:04:53:20 +0000] GET /michael/lifecounters/list.php?x=36&o=2 HTTP/1.1 200        112979    -       Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)                                -
C:\Temp\access.log.fixed 6      119.63.193.194 pleasantviewdental.com    -        [17/Nov/2013:04:56:31 +0000] GET / HTTP/1.1                                       200        5994      -       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)                                                   -
C:\Temp\access.log.fixed 7      202.46.51.46   pleasantviewdental.com    -        [17/Nov/2013:04:56:45 +0000] GET / HTTP/1.1                                       200        5994      -       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)                                                   -
C:\Temp\access.log.fixed 8      199.30.16.125  www.tribalworker.com      -        [17/Nov/2013:04:59:49 +0000] GET /mw/files/pics/latin1.gif HTTP/1.1               200        36461     -       Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b            -
C:\Temp\access.log.fixed 9      134.134.139.74 holtstrom.com             -        [17/Nov/2013:05:04:09 +0000] GET /michael/base/topbar.js HTTP/1.1                 304        0         -       Mozilla/4.0 (compatible;)                                                                            -
C:\Temp\access.log.fixed 10     147.32.226.54  holtstrom.com             -        [17/Nov/2013:05:06:42 +0000] GET /michael/tools/asn1decoder.php HTTP/1.1          200        32950     -       Opera/9.80 (Windows NT 6.1; Win64; x64) Presto/2.12.388 Version/12.16                                -
Press a key...
logparser "select datetime from C:\Temp\access.log.fixed order by datetime" -i:NCSA

[17/Nov/2013:04:51:30 +0000]

logparser "select datetime from C:\Temp\access.log.fixed order by datetime desc" -i:NCSA

[18/Nov/2013:12:38:47 +0000]
logparser "select RemoteHostName, sum(BytesSent) as sumbytes from C:\Temp\access.log.fixed group by RemoteHostName order by sumbytes desc" -i:NCSA -q:ON | more

192.184.52.170  6187705
24.146.28.130   3402656
192.0.134.34    3217799
66.249.73.213   1358009
124.128.236.189 1269997
178.238.235.184 998596
74.91.24.66     948997
174.255.96.31   738473
198.254.172.188 690564
37.233.27.142   664386
147.32.226.54   653913
209.85.238.59   534710
5.10.83.76      503062
122.248.112.128 439147
61.227.154.232  429566
94.153.9.210    413001
5.10.83.23      374867
165.154.33.194  316716
186.213.68.130  310410
logparser "select Request, sum(BytesSent) as sumbytes from C:\Temp\access.log.fixed group by Request order by sumbytes desc" -i:NCSA -q:ON | more

GET /michael/img/background.jpg HTTP/1.1                                          5115264
GET /michael/blog/cmd_slideshow.php HTTP/1.1                                      1925474
GET / HTTP/1.1                                                                    1839705
GET /michael/lifecounters/index.php?x=0&y=1296259200 HTTP/1.1                     1636809
GET /michael/chalkboards/index.php HTTP/1.0                                       1307040
GET /michael/blog-files/2005-08-01-Masters-Thesis.pdf HTTP/1.1                    1269997
GET /michael/lifecounters/ HTTP/1.1                                               1207085
GET /michael/base/util.js HTTP/1.1                                                1107280
GET /michael/chalkboards/ HTTP/1.0                                                1012956
POST /michael/lifecounters/ HTTP/1.1                                              994070
GET /font/glyphs/webhostinghub-glyphs.ttf HTTP/1.1                                840223
GET / HTTP/1.0                                                                    792108
GET /michael/base/stylesglobal.css HTTP/1.1                                       650946
GET /michael/blog/post.php HTTP/1.0                                               604576
GET /michael/photos/ HTTP/1.1                                                     593092
GET /michael/home.php HTTP/1.1                                                    578240
GET /public/profile/newuser.php HTTP/1.0                                          537828
GET /img/background.jpg HTTP/1.1                                                  521268
GET /mw/files/pics/fish/WhiteSeaUrchin.jpg HTTP/1.1                               500542
GET /michael/tools/asn1decoder.php HTTP/1.1                                       494250
GET /michael/blog/post.php HTTP/1.1                                               451863
GET /michael/punchclocks/index.php?x=8 HTTP/1.1                                   370746
GET /michael/sites.php HTTP/1.1                                                   343205
GET /img/banner5.jpg HTTP/1.1                                                     341584
logparser "select Referer, sum(BytesSent) as sumbytes from C:\Temp\access.log.fixed group by Referer order by sumbytes desc" -i:NCSA -q:ON | more

-                                                                                             23301786
http://frothing.com/                                                                           1895368
http://www.jonsnelson.com/slideshow.php                                                        1788844
http://www.tribalworker.com/michael/lifecounters/                                               994070
http://holtstrom.com/michael/tools/asn1decoder.php                                              973873
http://holtstrom.com/michael/base/stylesglobal.css                                              971143
http://www.tribalworker.com/michael/chalkboards/                                                947739
http://www.tribalworker.com/index.php                                                           805377
http://www.tribalworker.com/                                                                    797484
http://www.tribalworker.com/michael/chalkboards/index.php                                       702588
http://glendeningsporthorses.com/Farm                                                           685461
http://holtstrom.com/michael/blog/post.php                                                      566790
http://holtstrom.com/michael/blog/post/249/Migrate-to-SSD.html                                  564163
http://holtstrom.com/michael/blog/post/386/Amazon-EC2-for-Enterprise.html                       538323
http://www.tribalworker.com/public/profile/newuser.php                                          505300
http://holtstrom.com/michael/blog/post/408/Planning-Iceland.html                                492807
http://holtstrom.com/michael/blog/post/390/Migrate-EC2-to-a-different-Availability-Zone.html    447868
http://holtstrom.com/michael/blog/post/283/Paying-for-your-Amazon-EC2-Micro-Instance.html       447868
https://holtstrom.com/michael/blog/post.php                                                     408858
http://holtstrom.com/michael/tools/javatohex.php                                                368328
http://evilgoblin.com/                                                                          335718
http://holtstrom.com/                                                                           316548
http://holtstrom.com/michael/blog/post/234/Design-of-a-Simple-RISC-Processor.html               314389
http://holtstrom.com/michael/blog/post/62/Customizing-Blogger-Template.html                     276574
http://holtstrom.com/michael/blog/post/225/Apache-2.2-Proxy.html                                272937
http://holtstrom.com/michael/tools/onlinehexdump.php                                            247590
http://www.jonsnelson.com/                                                                      241387

Now for the big log...

logparser "select count(logfilename) from C:\Temp\access.log-20131027.fixed" -i:NCSA -q:ON

5,314,097

logparser "select sum(BytesSent) from C:\Temp\access.log-20131027.fixed" -i:NCSA -q:ON

14,678,977,093

logparser "select datetime from C:\Temp\access.log-20131027.fixed order by datetime" -i:NCSA

[20/Oct/2013:03:45:03 +0000]

logparser "select datetime from C:\Temp\access.log-20131027.fixed order by datetime desc" -i:NCSA

[27/Oct/2013:03:34:58 +0000]
logparser "select RemoteHostName, sum(BytesSent) as sumbytes from C:\Temp\access.log-20131027.fixed group by RemoteHostName order by sumbytes desc" -i:NCSA -q:ON | more

108.162.170.182 25,539,474
196.219.44.178  12,429,027
216.191.251.39  11,823,479
66.249.75.89    11,731,368
87.109.98.204   10,715,955
180.87.42.21    10,058,475
192.114.71.13    9,239,957
64.187.228.74    9,226,989
217.23.12.71     8,603,530
91.121.115.186   8,486,725
210.129.199.34   8,450,091
91.236.75.108    7,985,477
85.25.199.87     7,299,621
192.187.121.107  6,895,436
64.30.120.111    6,638,046
108.16.234.220   6,369,807
208.100.19.186   6,169,827
172.246.115.228  6,086,592
200.6.100.160    6,042,644
86.171.18.89     5,861,500
210.140.145.105  5,821,527
76.66.157.77     5,707,391
63.141.224.243   5,616,072
50.112.152.56    5,576,823
logparser "select Request, sum(BytesSent) as sumbytes from C:\Temp\access.log-20131027.fixed group by Request order by sumbytes desc" -i:NCSA -q:ON | more

GET /michael/sites.php HTTP/1.1                                           14,184,148,511
GET /michael/sites.php HTTP/1.0                                               70,322,975
GET /michael/img/background.jpg HTTP/1.1                                      44,699,096
GET /michael/chalkboards/ HTTP/1.0                                            25,191,174
GET /michael/chalkboards/index.php HTTP/1.0                                   16,697,790
GET /michael/blog/cmd_slideshow.php HTTP/1.1                                  15,519,379
GET / HTTP/1.1                                                                13,274,856
GET /michael/blog-files/2005-08-01-Masters-Thesis.pdf HTTP/1.1                12,354,710
GET /michael/base/util.js HTTP/1.1                                            10,491,843
GET / HTTP/1.0                                                                 8,818,311
GET /michael/chalkboards/ HTTP/1.1                                             7,809,684
GET /michael/lifecounters/ HTTP/1.1                                            6,745,656
POST /michael/lifecounters/ HTTP/1.1                                           6,532,552
GET /img/bak5.jpg HTTP/1.1                                                     6,219,115
GET /img/bak4.jpg HTTP/1.1                                                     5,912,883
GET /michael/base/stylesglobal.css HTTP/1.1                                    5,505,104
GET /public/profile/newuser.php HTTP/1.0                                       5,394,079
GET /michael/chalkboards/index.php?x=26 HTTP/1.0                               5,192,944
GET /michael/blog/post/426/Windows7-At-Home.html HTTP/1.1                      5,070,825
GET /michael/tools/asn1decoder.php HTTP/1.1                                    4,961,184
GET /img/bak1.jpg HTTP/1.1                                                     4,751,142
GET /michael/blog/post.php HTTP/1.0                                            4,744,826
GET /img/bak2.jpg HTTP/1.1                                                     4,143,821
GET /img/bak3.jpg HTTP/1.1                                                     3,727,464
GET /tajenkin/punchclocks/index.php?x=49 HTTP/1.1                              3,531,844
logparser "select Referer, sum(BytesSent) as sumbytes from C:\Temp\access.log-20131027.fixed group by Referer order by sumbytes desc" -i:NCSA -q:ON | more

http://www.egycivilization.com/                                                    5,906,314,148
http://www.egycivilization.com/search/label/Mortgage                                 305,215,774
http://www.egycivilization.com/2013/10/how-to-apportion-your-google-adsense-ads.html 236,485,914
http://www.egycivilization.com/2013/10/getting-started-with-google-adwords-ppc.html  231,056,486
http://www.egycivilization.com/2013/10/google-adwards-keyword-means.html             229,955,411
http://www.egycivilization.com/2013/10/google-ad-words-spy-on-your-ppc.html          222,584,452
http://www.egycivilization.com/2013/10/the-blogging-phenomenon-explained.html        217,436,752
http://www.egycivilization.com/search/label/google%20Adword                          215,647,428
http://www.egycivilization.com/search/label/blog                                     213,100,167
http://www.egycivilization.com/2013/10/what-to-visage-for-in-mortgage-broker.html    163,933,191
http://www.egycivilization.com/2013/10/which-oshkosh-mortgage-investor-to.html       162,785,478
http://www.egycivilization.com/2013/10/when-should-you-refinance-mortgage-do.html    161,572,455
http://www.egycivilization.com/search/label/search%20engine                          161,344,435
http://www.egycivilization.com/2013/10/womens-wear-things-to-cognise-before.html     156,214,008
http://www.egycivilization.com/search/label/Forex                                    151,251,781
http://www.egycivilization.com/search/label/SOCIAL                                   151,169,482
http://www.egycivilization.com/search/label/Internet%20Marketing                     150,833,682
http://www.egycivilization.com/search/label/domain                                   150,686,876
http://www.egycivilization.com/search/label/making%20money                           149,815,365
http://www.egycivilization.com/search/label/Email%20marketing                        149,486,052
-                                                                                    149,474,945
http://www.egycivilization.com/search/label/Site%20Promotion                         145,719,240
logparser "select sum(BytesSent) from C:\Temp\access.log-20131027.fixed where Referer like '%egycivilization%'" -i:NCSA -q:ON

14,251,032,690

And the next one...

logparser "select count(logfilename) from C:\Temp\access.log-20131103.fixed" -i:NCSA -q:ON

4,965,915

logparser "select sum(BytesSent) from C:\Temp\access.log-20131103.fixed" -i:NCSA -q:ON

14,540,275,692

logparser "select datetime from C:\Temp\access.log-20131103.fixed order by datetime" -i:NCSA

[27/Oct/2013:03:35:02 +0000]

logparser "select datetime from C:\Temp\access.log-20131103.fixed order by datetime desc" -i:NCSA

[03/Nov/2013:03:46:54 +0000]
logparser "select RemoteHostName, sum(BytesSent) as sumbytes from C:\Temp\access.log-20131103.fixed group by RemoteHostName order by sumbytes desc" -i:NCSA -q:ON | more

65.95.113.142   15,415,998
216.191.251.39  15,030,023
108.162.170.182 13,551,628
196.219.44.178  12,161,472
91.236.75.108    7,256,066
198.56.164.219   7,220,740
207.62.236.105   6,620,832
162.216.169.28   6,554,265
99.243.133.208   6,490,613
192.241.134.119  6,454,660
67.193.144.163   6,430,187
128.233.17.40    6,400,425
46.149.25.65     6,285,600
67.71.148.75     6,132,012
210.129.158.54   6,034,176
212.156.80.18    6,034,176
208.94.38.89     6,034,176
42.60.180.45     5,903,808
69.64.61.61      5,866,560
85.25.199.87     5,680,320
81.192.156.83    5,633,760
198.204.237.21   5,447,520
207.195.86.234   5,425,002
85.214.86.65     5,363,712
logparser "select Request, sum(BytesSent) as sumbytes from C:\Temp\access.log-20131103.fixed group by Request order by sumbytes desc" -i:NCSA -q:ON | more

GET /michael/sites.php HTTP/1.1          14,062,100,124
GET /michael/sites.php HTTP/1.0              54,478,996
GET /michael/img/background.jpg HTTP/1.1     46,132,218
GET /michael/chalkboards/ HTTP/1.0           22,330,975
GET /michael/chalkboards/index.php HTTP/1.0  20,655,595
GET /img/bak5.jpg HTTP/1.1                   18,794,146
GET /img/bak4.jpg HTTP/1.1                   17,048,206
GET /michael/blog/cmd_slideshow.php HTTP/1.1 15,855,209
GET / HTTP/1.1                               13,565,639
GET /img/bak1.jpg HTTP/1.1                   11,876,341
GET /img/bak2.jpg HTTP/1.1                   11,844,601
GET /michael/base/util.js HTTP/1.1           10,846,396
GET /img/bak3.jpg HTTP/1.1                    9,525,211
GET /michael/chalkboards/ HTTP/1.1            8,605,885
GET / HTTP/1.0                                8,221,673
GET /michael/base/stylesglobal.css HTTP/1.1   5,731,520
GET /public/profile/newuser.php HTTP/1.0      5,720,568
GET /michael/lifecounters/ HTTP/1.1           5,680,240
logparser "select Referer, sum(BytesSent) as sumbytes from C:\Temp\access.log-20131103.fixed group by Referer order by sumbytes desc" -i:NCSA -q:ON | more

http://www.egycivilization.com/                                                     5,996,255,678
http://www.egycivilization.com/search/label/search%20engine                           346,444,948
http://www.egycivilization.com/2013/10/getting-started-with-google-adwords-ppc.html   266,285,829
http://www.egycivilization.com/2013/10/google-adwards-keyword-means.html              262,293,813
http://www.egycivilization.com/search/label/google%20Adword                           254,383,456
http://www.egycivilization.com/search/label/Internet%20Marketing                      198,474,888
http://www.egycivilization.com/search/label/blog                                      148,627,392
http://www.egycivilization.com/search/label/sales%20Training                          147,945,050
http://www.egycivilization.com/search/label/domain                                    147,445,168
http://www.egycivilization.com/search/label/Email%20marketing                         146,889,205
http://www.egycivilization.com/search/label/Profit%20from%20the%20Internet            146,558,042
http://www.egycivilization.com/search/label/advertising%20of%20facebooke              146,325,462
http://www.egycivilization.com/search/label/Ancient%20Egyptian%20civilization         145,968,300
http://www.egycivilization.com/search/label/making%20money                            145,648,926
http://www.egycivilization.com/search/label/advertising                               144,555,182
http://www.egycivilization.com/search/label/SEO                                       144,175,613
http://www.egycivilization.com/search/label/Ebay                                      143,468,061
http://www.egycivilization.com/search/label/Affiliate%20Marketing                     142,542,707
http://www.egycivilization.com/search/label/Adsense                                   142,033,513
http://www.egycivilization.com/search/label/mesothelioma                              141,530,928
http://www.egycivilization.com/search/label/Forex                                     141,471,447
http://www.egycivilization.com/search/label/Site%20Promotion                          141,448,400
http://www.egycivilization.com/search/label/SOCIAL                                    139,371,604
logparser "select sum(BytesSent) from C:\Temp\access.log-20131103.fixed where Referer like '%egycivilization%'" -i:NCSA -q:ON

14,113,655,622

Later Logs

logparser "select sum(BytesSent) from C:\Temp\access.log-20131110.fixed where Referer like '%egycivilization%'" -i:NCSA -q:ON

5,110,249,685

logparser "select sum(BytesSent) from C:\Temp\access.log-20131117.fixed where Referer like '%egycivilization%'" -i:NCSA -q:ON

74,176

logparser "select sum(BytesSent) from C:\Temp\access.log.fixed where Referer like '%egycivilization%'" -i:NCSA -q:ON

0

And gros bytes

logparser "select sum(BytesSent) from C:\Temp\access.log-20131110.fixed" -i:NCSA -q:ON

5,576,651,600

logparser "select sum(BytesSent) from C:\Temp\access.log-20131117.fixed" -i:NCSA -q:ON

397,941,289

logparser "select sum(BytesSent) from C:\Temp\access.log.fixed" -i:NCSA -q:ON

47,843,999

What does it all mean?

In the week of Oct 20th

 5 million logs
 14 GB transferred
 Biggest client IP took 25 MB
 By far, the page most frequently served was the homepage /michael/sites.php
 It was served 5 million times
 By far, the biggest referrer was www.egycivilization.com
 They caused 14 GB

In the week of Oct 27th

 5 million logs
 14 GB transferred
 Biggest client IP took 15 MB
 Three big IPs in common with the previous week
 By far, the page most frequently served was the homepage /michael/sites.php
 It was served 5 million times
 By far, the biggest referrer was www.egycivilization.com
 They caused 14 GB

So for some insane reason the people of egycivilization are causing someone to hit my home page 5 million times a week and eating 14 GB of my bandwidth. There's nothing to gain by that. I wonder what their motive is? Searching for holtstrom site:www.egycivilization.com on google gives no hits. Looking at later logs it seems they've stopped hitting me. Was it a bug on their side? Was it a DOS attack?

Update

logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131027.fixed group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName  numhits
--------------- -------
196.219.44.178  4480
87.109.98.204   4467
64.187.228.74   4104
108.162.170.182 3366
192.187.121.107 3342
85.25.199.87    3240
180.87.42.21    3071
172.246.115.228 2706
200.6.100.160   2692
210.129.199.34  2499
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131103.fixed group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName  numhits
--------------- -------
196.219.44.178  4048
46.149.25.65    2700
212.156.80.18   2592
42.60.180.45    2546
69.64.61.61     2520
81.192.156.83   2452
85.25.199.87    2446
198.204.237.21  2340
108.162.170.182 2325
69.198.191.230  2304
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131110.fixed group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName  numhits
--------------- -------
108.162.170.182 7915
87.109.99.4     1963
192.0.134.34    1723
216.191.251.39  1556
46.149.25.65    1541
85.25.199.87    1425
212.175.133.30  1296
192.187.121.107 1270
196.219.44.178  1192
198.50.212.150  1188
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131117.fixed group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName numhits
-------------- -------
192.0.134.34   2145
216.191.251.39 876
173.34.31.254  424
100.43.83.161  337
65.160.18.178  256
162.243.87.165 252
198.56.164.221 221
74.91.24.66    193
124.232.163.29 169
66.249.66.89   163

Rate-limiting by IP at 4k could have blocked

196.219.44.178  4480
87.109.98.204   4467
64.187.228.74   4104
196.219.44.178  4048
108.162.170.182 7915
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131027.fixed where Referer like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName numhits
-------------- -------
196.219.44.178 4048
46.149.25.65   2700
212.156.80.18  2592
42.60.180.45   2546
69.64.61.61    2520
81.192.156.83  2452
85.25.199.87   2446
198.204.237.21 2340
69.198.191.230 2304
63.141.248.220 2268
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131103.fixed where Referer like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName  numhits
--------------- -------
87.109.99.4     1963
46.149.25.65    1541
85.25.199.87    1425
212.175.133.30  1296
192.187.121.107 1270
196.219.44.178  1192
198.50.212.150  1188
83.149.127.59   1152
63.141.236.142  1116
192.187.106.76  1080
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131110.fixed where Referer like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName numhits
-------------- -------
197.1.215.187  25
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131117.fixed where Referer like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

-none-
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131027.fixed where Referer not like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName  numhits
--------------- -------
108.162.170.182 3366
216.191.251.39  871
100.43.83.161   648
216.191.251.36  479
66.249.75.23    389
192.114.71.13   368
95.108.158.245  351
66.249.75.89    328
91.121.115.186  325
91.236.75.108   224
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131103.fixed where Referer not like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName  numhits
--------------- -------
108.162.170.182 2325
216.191.251.39  1003
183.60.244.29   539
110.86.167.104  402
100.43.83.161   345
198.56.164.219  295
216.191.251.36  283
198.199.122.131 252
174.3.235.170   238
95.108.158.245  225
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131110.fixed where Referer not like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName  numhits
--------------- -------
108.162.170.182 7915
192.0.134.34    1723
216.191.251.39  1556
174.3.235.170   1184
193.203.49.60   568
216.191.251.36  528
183.60.244.44   508
198.56.164.219  431
198.199.122.131 252
100.43.83.161   219
logparser "select RemoteHostName, count(RemoteHostName) as numhits from C:\Temp\access.log-20131117.fixed where Referer not like '%egycivilization%' group by RemoteHostName order by numhits desc" -i:NCSA

RemoteHostName numhits
-------------- -------
192.0.134.34   2145
216.191.251.39 876
173.34.31.254  424
100.43.83.161  337
65.160.18.178  256
162.243.87.165 252
198.56.164.221 221
74.91.24.66    193
124.232.163.29 169
66.249.66.89   163

okay, "where referer like" results range between

196.219.44.178 4048
197.1.215.187    25

"where referer not like" results range between

108.162.170.182 7915
66.249.66.89     163

There are no ips from "where referrer like" in "where referrer not like", so it seems that simple rate limiting won't help me.

Update

logparser "select count(RemoteHostName) as numhits from C:\Temp\access.log-20131027.fixed" -i:NCSA

5314097

logparser "select count(distinct RemoteHostName) as distincthits from C:\Temp\access.log-20131027.fixed" -i:NCSA

22442

logparser "select count(RemoteHostName) as iphits from C:\Temp\access.log-20131027.fixed group by RemoteHostName order by iphits desc" -i:NCSA

iphits
------
4480
4467
4104
logparser "select count(RemoteHostName) as numhits from C:\Temp\access.log-20131103.fixed" -i:NCSA

4965915

logparser "select count(distinct RemoteHostName) as distincthits from C:\Temp\access.log-20131103.fixed" -i:NCSA

19700

logparser "select count(RemoteHostName) as iphits from C:\Temp\access.log-20131103.fixed group by RemoteHostName order by iphits desc" -i:NCSA

iphits
------
4048
2700
2592
logparser "select count(RemoteHostName) as numhits from C:\Temp\access.log-20131110.fixed" -i:NCSA

1915354

logparser "select count(distinct RemoteHostName) as distincthits from C:\Temp\access.log-20131110.fixed" -i:NCSA

12535

logparser "select count(RemoteHostName) as iphits from C:\Temp\access.log-20131110.fixed group by RemoteHostName order by iphits desc" -i:NCSA

iphits
------
7915
1963
1723
logparser "select count(RemoteHostName) as numhits from C:\Temp\access.log-20131117.fixed" -i:NCSA

21145

logparser "select count(distinct RemoteHostName) as distincthits from C:\Temp\access.log-20131117.fixed" -i:NCSA

1812

logparser "select count(RemoteHostName) as iphits from C:\Temp\access.log-20131117.fixed group by RemoteHostName order by iphits desc" -i:NCSA

iphits
------
2145
876
424

What does it mean?

Total Hits Per Week: 5314097
                     4965915
                     1915354
                       21145 -- (the attack is over)

Distinct IPs Per Week: 22442
                       19700
                       12535
                        1812 -- (the attack is over)

Hits Per IP Per Week: 4480, 4467, 4104
                      4048, 2700, 2592
                      7915, 1963, 1723 -- (the 7k is probably me)
                      2145,  876,  424 -- (the attack is over)

So the attacker doesn't over-use a particular IP. And from earlier analysis, I think the 7915 hits in one week was due to me using CTRL+F5 while developing the site. Thus simple rate-limiting by doesn't look like it will help.

aws
{ "url": "https:\/\/holtstrom.com\/michael\/", "base": "\/michael", "domain": "holtstrom.com", "loggedin": false, "avatar": "", "render": "nothing", "doTracking": true, "trackingID": "UA-36983794-1", "description": "", "tmdbUrl": "http:\/\/www.themoviedb.org\/", "tmdbPoster": "http:\/\/image.tmdb.org\/t\/p\/w342" }