LinkedIn Sourceforge Twitter

Vincent's Blog

Pleasure in the job puts perfection in the work (Aristote)

Block bad visitors with PF

Posted on 2016-11-06 17:04:00 from Vincent in Open Bsd

Those days, when you fire-up a new machine, it takes few hours that bad persons are trying to connect to your machine via ssh, try default URL with default passwords, ...

This post will explain how OpenBSD can easily helps you to ban those persons out of your machine for quite a long time.


update: please look at log2table here

The main goal is to attentively look at your log files and specifically authlog. This idea has been largely developped since several years on Linux with fail2ban.

In /var/log/authlog OpenBSD informs you about any remote connections and their status: success, failed, disconnected, ...

So, basically the idea is to read this file regularly and take the IP of the "bad persons", add it in a specific PF table which will reject for a long time.

Let me explain how I did it in detail

1. Approach

1.1 Recognize the layout of the messages

Here after some of the main message I can see in this authlog file:

Oct 24 23:12:44 myopenbsdserver sshd[50476]: Failed password for root from xx.xx.xx.108 port 48822 ssh2
Oct 24 23:12:46 myopenbsdserver sshd[50476]: Received disconnect from xx.xx.xx.108 port 48822:11:  [preauth]
Oct 24 23:13:02 myopenbsdserver sshd[32493]: Invalid user adam from xx.xx.xx.218 port 24329
Oct 24 23:13:02 myopenbsdserver sshd[32493]: Failed password for invalid user adam from xx.xx.xx.218 port 24329 ssh2
...
Oct 25 11:49:34 myopenbsdserver sshd[94860]: Failed password for invalid user ubnt from xx.xx.xx.60 port 36656 ssh2
Oct 25 11:49:34 myopenbsdserver sshd[94860]: Connection closed by xx.xx.xx.60 port 36656 [preauth]
Oct 25 12:41:54 myopenbsdserver sshd[72644]: Connection closed by xx.xx.xx.216 port 40666 [preauth]
Oct 25 12:43:29 myopenbsdserver sshd[26963]: Connection closed by xx.xx.xx.216 port 53315 [preauth]
Oct 25 12:49:55 myopenbsdserver sshd[140]: Connection closed by xx.xx.xx.216 port 47433 [preauth]
Oct 25 13:25:05 myopenbsdserver sshd[45542]: Connection closed by xx.xx.xx.45 port 55353 [preauth]

(I have masked the IPs to avoid problems with their owners)

Basically the idea will be to recognize some of the most annoying messages and see where we can find the IP of the requestor.

In my case I'm interested to block persons trying false passwords and persons opening and closing connections. Thus my searching rules will be:

RE_FAILED = re.compile(".* Failed .* from ([\S]+)")
RE_CLOSED = re.compile(".* Connection closed by ([\S]+) .*")

As you may have deduce, I'm using Python's regular expression to help for such task. You can use several other programming languages, most of them have pattern recognition.

What this code does is:

  • to record the word just after "from", when we have Failed in the log file.
  • to record the word just after "closed by" when we have Connection closed by in the log file.

Thanks to PF, adding a bad IP is quite easy and could be done with a command like this:

pfctl -t bruteforce -T add 111.222.333.444

1.2 Block bad IPs listed in the associated PF table

As we are running this on OpenBSD, we can rely on the wonderful packet filter (called PF) provided with the base system.

So, within your PF rules you should have in /etc/pf.conf, you should add the following:

table <bruteforce> persist
block in quick proto tcp from <bruteforce> to any

Such command is quite easy to understand. It says that every IP listed in the table called bruteforce are immediately blocked.
We block them on any ports. So, they will no more be able to access the server.
In my case we block the TCP protocol. But you could use udp and icmp too.

1.3 Add some white lists

Imagine that, during one very hard day, you are so tired you mistype your password during your ssh connection. In such case, OpenBSD will says in the authlog file that your IP has provided a false password. The RE_FAILED rule will be triggered and you will be in big troubles.

So, let's define a white list of IPs that your script will never adds in the bruteforce table of PF.

2 Let's glue all those concept into one small program.

As said, I'm mainly using Python for my scripts. Feel free to use others languages if your prefer. Here after I'll just share how I've glued all those concepts together.

Before putting all those elements together, I would introduce one Python nice tool called pygtail which will allow you to read the file just where you were after the previous analysis. This avoid le to treat two times the same data of the log file. As for each Python's extension, you can install it with pip install pygtail command.

Our script will become something like this:

#if not present, install this extension with the command "pip install pygtail"
from pygtail import Pygtail
import re,os,os.path
import sys
import datetime

#the file we will check 
LOGFILE = "/var/log/authlog"
#the file containing our white list of IPs. One IP per line
CONFIG = "/etc/accepted_ips"

#we are using dictionary for IPs to simplify and speedup the search 
accept_ips={}
bad_ips = {}
cmd=""

#we populate IPs contained in the CONFIG file into the accept_ips dictionary
def read_ips():
    if os.path.isfile(CONFIG):
        fid = open(CONFIG,"r")
        for line in fid.readlines():
            if line.strip():
                accept_ips[line.strip()] = ""
        fid.close()

#We let the possibility to use an another input file for testing purposes   
if len(sys.argv) == 3 and sys.argv[1] == "-f":
   LOGFILE = sys.argv[2]

#Our 2 main rules
#
#May 31 01:15:15 myvultr sshd[18229]: Failed password for invalid user guest from xx.xx.xx.4 port 57493 ssh2
RE_FAILED = re.compile(".* Failed .* from ([\S]+)")

#May 31 09:40:23 myvultr sshd[22894]: Connection closed by xx.xx.xx.73 port 32863 [preauth]
RE_CLOSED = re.compile(".* Connection closed by ([\S]+) .*")

read_ips()
print datetime.datetime.now()

#The follwing part read the log file and store the bad IPs in the bad_ips dictionary
#This code, assign a counter to each rule defined. 
#This allows me to not ban a user having done very few bad manipulations. 
for line in Pygtail(LOGFILE):
    ret = RE_FAILED.match(line)
    if ret:
        bad_ip = ret.groups()[0]
        bad_ips.setdefault(bad_ip,0)
        bad_ips[bad_ip] += 9           #9 points for such "bad " event
    ret = RE_CLOSED.match(line)
    if ret:
        bad_ip = ret.groups()[0]
        bad_ips.setdefault(bad_ip,0)
        bad_ips[bad_ip] += 1           #1 point for such event

#We loop accross the bad_ips
for bad_ip,occurence in bad_ips.items():
    #Only if the bad ip identified is not present in our "white list" we will add it in the PF table
    if bad_ip not in accept_ips:
        print "Occurence: %s, IP:%s" % (occurence,bad_ip)
            #if the counter is above 5, we ban the IP. 
        if occurence>5:
            cmd = "pfctl -t bruteforce -T add %s " % bad_ip
            print "CMD:%s" % (cmd)
            if len(sys.argv)==3 and sys.argv[1] == "-f":
                    #we are in testing mode and we don't want to bad our test IPs
                pass
            else:
                os.popen(cmd)

print "="*10

As you see, I'm providing points to bad IPs depending on their actions. If the total, for a specific IP, is bigger than 5, I ban it by adding this IP in the bruteforce table of PF.

The remaining task will be to add this scrip in the crontab. Since we are using pfctl command, the crontab of root will be required.
You could use doas and assure that your user is able to run this command in /etc/doas.conf. If you prefer this, I let you check the man pages of doas.conf.

*/5     *       *       *       *       /usr/local/bin/python2.7 -u /usr/local/bin/parse_authlog.py >> /var/log/parse_authlog.log 2>&1

In this case, I'm doing the check every 5 minutes. Up to you to adapt this value. This is in fact a trade-off between the delay you let hackers try different techniques and the number of records you have in /var/log/authlog. If this delay is too short, you could not identify slow, but real attacks.

Please note that I'm using python -u to unbuffer the outputs. This allow me to follow, on real time, the data added into parse_authlog.log in real time. I know this not the ideal way of doing it, but I'm lazy :(

2.2 Performing some cleanup

As you could imagine our bruteforce table will increase days after days. This could become a problem if the table becomes too big.

So, here again, PF has a very decent solution:

pfctl -t bruteforce -T expire 86400

With this command, you can cleanup IPs added into bruteforce more than 1 day ago.

So, you can add this line in your crontab:

1       *       *       *       *       /sbin/pfctl -t bruteforce -T expire 86400 >> /var/log/parse_authlog.log 2>&1

Here I'm doing the cleanup every hours. Up to you to adapt this to your needs.

You can always check the content of the bruteforce table with the following command:

pfctl -t bruteforce -T show

For your info, my parse_autolog.log file look like this:

...
2016-10-24 01:55:01.929585
==========
2016-10-24 02:00:02.162995
Occurence: 10, IP:xx.xx.xx.131
CMD:pfctl -t bruteforce -T add xx.xx.xx.131
1/1 addresses added.
==========
1/1 addresses expired.
2016-10-24 02:05:01.285829
==========
2016-10-24 02:10:01.439906
==========
2016-10-24 02:15:01.580643
...

We see that 1 IP has been banned because is score is above 5 (score is 10).
We see too that 1 IP has been remove from the bruteforce table.
During the other scan, nothing has been performed.

3 Lessons learned

This small script runs quite well since few months.
I have nearly always 10 IPs banned. But they are not always the same.

This is a good alternative to the fail2ban, and it does a good job.

There are several possible improvement, like:

  • ban an IP forever (never remove it from bruteforce) if this IP try 2 attacks in the same week.
  • ban that range in which the banned IP reside
  • check the country of the IP before ban it
  • ...


22, 17
displayed: 2944
Comments:

1. From thuban on Mon Nov 28 11:15:58 2016

This seems amazing and yet simple ! I think I'll hack it a little when I'll have some time. keep on the good work ;)

2. From thuban on Mon Dec 19 16:55:58 2016

Hi, I used your work to write something usable for any file. See here : https://obsd4a.net/qa/viewtopic.php?pid=191#p191 Regards.

3. From TumancMow on Mon Dec 3 17:33:16 2018

Today any companies provide an opportunity to use payday loans online in USA for different needs. It may be different global acquiring or apartment renovation. It may be desiring to take payday loans in USA for car or garden. You can obtain a loan at maybeloan.com very fast. Loan clearance procedure is very easy. It is important to fill in the application and you can obtain loan for interesting sum. All loans created for folk with different life tasks. You can take cash very fast and be lucky. It is important to communicate with administrators even you will have any problems. However, company give an opportunity to receive cash without income certificate. You may receive coins even you have bad credit payday loans in the NY. If you wish to get [url=https://maybeloan.com/payday-loans/al]quickest payday loans in AL[/url] in United States of America, you should go to site. It is very light to specify personal data. Last year in our county it is very hard economic situation. That is why more and more young families take credits for their aims. You should search best variant for you. It is very easy to do. You can use cash in different aims. No one will check purpose of payment. You could spend money for private goals. Last time more and more folk ask administrators about application form. If you desire to take loans interactive, you should ask administrators how possible to do it so fast. They will grant for you best service. Now money credits in USA are provide for people who are over 18 years old. You must to understand that if you have fine cash history, you may take short term payday credits in USA very fast. With cash you can realize your needs. Currently at [url=https://maybeloan.com/payday-loans/me]https://maybeloan.com/payday-loans/me[/url] different users receive fast payday loans in USA. If you wish to receive payday mortgage in AZ, it is very light to do. Even you haven’t opportunities to get credit online by yourself, you must to ask administrators how probable it to do. It is very easy to connect with specialists by e-mail info@maybeloan.com. As well, you can use customer services. It is possible to communicate with specialists and ask them how feasible to get loans in USA even the men have bad credit history. However you should know that all folk are under protection. When anyone take credits, it is very easy to get positive online decision. Now company situated at 317 S Broadway, Los Angeles, CA 90013 where you can arrive. Also, people who live in NY like to utilize credits online. They use different online wallets and like to get credits for few months. Last days men are actively speaking to loans companies. As rule, small loans don’t arrange folk. That is why you may also ask managers how possible to get money. It is deleted different problems about waste of time or long queues. Maybeloan.com get for people various special offers. It is possible to obtain cash for various aims for young guys or men after 50 years old. All procedures about fast payday credits in USA are reliable. If you want to instant payday credits in FL, you must go to website. If you have any problems with money in your town, you may go at website and receive payday loans in Hawaii. Any women haven’t chances to have same day payday credits in Iowa in banking houses. If you faced with this position, you should utilize virtual credits. You can receive cash for different time. In different ways you can delay your payment. Any people in county haven’t possibility to pay for utilities in time. We recommend use online credits and be sure in them. It is very easy to have credits online. Then you should communicate with managers and they will ask for various your questions.

4. From Vsezaiksnum on Sat Dec 15 17:03:17 2018

Если вы решили получить быстрый займ, вам надо определиться с компанией, где оптимально взять займ online. В наши дни не все организации предоставляют моментальные займы. Если вы стремитесь получить быстрые займы, вам необходимо просмотреть каталог компаний, которые предоставляют займы на выгодных условиях. На все-займы-тут.рф находится каталог организацией, которые выдают мгновенные займы людям. На ресурсе вы можете найти разные МФО, прочитать отклики и определиться с параметрами займа. Большое количество компаний предоставляют деньги на оптимальных условиях. Получить займ вы можете в наши дни с паспортом. Для того, чтобы посетитель мог получить микрозайм на карту надо выбрать компанию, изучить параметры займа. Каждый клиент попадал в положение, когда финансы нужны очень срочно. Близкие люди не всегда могут помочь финансово. Из-за этого необходимо изучить шанс взять online займ на карту. Если вы стремитесь получить для себя [url=http://все-займы-тут.рф/]займы на карту без отказа под 0[/url] , вам надо выбрать условия. Если вы добросовестный заемщик, деньги будут переведены вам в сжатые сроки. Получить займ можно в рублях на конкретный срок. Необходимо выделить, что сервис гарантирует максимальную конфиденциальность. На ресурсе собраны МФО, которые всегда идут клиентам навстречу. Новые клиенты также не будут ощущать какого-то дискомфорта. Лояльные условия кредитования также доступны для многих клиентов. У вас существует вероятность получить микрозаймы быстрее, чем это проходит в банках. Вам не придется собирать справки. Быстрые займы – уникальный выход, если вам в сжатые сроки нужны средства. Для получения денег следует заполнить анкету на ресурсе МФО. С вами моментально свяжется администратор и обсудит все детали. Если вы не знаете, где можно изучить список лучших МФО, можете смело просмотреть его на ресурсе. Лучшие микрофинансовые организации размещены на портале. У всех МФО есть техническая поддержка. Получить кредит можно на банковскую карту любого банка. Вы можете также получить деньги за 15 минут без каких-то вопросов от банковских менеджеров. После того, как ваша заявка будет подтверждена, вы получите подтверждение. Более 90% заявок получают одобрение. Оформить займ вы можете даже через планшет. Для этого нужно иметь выход в интернет. Также вам нужен будет паспорт и банковская карта, куда вы будете получать деньги.

5. From KatzimsBoabe on Tue Dec 18 21:48:49 2018

Если вы приняли решение получить мгновенный займ, вам нужно определиться с МФО, где оптимально взять займ онлайн. Сегодня не все МФО оформляют моментальные займы. Если вы стремитесь получить быстрые займы, вам надо просмотреть перечень компаний, которые предоставляют займы на выгодных условиях. На все-займы-тут.рф есть список организацией, которые выдают быстрые займы людям. На ресурсе реально найти разные организации, прочитать отзывы и определиться с параметрами займа. Большое количество компаний предоставляют средства на выгодных условиях. Получить займ реально сегодня с паспортом. Для того, чтобы посетитель мог получить микрозайм на карту нужно выбрать компанию, изучить параметры займа. Каждый человек попадал в ситуацию, когда средства могут потребоваться очень срочно. Близкие люди не всегда могут помочь в финансовом плане. По этой причине необходимо изучить возможность взять онлайн займ на карту. Если вы стремитесь оформить [url=http://все-займы-тут.рф/]займы без процентов на карту срочно без отказа[/url] , вам нужно выбрать условия. Если вы добросовестный заемщик, средства будут переведены вам в быстрые сроки. Получить деньги вы можете в рублях на конкретный срок. Нужно выделить, что сервис гарантирует полную конфиденциальность. На ресурсе собраны МФО, которые всегда идут посетителям навстречу. Новые лица также не будут ощущать какого-то дискомфорта. Отличные условия кредитования также существуют для многих людей. У вас есть возможность получить микрозаймы быстрее, чем это проходит в банковских учреждениях. Вам не нужно собирать документы. Быстрые займы – универсальный выход, если вам срочно потребовались деньги. Для получения средств следует заполнить анкету на ресурсе МФО. С вами в сжатые сроки свяжется менеджер и обсудит все подробности. Если вы не уверены, где вы можете найти каталог лучших МФО, можете смело изучить его на сайте. Лучшие микрофинансовые организации есть на ресурсе. У всех сервисов есть техническая поддержка. Получить деньги можно на банковскую карту любого банка. Вы можете также получить деньги за 5 минут без лишних вопросов от банковских сотрудников. После того, как ваша заявка будет одобрена, вы получите подтверждение. Больше чем 90% заявок получают одобрение. Оформить мирозайм вы можете даже через планшет. Для этого понадобится иметь выход в сеть. Также вам нужен будет паспорт и банковская карта, куда вы будете получать деньги.




How much does 2 plus 2 ?