I have a very simple situation but I can't figure out how regex works...
I have an application generating a log only when a login problem occurs. So there is no line in log except in case of wrong login or in case of attempt to reset a password too many times. So potentially, I don't even need to search a particular string in log, any entry matches. Here is a log example :
2019-10-20 18:44:35 127.0.0.1 login.php : Authentication error - account not initialized : client XXXX, login YYYY
2019-10-20 21:31:17 127.0.0.1 login.php : Authentication error - password error : client XXXX, login XXXX
2019-10-20 21:29:39 127.0.0.1 login.php : Authentication error - client contains wrong chars : client XXXX, login YYYY
2019-10-21 06:25:25 127.0.0.1 login.php : Authentication error - account locked : client XXXX, login YYYY
2019-10-21 06:48:11 127.0.0.1 user.php : Authentication - Unlocking : client XXXX, login YYYYY
I have a problem with regular expression cause I can't understand how it works (for years). All I tried give me errors when I start fail2ban : Unable to compile regular expression, No failure-id group in 'Authentication error', ... Damned, it looks so easy !
Finally...
[INCLUDES]
before = common.conf
[Definition]
failregex = <HOST> .* Authentication
Please note this works but I found without understand anything. If someone has a link where how fail2ban works is explained. I found many but none of them have clear explanations.
The main trick is the fact that error.log files look different from tutorials and you need to rewrite failregex manually. Another trick is that <HOST> is a predefined regexp that tries to match an ip but it could match something like datetime instead
In such case like
[Tue Nov 08 03:03:03.349852] bla-bla [client 1.2.3.4:5] bla-bla xxx bla-bla
In case you're banning when xxx appear
[Definition]
failregex = client <HOST>(.*)xxx
Sadly quick google search upon fail2ban and tutorials indeed explain little. I also wanted to ban by simple string match.
Related
I'm having a strange problem. I followed few guides from the net. My goal is to create a SMTP postfix that will use Cyrus SASL to authenticate users upon sending email with different imap server.
Making all more simple: have to transfer/replace current smtp server with new one as current is on public cloud and gets on black lists pretty often.
What I managed so far is:
Working Postfix
Authentication working when using :
testsaslauthd -u user#domain.com -p password
I'm getting Ok "Success" so I assume sasl itself work.
When I invoke saslfinger -s
I'm getting:
There is no smtpd.conf that defines what SASL should do for Postfix.
SMTP AUTH can't work!
but it seems that all is fine within the configuration files:
/etc/postfix/sasls/smtp.conf:
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN
/etc/postfix/main.cf:
smtpd_recipient_restrictions = reject_invalid_hostname,
permit permit_mynetworks,
permit_sasl_authenticated
disable_vrfy_command = yes
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
/etc/default/saslauthd-postfix:
START=yes
MECHANISMS="rimap"
MECH_OPTIONS="domain.com -r"
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"
I'm running postfix chroot'ed so had to create a symlink but like I said. It all seems to work independently, just need to be linked somehow.
When I try to setup account in outlook, I got wrong name or password.
The log on Debian says:
May 11 23:35:43 smtp-test postfix/smtpd[741]: warning: unknown[192.168.108.1]: SASL NTLM authentication failed: authentication failure
May 11 23:35:43 smtp-test postfix/smtpd[741]: warning: SASL authentication failure: unable to canonify user and get auxprops
May 11 23:35:43 smtp-test postfix/smtpd[741]: warning: unknown[192.168.108.1]: SASL DIGEST-MD5 authentication failed: authentication failure
May 11 23:35:43 smtp-test postfix/smtpd[741]: warning: unknown[192.168.108.1]: SASL LOGIN authentication failed: authentication failure
May 11 23:35:43 smtp-test postfix/smtpd[741]: lost connection after AUTH from unknown[192.168.108.1]
May 11 23:35:43 smtp-test postfix/smtpd[741]: disconnect from unknown[192.168.108.1]
Strange thing is it tries NTLM(not mentioned anywhere) instead of RIMAP. And cannot make canonical name of user even after adding -r switch that should combine name and realm/domain name.
I guess that is related to first warning from saslfinger but cannot find the cause.
All updated to newest available versions.
Any help?
While trying to setup postfix to allow a website to sent password reset emails I've noticed that when sending emails form the command line using
echo "body" | mail -s "subject" test#example.com
the emails arrive just fine and are not marked as spam. However, when sending from withing the sites server, the header is set as being received from localhost and thus is marked as spam.
Received: from example.com (localhost [127.0.0.1])
by example.com (Postfix) with ESMTPS id CE1299C435
for <my_email#gmail.com>; Tue, 2 Aug 2016 05:28:41 +0000 (UTC)
As a note, dkim, spf and dmarc test all list as passed, so they are not the problem.
I have also tried sending an email with the same content using the above echo method and had it arrive fine (although plaintext as I used it as above and didn't add the content type headers).
Initially this was sent using smtp.SendMail
err := smtp.SendMail(hostname + ":25", nil, sender, recipients, []byte(msgstr))
which provides no way to set sending host, so in an attempt to correct this I switched to using a manual dial, sending a HELO message, then starting tls (attempting to start tls before the HELO resulted in a panic saying the HELO had to be the first call)
c, err := smtp.Dial(host + ":25")
if err != nil {
log.Panic(err)
}
if err = c.Hello(host); err != nil {
log.Panic(err)
}
if err = c.StartTLS(config.TLS()); err != nil {
log.Panic(err)
}
This still sends the email successfully, but as before the received header still shows localhost as the origin, I feel I've misunderstood something as I believed that HELO served to identify where the call was coming from.
Your advice and help is appreciated, this is the first time I've tried to configure an MTA and use it like this, there has been a bit of a learning curve that I'm evidently not over yet.
As Juergen Gmeiner suggested in the comments, my problem ended up being caused by a poorly configured /etc/hosts file, such that it wasn't effecting other services.
The existing entry was:
127.0.0.1 localhost example.com
Changing this to:
127.0.0.1 example.com localhost
solved the problem while not breaking the existing services that needed localhost to evaluate properly.
This solution worked for my locally hosted server process, if the same problem occurs with a remote server, and equivalent like would need to be created with the ip and hostname of the remote. i.e.
<remote ip> <hostname>
note: while I tested that this last part by calling from my local development build to our production server and it worked, its not a 100% tested solution.
Thank you all for your help, and I hope that this mistake on my part helps anyone else who makes a similar error in the future.
I was following this guide to setup an LDAP server on CentOS 6.6: http://www.learnitguide.net/2016/01/configure-openldap-server-on-rhel7.html. I know the guide is for RHEL, but I think the vast majority of steps should be the same. I went through the tutorial and everything seems to work right except for sshing into the server as an LDAP user. As root I am able to su to any LDAP user. getent passwd username returns appropriate results. But every time I try to ssh in as a user I get permission denied and these messages show up in /var/log/secure:
Aug 8 22:13:14 servername sshd[5900]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rhostname user=username
Aug 8 22:13:14 servername sshd[5900]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=rhostname user=username
Aug 8 22:13:14 servername sshd[5900]: pam_sss(sshd:auth): received for user username: 6 (Permission denied)
Aug 8 22:13:17 servername sshd[5900]: Failed password for username from rhostIPaddress port 34758 ssh2
I was hoping maybe someone may know of a step that guide left out that would cause this behavior. Thanks.
Turns out the self signed certificate I was using needed to have the IP address as the common name as opposed to the FQDN, not sure why that is but it's working now.
Information that can help others.
In my case this message was a time difference problem between the FreeIPA server and the clients. The command systemctl status sssd -l presented the message (skew clock too great).
This message from the Kerberos authentication server appears if the difference hour in servers is too large (more than three or four minutes).
OS: Ubuntu Server
Need some help related to create a custom filter for custom app which is websocket server written in node.js . As per my understanding from other articles the custom node.js app needs to write a log which enters any authentication failed attempts which will further be read by Fail2ban to block IP in question . Now I need help with example for log which my app should create which can be read or scanned by fail2ban and also need example to add custom filter for fail2ban to read that log to block ip for brute force .
Its really old question but I found it in google so I will write answer.
The most important thing is that line you logging needs to have right timestamp because fail2ban uses it to ban and unban. If time in log file is different than system time, fail2ban will not find it so set right timezone and time in host system. In given example I used UTC time and time zone offset and everything is working. Fail2Ban recognizes different types of timestamps but I didn't found description. But in fail2ban manual you can find two examples. There also exist command to check if your line is recognized by written regular expression. I really recommend to use it. I recommend also to use "regular expression tester". For example this one.
Rest of the log line is not really important. You just need to pass user ip.
This are most important informations but I will write also example. Im just learning so I did it for educational purposes and Im not sure if given example will have sense but it works. I used nginx, fail2ban, pm2, and node.js with express working on Debian 10 to ban empty/bad post requests based on google recaptcha. So set right time in Your system:
For debian 10 worked:
timedatectl list-timezones
sudo timedatectl set-timezone your_time_zone
timedatectl <-to check time
First of all You need to pass real user ip in nginx. This helped me so You need to add line in You server block.
sudo nano /etc/nginx/sites-available/example.com.
Find location and add this line:
location / {
...
proxy_set_header X-Forwarded-For $remote_addr;
...
}
More about reverse proxy.
Now in node.js app just add
app.set('trust proxy', true)
and you can get user ip now using:
req.ip
Making it work with recaptcha:
All about recaptcha is here: Google Developers
When You get user response token then you need to send post request to google to verify it. I did it using axios. This is how to send post request. Secret is your secret, response is user response.
const axios = require('axios');
axios
.post(`https://www.google.com/recaptcha/api/siteverify?secret=${secret}&response=${response}`, {}, {
headers: {
"Content-Type": "application/x-www-form-urlencoded; charset=utf-8"
},
})
.then(async function (tokenres) {
const {
success, //gives true or false value
challenge_ts,
hostname
} = tokenres.data;
if (success) {
//Do something
} else {
//For fail2ban. You need to make correct timestamp.
//Maybe its easier way to get this but on my level of experience
//I did it like this:
const now = new Date();
const tZOffset = now.getTimezoneOffset()/60;
const month = now.toLocaleString('en-US', { month: 'short' });
const day = now.getUTCDate();
const hours = now.getUTCHours()-tZOffset;
const minutes = now.getUTCMinutes();
const seconds = now.getUTCSeconds();
console.log(`${month} ${day} ${hours}:${minutes}:${seconds} Captcha verification failed [${req.ip}]`);
res.send(//something)
}
Time zone offset to set right time. Now pm2 save console.log instructions in log file in /home/youruserdir/.pm2/logs/yourappname-out.log
Make empty post request now. Example line of bad request will look like this:
Oct 14 19:5:3 Captcha verification failed [IP ADRESS]
Now I noticed that minutes and seconds have no 0 but fail2ban still recognizes them so its no problem. BUT CHECK IF DATE AND TIME PASSES WITH YOUR SYSTEM TIME.
Now make filter file for fail2ban:
sudo nano /etc/fail2ban/filter.d/your-filter.conf
paste:
[Definition]
failregex = Captcha verification failed \[<HOST>\]
ignoreregex =
Now ctrl+o, ctrl+x and you can check if fail2ban will recognize error lines using fail2ban-regex command:
fail2ban-regex /home/youruserdir/.pm2/logs/yourappname-out.log /etc/fail2ban/filter.d/your-filter.conf
Result will be:
Failregex: 38 total
|- #) [# of hits] regular expression
| 1) [38] Captcha verification failed \[<HOST>\]
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [38] {^LN-BEG}(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?
`-
Lines: 42 lines, 0 ignored, 38 matched, 4 missed
[processed in 0.04 sec]
As You can see 38 matched. You will have one. If You have no matches, check pm2 log file. When I was testing on localhost my app gave IP address with ::127.0.0.1. It can be ipv6 related. It can maybe make make a problem.
Next:
sudo nano /etc/fail2ban/jail.local
Add following block:
[Your-Jail-Name]
enabled = true
filter = your-filter
logpath = /home/YOURUSERDIR/.pm2/logs/YOUR-APP-NAME-out.log
maxretry = 5
findtime = 10m
bantime = 10m
So now. Be sure that you wrote filter name without .conf extension.
In logpath be sure to write right user dir and log name. If You will get 5(maxrety) wrong post requests in 10minutes(finditme) then user will be banned for 10 minutes. You can change this values.
Now just restart nginx and fail2ban:
sudo systemctl restart nginx
sudo systemctl restart fail2ban
After You can check if Your jail is working using commands:
sudo fail2ban-client status YOUR-JAIL-NAME
There will be written how much matches was found and how much ips are banned. More information You can find in fail2ban log file.
cat /var/log/fail2ban.log
Found IPADDR - 2021-10-13 13:12:57
NOTICE [YOUR-JAIL-NAME] Ban IPADDRES
I wrote this step-by-step because probably only people with little experience will look for this. If You see mistakes or you can suggest me something then just comment.
I am NO GOOD at regex - and as such i can not seem to match the plesk mail log string that indicates a brute force smtp attack -
my log looks like this:
May 19 03:24:58 gohhllc smtp_auth[22702]: SMTP connect from mail.globaltrbilisim.com [213.144.99.201]
May 19 03:24:58 gohhllc smtp_auth[22702]: No such user 'chuong#drophit.net' in mail authorization database
May 19 03:24:58 gohhllc smtp_auth[22702]: FAILED: chuong#drophit.net - password incorrect from mail.globaltrbilisim.com [213.144.99.201]
In some cases it also looks like this
May 19 03:25:22 gohhllc smtp_auth[23056]: SMTP connect from 89-97-124-22.fweds-spc.it [89.97.124.22]
May 19 03:25:22 gohhllc smtp_auth[23056]: FAILED: element - password incorrect from 89-97-124-22.fweds-spc.it [89.97.124.22]
My regex attempts to match both username failures and password look like this
failregex = No such user '.*' in mail authorization database
FAILED: .* - password incorrect from [<HOST>]
Along with 20+ other combos with no avail - most of the time teh result is an error like this
Unable to compile regular expression 'FAILED:
Thanks
I worked through this and using http://www.regexr.com/ i was able to write a fairly easy regex (i guess im getting better at it) to make this work.
The resulting statement for smtp-auth when using Pleask and Qmail (atleast on my server) is
failregex = FAILED: [-/\w]+ - password incorrect from <HOST>
AS for "no such user" entries i was unable to make this work as there is no hostname in the log file for this entry and fail2ban requires the hostname :(