diff options
| author | Yaroslav Halchenko <debian@onerussian.com> | 2016-12-09 09:37:33 -0500 |
|---|---|---|
| committer | Yaroslav Halchenko <debian@onerussian.com> | 2016-12-09 09:37:33 -0500 |
| commit | 623bb39ca6feb048602e9f570d9d5a30fe4dedef (patch) | |
| tree | c4ae811718f0ae4b0f3180a65aee6c477caf4563 | |
| parent | 8f42580c050fd2d229ba598afbb0c2e3718c0b3e (diff) | |
| parent | 3605155978efc95a2a44337645994dafc2f2b366 (diff) | |
| download | fail2ban-623bb39ca6feb048602e9f570d9d5a30fe4dedef.tar.gz | |
Merge branch 'enh-rel0.9.6' into debian
* enh-rel0.9.6: (60 commits)
updated man pages
ENH: prep for 0.9.6 release (as of tomorrow)
BF: added missing entires into MANIFEST
Update ChangeLog
ChangeLog entry added + jail.conf review
code review, makes the test cases workable, added dev-notes
ChangeLog update
`filter.d/apache-modsecurity.conf` - fixed for newer version (one space, closes gh-1626) reviewed and optimized: - non-greedy catch-all replaced for safer match - unneeded catch-all anchoring removed - non-capturing groups
filter.d/dovecot.conf update: - fixes failregex, that ignores failures through some irrelevant info (closes #1623); - ignores whole additionally irrelevant info in anchored regex before fixed failure data `\((?:auth failed, \d+ attempts( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\)` - review, IPv6 compatibility fix, non-capturing groups
Update jail.conf
Use Fedora's backend-settings for openSUSE
amend after code review of merge gh-1581
Make changes and add test file
Add Mongodb-auth filter and jail
Update FILTERS
filter.d/sshd.conf: Match 'Invalid user' with 'port \d*'
ChangeLog entry added
filter.d/sendmail-reject.conf: double space (should be by missing dns-host only) Closes #1578
Update Changelog to reflect the new np.conf action
Create npf.conf for the NPF packet filter
...
68 files changed, 1147 insertions, 320 deletions
diff --git a/.travis.yml b/.travis.yml index 50894a94..9ef607da 100644 --- a/.travis.yml +++ b/.travis.yml @@ -41,6 +41,8 @@ script: - if [[ "$F2B_PY_3" ]]; then coverage run bin/fail2ban-testcases; fi # Use $VENV_BIN (not python) or else sudo will always run the system's python (2.7) - sudo $VENV_BIN/pip install . + # Doc files should get installed on Travis under Linux + - test -e /usr/share/doc/fail2ban/FILTERS after_success: - coveralls - codecov @@ -6,7 +6,7 @@ Fail2Ban: Changelog =================== -ver. 0.9.5 (2016/07/15) - old-not-obsolete +ver. 0.9.6 (2016/12/10) - stretch-is-coming ----------- 0.9.x line is no longer heavily developed. If you are interested in @@ -14,6 +14,78 @@ new features (e.g. IPv6 support), please consider 0.10 branch and its releases. ### Fixes +* Misleading add resp. enable of (already available) jail in database, that + induced a subsequent error: last position of log file will be never retrieved (gh-795) +* Fixed a distribution related bug within testReadStockJailConfForceEnabled + (e.g. test-cases faults on Fedora, see gh-1353) +* Fixed pythonic filters and test scripts (running via wrong python version, + uses "fail2ban-python" now); +* Fixed test case "testSetupInstallRoot" for not default python version (also + using direct call, out of virtualenv); +* Fixed ambiguous wrong recognized date pattern resp. its optional parts (see gh-1512); +* FIPS compliant, use sha1 instead of md5 if it not allowed (see gh-1540) +* Monit config: scripting is not supported in path (gh-1556) +* `filter.d/apache-modsecurity.conf` + - Fixed for newer version (one space, gh-1626), optimized: non-greedy catch-all + replaced for safer match, unneeded catch-all anchoring removed, non-capturing +* `filter.d/asterisk.conf` + - Fixed to match different asterisk log prefix (source file: method:) +* `filter.d/dovecot.conf` + - Fixed failregex ignores failures through some not relevant info (gh-1623) +* `filter.d/ignorecommands/apache-fakegooglebot` + - Fixed error within apache-fakegooglebot, that will be called + with wrong python version (gh-1506) +* `filter.d/assp.conf` + - Extended failregex and test cases to handle ASSP V1 and V2 (gh-1494) +* `filter.d/postfix-sasl.conf` + - Allow for having no trailing space after 'failed:' (gh-1497) +* `filter.d/vsftpd.conf` + - Optional reason part in message after FAIL LOGIN (gh-1543) +* `filter.d/sendmail-reject.conf` + - removed mandatory double space (if dns-host available, gh-1579) +* filter.d/sshd.conf + - recognized "Failed publickey for" (gh-1477); + - optimized failregex to match all of "Failed any-method for ... from <HOST>" (gh-1479) + - eliminated possible complex injections (on user-name resp. auth-info, see gh-1479) + - optional port part after host (see gh-1533, gh-1581) + +### New Features +* New Actions: + - `action.d/npf.conf` for NPF, the latest packet filter for NetBSD +* New Filters: + - `filter.d/mongodb-auth.conf` for MongoDB (document-oriented NoSQL database engine) + (gh-1586, gh-1606 and gh-1607) + +### Enhancements +* DateTemplate regexp extended with the word-end boundary, additionally to + word-start boundary +* Introduces new command "fail2ban-python", as automatically created symlink to + python executable, where fail2ban currently installed (resp. its modules are located): + - allows to use the same version, fail2ban currently running, e.g. in + external scripts just via replace python with fail2ban-python: + ```diff + -#!/usr/bin/env python + +#!/usr/bin/env fail2ban-python + ``` + - always the same pickle protocol + - the same (and also guaranteed available) fail2ban modules + - simplified stand-alone install, resp. stand-alone installation possibility + via setup (like gh-1487) is getting closer +* Several test cases rewritten using new methods assertIn, assertNotIn +* New forward compatibility method assertRaisesRegexp (normally python >= 2.7). + Methods assertIn, assertNotIn, assertRaisesRegexp, assertLogged, assertNotLogged + are test covered now +* Jail configuration extended with new syntax to pass options to the backend (see gh-1408), + examples: + - `backend = systemd[journalpath=/run/log/journal/machine-1]` + - `backend = systemd[journalfiles="/run/log/journal/machine-1/system.journal, /run/log/journal/machine-1/user.journal"]` + - `backend = systemd[journalflags=2]` + + +ver. 0.9.5 (2016/07/15) - old-not-obsolete +----------- + +### Fixes * `filter.d/monit.conf` - Extended failregex with new monit "access denied" version (gh-1355) - failregex of previous monit version merged as single expression @@ -227,7 +227,7 @@ Regular expressions (failregex, ignoreregex) assume that the date/time has been removed from the log line (this is just how fail2ban works internally ATM). If the format is like '<date...> error 1.2.3.4 is evil' then you need to match -the < at the start so regex should be similar to '^<> <HOST> is evil$' using +the <> at the start so regex should be similar to '^<> error <HOST> is evil$' using <HOST> where the IP/domain name appears in the log line. The following general rules apply to regular expressions: @@ -33,12 +33,14 @@ config/action.d/iptables-new.conf config/action.d/iptables-xt_recent-echo.conf config/action.d/mail-buffered.conf config/action.d/mail.conf +config/action.d/mail-whois-common.conf config/action.d/mail-whois.conf config/action.d/mail-whois-lines.conf config/action.d/mynetwatchman.conf config/action.d/nftables-allports.conf config/action.d/nftables-common.conf config/action.d/nftables-multiport.conf +config/action.d/npf.conf config/action.d/nsupdate.conf config/action.d/osx-afctl.conf config/action.d/osx-ipfw.conf @@ -54,6 +56,7 @@ config/action.d/sendmail-whois-ipmatches.conf config/action.d/sendmail-whois-lines.conf config/action.d/sendmail-whois-matches.conf config/action.d/shorewall.conf +config/action.d/shorewall-ipset-proto6.conf config/action.d/smtp.py config/action.d/symbiosis-blacklist-allports.conf config/action.d/ufw.conf @@ -69,6 +72,7 @@ config/filter.d/apache-modsecurity.conf config/filter.d/apache-nohome.conf config/filter.d/apache-noscript.conf config/filter.d/apache-overflows.conf +config/filter.d/apache-pass.conf config/filter.d/apache-shellshock.conf config/filter.d/assp.conf config/filter.d/asterisk.conf @@ -81,11 +85,13 @@ config/filter.d/cyrus-imap.conf config/filter.d/directadmin.conf config/filter.d/dovecot.conf config/filter.d/dropbear.conf +config/filter.d/drupal-auth.conf config/filter.d/ejabberd-auth.conf config/filter.d/exim-common.conf config/filter.d/exim.conf config/filter.d/exim-spam.conf config/filter.d/freeswitch.conf +config/filter.d/froxlor-auth.conf config/filter.d/groupoffice.conf config/filter.d/gssftpd.conf config/filter.d/guacamole.conf @@ -95,6 +101,7 @@ config/filter.d/ignorecommands config/filter.d/ignorecommands/apache-fakegooglebot config/filter.d/kerio.conf config/filter.d/lighttpd-auth.conf +config/filter.d/mongodb-auth.conf config/filter.d/monit.conf config/filter.d/murmur.conf config/filter.d/mysqld-auth.conf @@ -150,6 +157,7 @@ config/paths-opensuse.conf config/paths-osx.conf CONTRIBUTING.md COPYING +.coveragerc DEVELOP doc/run-rootless.txt fail2ban-2to3 @@ -194,6 +202,7 @@ fail2ban/server/server.py fail2ban/server/strptime.py fail2ban/server/ticket.py fail2ban/server/transmitter.py +fail2ban/setup.py fail2ban-testcases-all fail2ban-testcases-all-python3 fail2ban/tests/action_d/__init__.py @@ -205,6 +214,7 @@ fail2ban/tests/banmanagertestcase.py fail2ban/tests/clientreadertestcase.py fail2ban/tests/config/action.d/brokenaction.conf fail2ban/tests/config/fail2ban.conf +fail2ban/tests/config/filter.d/common.conf fail2ban/tests/config/filter.d/simple.conf fail2ban/tests/config/filter.d/test.conf fail2ban/tests/config/filter.d/test.local @@ -256,6 +266,7 @@ fail2ban/tests/files/logs/apache-modsecurity fail2ban/tests/files/logs/apache-nohome fail2ban/tests/files/logs/apache-noscript fail2ban/tests/files/logs/apache-overflows +fail2ban/tests/files/logs/apache-pass fail2ban/tests/files/logs/apache-shellshock fail2ban/tests/files/logs/assp fail2ban/tests/files/logs/asterisk @@ -269,10 +280,12 @@ fail2ban/tests/files/logs/cyrus-imap fail2ban/tests/files/logs/directadmin fail2ban/tests/files/logs/dovecot fail2ban/tests/files/logs/dropbear +fail2ban/tests/files/logs/drupal-auth fail2ban/tests/files/logs/ejabberd-auth fail2ban/tests/files/logs/exim fail2ban/tests/files/logs/exim-spam fail2ban/tests/files/logs/freeswitch +fail2ban/tests/files/logs/froxlor-auth fail2ban/tests/files/logs/groupoffice fail2ban/tests/files/logs/gssftpd fail2ban/tests/files/logs/guacamole @@ -280,6 +293,7 @@ fail2ban/tests/files/logs/haproxy-http-auth fail2ban/tests/files/logs/horde fail2ban/tests/files/logs/kerio fail2ban/tests/files/logs/lighttpd-auth +fail2ban/tests/files/logs/mongodb-auth fail2ban/tests/files/logs/monit fail2ban/tests/files/logs/murmur fail2ban/tests/files/logs/mysqld-auth @@ -356,6 +370,8 @@ files/gentoo-confd files/gentoo-initd files/ipmasq-ZZZzzz_fail2ban.rul files/logwatch/fail2ban +files/logwatch/fail2ban-0.8.log +files/logwatch/fail2ban-0.9.log files/macosx-initd files/monit/fail2ban files/nagios/check_fail2ban @@ -373,8 +389,11 @@ man/fail2ban-regex.1 man/fail2ban-regex.h2m man/fail2ban-server.1 man/fail2ban-server.h2m +man/fail2ban-testcases.1 +man/fail2ban-testcases.h2m man/generate-man man/jail.conf.5 +.pylintrc README.md README.Solaris RELEASE @@ -2,7 +2,7 @@ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| - v0.9.5 2016/07/15 + v0.9.6 2016/12/10 ## Fail2Ban: ban hosts that cause multiple authentication errors @@ -39,8 +39,8 @@ Optional: To install, just do: - tar xvfj fail2ban-0.9.5.tar.bz2 - cd fail2ban-0.9.5 + tar xvfj fail2ban-0.9.6.tar.bz2 + cd fail2ban-0.9.6 python setup.py install This will install Fail2Ban into the python library directory. The executable @@ -53,7 +53,7 @@ Preparation or an alternative for comparison with previous release - git diff 0.9.5 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST + git diff 0.9.6 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST sort MANIFEST | uniq | sponge MANIFEST * Run:: @@ -70,7 +70,7 @@ Preparation * clean up current directory:: - diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.5/ + diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.6/ * Only differences should be files that you don't want distributed. @@ -83,7 +83,7 @@ Preparation * To generate a list of committers use e.g.:: - git shortlog -sn 0.9.5.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' + git shortlog -sn 0.9.6.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' * Ensure the top of the ChangeLog has the right version and current date. * Ensure the top entry of the ChangeLog has the right version and current date. @@ -106,7 +106,7 @@ Preparation * Tag the release by using a signed (and annotated) tag. Cut/paste release ChangeLog entry as tag annotation:: - git tag -s 0.9.5 + git tag -s 0.9.6 Pre Release =========== @@ -119,6 +119,7 @@ Thomas Mayer Tom Pike Tom Hendrikx Tomas Pihl +Thomas Skierlo (phaleas) Tony Lawrence Tomasz Ciolek Tyler diff --git a/bin/fail2ban-client b/bin/fail2ban-client index bc0c0be8..dc2f7d84 100755 --- a/bin/fail2ban-client +++ b/bin/fail2ban-client @@ -176,7 +176,7 @@ class Fail2banClient: if showRet: self.__logSocketError() return False - except Exception, e: + except Exception as e: if showRet: logSys.error(e) return False @@ -429,7 +429,7 @@ class Fail2banClient: elif not cmd == "": try: self.__processCommand(shlex.split(cmd)) - except Exception, e: + except Exception as e: logSys.error(e) except (EOFError, KeyboardInterrupt): print @@ -451,7 +451,7 @@ class Fail2banClient: ret = self.__configurator.getOptions(jail) self.__configurator.convertToProtocol() self.__stream = self.__configurator.getConfigStream() - except Exception, e: + except Exception as e: logSys.error("Failed during configuration: %s" % e) ret = False return ret diff --git a/bin/fail2ban-server b/bin/fail2ban-server index f522f418..23eae136 100755 --- a/bin/fail2ban-server +++ b/bin/fail2ban-server @@ -127,7 +127,7 @@ class Fail2banServer: self.__conf["pidfile"], self.__conf["force"]) return True - except Exception, e: + except Exception as e: logSys.exception(e) self.__server.quit() return False diff --git a/bin/fail2ban-testcases b/bin/fail2ban-testcases index dd6547a5..a711e07c 100755 --- a/bin/fail2ban-testcases +++ b/bin/fail2ban-testcases @@ -39,10 +39,18 @@ from fail2ban.version import version from fail2ban.tests.utils import gatherTests from fail2ban.helpers import FormatterWithTraceBack, getLogger +from fail2ban.setup import updatePyExec from fail2ban.server.mytime import MyTime from optparse import OptionParser, Option +# Update fail2ban-python env to current python version (where f2b-modules located/installed) +bindir = os.path.dirname( + # __file__ seems to be overwritten sometimes on some python versions (e.g. bug of 2.6 by running under cProfile, etc.): + sys.argv[0] if os.path.basename(sys.argv[0]) == 'fail2ban-testcases' else __file__ +) +updatePyExec(bindir) + def get_opt_parser(): # use module docstring for help output p = OptionParser( diff --git a/config/action.d/badips.conf b/config/action.d/badips.conf index 70b46546..6f9513f6 100644 --- a/config/action.d/badips.conf +++ b/config/action.d/badips.conf @@ -1,6 +1,6 @@ # Fail2ban reporting to badips.com # -# Note: This reports and IP only and does not actually ban traffic. Use +# Note: This reports an IP only and does not actually ban traffic. Use # another action in the same jail if you want bans to occur. # # Set the category to the appropriate value before use. diff --git a/config/action.d/npf.conf b/config/action.d/npf.conf new file mode 100644 index 00000000..8b00d177 --- /dev/null +++ b/config/action.d/npf.conf @@ -0,0 +1,61 @@ +# Fail2Ban configuration file +# +# NetBSD npf ban/unban +# +# Author: Nils Ratusznik <nils@NetBSD.org> +# Based on pf.conf action file +# + +[Definition] + +# Option: actionstart +# Notes.: command executed once at the start of Fail2Ban. +# Values: CMD +# +# we don't enable NPF automatically, as it will be enabled elsewhere +actionstart = + + +# Option: actionstop +# Notes.: command executed once at the end of Fail2Ban +# Values: CMD +# +# we don't disable NPF automatically either +actionstop = + + +# Option: actioncheck +# Notes.: command executed once before each actionban command +# Values: CMD +# +actioncheck = + + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: <ip> IP address +# <failures> number of failures +# <time> unix timestamp of the ban time +# Values: CMD +# +actionban = /sbin/npfctl table <tablename> add <ip> + + +# Option: actionunban +# Notes.: command executed when unbanning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: <ip> IP address +# <failures> number of failures +# <time> unix timestamp of the ban time +# Values: CMD +# +# note -r option used to remove matching rule +actionunban = /sbin/npfctl table <tablename> rem <ip> + +[Init] +# Option: tablename +# Notes.: The pf table name. +# Values: [ STRING ] +# +tablename = fail2ban diff --git a/config/filter.d/apache-modsecurity.conf b/config/filter.d/apache-modsecurity.conf index ad7e9b24..13e9c5ea 100644 --- a/config/filter.d/apache-modsecurity.conf +++ b/config/filter.d/apache-modsecurity.conf @@ -10,9 +10,10 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s ModSecurity: (\[.*?\] )*Access denied with code [45]\d\d.*$ +failregex = ^%(_apache_error_client)s ModSecurity:\s+(?:\[(?:\w+ \"[^\"]*\"|[^\]]*)\]\s*)*Access denied with code [45]\d\d ignoreregex = # https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats # Author: Daniel Black +# Sergey G. Brester aka sebres (review, optimization)
\ No newline at end of file diff --git a/config/filter.d/assp.conf b/config/filter.d/assp.conf index 2aa8958c..278e25cb 100644 --- a/config/filter.d/assp.conf +++ b/config/filter.d/assp.conf @@ -1,24 +1,43 @@ -# Fail2Ban filter for Anti-Spam SMTP Proxy Server also known as ASSP -# -# Honmepage: http://www.magicvillage.de/~Fritz_Borgstedt/assp/0003D91C-8000001C/ -# ProjektSite: http://sourceforge.net/projects/assp/?source=directory +# Fail2Ban filter for Anti-Spam SMTP Proxy Server (ASSP) +# Filter works in theory for both ASSP V1 and V2. Recommended ASSP is V2.5.1 or later. +# Support for ASSP V1 ended in 2014 so if you are still running ASSP V1 an immediate upgrade is recommended. +# +# Homepage: http://sourceforge.net/projects/assp/ +# ProjectSite: http://sourceforge.net/projects/assp/?source=directory # # [Definition] +# Note: First three failregex matches below are for ASSP V1 with the remaining being designed for V2. Deleting the V1 regex is recommended but I left it in for compatibilty reasons. __assp_actions = (?:dropping|refusing) failregex = ^(:? \[SSL-out\])? <HOST> max sender authentication errors \(\d{,3}\) exceeded -- %(__assp_actions)s connection - after reply: \d{3} \d{1}\.\d{1}.\d{1} Error: authentication failed: \w+;$ ^(?: \[SSL-out\])? <HOST> SSL negotiation with client failed: SSL accept attempt failed with unknown error.*:unknown protocol;$ ^ Blocking <HOST> - too much AUTH errors \(\d{,3}\);$ + ^\s*(?:[\w\-]+\s+)*(?:\[\S+\]\s+)*<HOST> (?:\<\S+@\S+\.\S+\> )*(?:to: \S+@\S+\.\S+ )*relay attempt blocked for(?: \(parsing\))?: \S+$ + ^\s*(?:[\w\-]+\s+)*(?:\[\S+\]\s+)*<HOST> \[SMTP Error\] 535 5\.7\.8 Error: authentication failed:\s+(?:\S+|Connection lost to authentication server|Invalid authentication mechanism|Invalid base64 data in continued response)?$ ignoreregex = # DEV Notes: +# V1 Examples matches: +# Apr-27-13 02:33:09 Blocking 217.194.197.97 - too much AUTH errors (41); +# Dec-29-12 17:10:31 [SSL-out] 200.247.87.82 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol; +# Dec-30-12 04:01:47 [SSL-out] 81.82.232.66 max sender authentication errors (5) exceeded # -# Examples: Apr-27-13 02:33:09 Blocking 217.194.197.97 - too much AUTH errors (41); -# Dec-29-12 17:10:31 [SSL-out] 200.247.87.82 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol; -# Dec-30-12 04:01:47 [SSL-out] 81.82.232.66 max sender authentication errors (5) exceeded +# V2 Examples matches: +# Jul-29-16 16:49:52 m1-25391-06124 [Worker_1] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user@example.org relay attempt blocked for: someone@example.org +# Jul-30-16 16:59:42 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6 +# Jul-30-16 00:15:36 m1-52131-09651 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6 +# Jul-31-16 06:45:59 [Worker_1] [TLS-in] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: +# Jan-05-16 08:38:49 m1-01129-09140 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> relay attempt blocked for (parsing): <user2@example> +# Jun-12-16 16:43:37 m1-64217-12013 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user2@example.com relay attempt blocked for (parsing): <a.notheruser69@example.c> +# Jan-22-16 22:25:51 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid authentication mechanism +# Mar-19-16 13:42:20 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid base64 data in continued response +# Jul-18-16 16:54:21 [Worker_2] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server +# Jul-18-16 17:14:23 m1-76453-02949 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server + # # Author: Enrico Labedzki (enrico.labedzki@deiwos.de) +# V2 Filters: Robert Hardy (rhardy@webcon.ca) diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index 6af452e2..79bd1ff2 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -16,7 +16,7 @@ __pid_re = (?:\[\d+\]) iso8601 = \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+[+-]\d{4} # All Asterisk log messages begin like this: -log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])? [^:]+:\d*( in \w+:)? +log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])? [^:]+:\d*(?:(?: in)? \w+:)? failregex = ^%(__prefix_line)s%(log_prefix)s Registration from '[^']*' failed for '<HOST>(:\d+)?' - (Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$ ^%(__prefix_line)s%(log_prefix)s Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index 136a3947..a3dc60bb 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -9,11 +9,11 @@ before = common.conf _daemon = (auth|dovecot(-auth)?|auth-worker) -failregex = ^%(__prefix_line)s(%(__pam_auth)s(\(dovecot:auth\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=<HOST>(\s+user=\S*)?\s*$ - ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((auth failed, \d+ attempts)( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=<HOST>(, lip=(\d{1,3}\.){3}\d{1,3})?(, TLS( handshaking(: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$ - ^%(__prefix_line)s(Info|dovecot: auth\(default\)|auth-worker\(\d+\)): pam\(\S+,<HOST>\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$ - ^%(__prefix_line)s(auth|auth-worker\(\d+\)): (pam|passwd-file)\(\S+,<HOST>\): unknown user\s*$ - ^%(__prefix_line)s(auth|auth-worker\(\d+\)): Info: ldap\(\S*,<HOST>,\S*\): invalid credentials\s*$ +failregex = ^%(__prefix_line)s(?:%(__pam_auth)s(?:\(dovecot:auth\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=<HOST>(?:\s+user=\S*)?\s*$ + ^%(__prefix_line)s(?:pop3|imap)-login: (?:Info: )?(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<[^>]+>,)?( method=\S+,)? rip=<HOST>(?:, lip=\S+)?(?:, TLS(?: handshaking(?:: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$ + ^%(__prefix_line)s(?:Info|dovecot: auth\(default\)|auth-worker\(\d+\)): pam\(\S+,<HOST>\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$ + ^%(__prefix_line)s(?:auth|auth-worker\(\d+\)): (?:pam|passwd-file)\(\S+,<HOST>\): unknown user\s*$ + ^%(__prefix_line)s(?:auth|auth-worker\(\d+\)): Info: ldap\(\S*,<HOST>,\S*\): invalid credentials\s*$ ignoreregex = @@ -30,3 +30,4 @@ journalmatch = _SYSTEMD_UNIT=dovecot.service # Author: Martin Waschbuesch # Daniel Black (rewrote with begin and end anchors) # Martin O'Neal (added LDAP authentication failure regex) +# Sergey G. Brester aka sebres (reviewed, optimized, IPv6-compatibility) diff --git a/config/filter.d/ignorecommands/apache-fakegooglebot b/config/filter.d/ignorecommands/apache-fakegooglebot index 19fb5107..9e6f4459 100755 --- a/config/filter.d/ignorecommands/apache-fakegooglebot +++ b/config/filter.d/ignorecommands/apache-fakegooglebot @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env fail2ban-python # Inspired by https://isc.sans.edu/forums/diary/When+Google+isnt+Google/15968/ # # Written in Python to reuse built-in Python batteries and not depend on diff --git a/config/filter.d/mongodb-auth.conf b/config/filter.d/mongodb-auth.conf new file mode 100644 index 00000000..66c27abb --- /dev/null +++ b/config/filter.d/mongodb-auth.conf @@ -0,0 +1,49 @@ +# Fail2Ban filter for unsuccesfull MongoDB authentication attempts +# +# Logfile /var/log/mongodb/mongodb.log +# +# add setting in /etc/mongodb.conf +# logpath=/var/log/mongodb/mongodb.log +# +# and use of the authentication +# auth = true +# + +[Definition] +#failregex = ^\s+\[initandlisten\] connection accepted from <HOST>:\d+ \#(?P<__connid>\d+) \(1 connection now open\)<SKIPLINES>\s+\[conn(?P=__connid)\] Failed to authenticate\s+ +failregex = ^\s+\[conn(?P<__connid>\d+)\] Failed to authenticate [^\n]+<SKIPLINES>\s+\[conn(?P=__connid)\] end connection <HOST> + +ignoreregex = + + +[Init] +maxlines = 10 + +# DEV Notes: +# +# Regarding the multiline regex: +# +# There can be a nunber of non-related lines between the first and second part +# of this regex maxlines of 10 is quite generious. +# +# Note the capture __connid, includes the connection ID, used in second part of regex. +# +# The first regex is commented out (but will match also), because it is better to use +# the host from "end connection" line (uncommented above): +# - it has the same prefix, searching begins directly with failure message +# (so faster, because ignores success connections at all) +# - it is not so vulnerable in case of possible race condition +# +# Log example: +# 2016-10-20T09:54:27.108+0200 [initandlisten] connection accepted from 127.0.0.1:53276 #1 (1 connection now open) +# 2016-10-20T09:54:27.109+0200 [conn1] authenticate db: test { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# 2016-10-20T09:54:27.110+0200 [conn1] Failed to authenticate root@test with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@test +# 2016-11-09T09:54:27.894+0100 [conn1] end connection 127.0.0.1:53276 (0 connections now open) +# 2016-11-09T11:55:58.890+0100 [initandlisten] connection accepted from 127.0.0.1:54266 #1510 (1 connection now open) +# 2016-11-09T11:55:58.892+0100 [conn1510] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# 2016-11-09T11:55:58.892+0100 [conn1510] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# 2016-11-09T11:55:58.894+0100 [conn1510] end connection 127.0.0.1:54266 (0 connections now open) +# +# Authors: Alexander Finkhäuser +# Sergey G. Brester (sebres) + diff --git a/config/filter.d/postfix-sasl.conf b/config/filter.d/postfix-sasl.conf index 4a6ceaaa..1a24ca94 100644 --- a/config/filter.d/postfix-sasl.conf +++ b/config/filter.d/postfix-sasl.conf @@ -9,7 +9,7 @@ before = common.conf _daemon = postfix(-\w+)?/(?:submission/|smtps/)?smtp[ds] -failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(: [ A-Za-z0-9+/:]*={0,2})?\s*$ +failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(:[ A-Za-z0-9+/:]*={0,2})?\s*$ ignoreregex = authentication failed: Connection lost to authentication server$ diff --git a/config/filter.d/sendmail-reject.conf b/config/filter.d/sendmail-reject.conf index 93b8343c..20d3648e 100644 --- a/config/filter.d/sendmail-reject.conf +++ b/config/filter.d/sendmail-reject.conf @@ -23,7 +23,7 @@ _daemon = (?:(sm-(mta|acceptingconnections)|sendmail)) failregex = ^%(__prefix_line)s\w{14}: ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[<HOST>\]( \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$ ^%(__prefix_line)sruleset=check_relay, arg1=(?P<dom>\S+), arg2=<HOST>, relay=((?P=dom) )?\[(\d+\.){3}\d+\]( \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$ - ^%(__prefix_line)s\w{14}: rejecting commands from (\S+ )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$ + ^%(__prefix_line)s\w{14}: rejecting commands from (\S* )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$ ^%(__prefix_line)s\w{14}: (\S+ )?\[<HOST>\]: ((?i)expn|vrfy) \S+ \[rejected\]$ ^(?P<__prefix>%(__prefix_line)s\w+: )<[^@]+@[^>]+>\.\.\. No such user here<SKIPLINES>(?P=__prefix)from=<[^@]+@[^>]+>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[<HOST>\]$ diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index eeb1518e..35cd8754 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -20,9 +20,9 @@ _daemon = sshd failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error|failed) for .* from <HOST>( via \S+)?\s*$ ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from <HOST>\s*$ - ^%(__prefix_line)sFailed \S+ for .*? from <HOST>(?: port \d*)?(?: ssh\d*)?(: (ruser .*|(\S+ ID \S+ \(serial \d+\) CA )?\S+ %(__md5hex)s(, client user ".*", client host ".*")?))?\s*$ + ^%(__prefix_line)sFailed \S+ for (?P<cond_inv>invalid user )?(?P<user>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)) from <HOST>(?: port \d+)?(?: ssh\d*)?(?(cond_user):|(?:(?:(?! from ).)*)$) ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$ - ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$ + ^%(__prefix_line)s[iI](?:llegal|nvalid) user .*? from <HOST>(?: port \d+)?\s*$ ^%(__prefix_line)sUser .+ from <HOST> not allowed because not listed in AllowUsers\s*$ ^%(__prefix_line)sUser .+ from <HOST> not allowed because listed in DenyUsers\s*$ ^%(__prefix_line)sUser .+ from <HOST> not allowed because not in any group\s*$ diff --git a/config/filter.d/vsftpd.conf b/config/filter.d/vsftpd.conf index 930b0d7e..2ecc44d3 100644 --- a/config/filter.d/vsftpd.conf +++ b/config/filter.d/vsftpd.conf @@ -14,7 +14,7 @@ __pam_re=\(?%(__pam_auth)s(?:\(\S+\))?\)?:? _daemon = vsftpd failregex = ^%(__prefix_line)s%(__pam_re)s\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=(ftp)? ruser=\S* rhost=<HOST>(?:\s+user=.*)?\s*$ - ^ \[pid \d+\] \[.+\] FAIL LOGIN: Client "<HOST>"\s*$ + ^ \[pid \d+\] \[[^\]]+\] FAIL LOGIN: Client "<HOST>"(?:\s*$|,) ignoreregex = diff --git a/config/jail.conf b/config/jail.conf index 41d8d3b8..d80e3d0a 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -731,6 +731,13 @@ logpath = %(mysql_log)s backend = %(mysql_backend)s +# Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf') +[mongodb-auth] +# change port when running with "--shardsvr" or "--configsvr" runtime operation +port = 27017 +logpath = /var/log/mongodb/mongodb.log + + # Jail for more extended banning of persistent abusers # !!! WARNINGS !!! # 1. Make sure that your loglevel specified in fail2ban.conf/.local @@ -810,8 +817,9 @@ maxretry = 1 [pass2allow-ftp] # this pass2allow example allows FTP traffic after successful HTTP authentication port = ftp,ftp-data,ftps,ftps-data -# knocking_url variable must be overridden to some secret value in filter.d/apache-pass.local -filter = apache-pass +# knocking_url variable must be overridden to some secret value in jail.local +knocking_url = /knocking/ +filter = apache-pass[knocking_url="%(knocking_url)s"] # access log of the website with HTTP auth logpath = %(apache_access_log)s blocktype = RETURN diff --git a/config/paths-opensuse.conf b/config/paths-opensuse.conf index 0d6ad522..227a5e98 100644 --- a/config/paths-opensuse.conf +++ b/config/paths-opensuse.conf @@ -36,3 +36,15 @@ mysql_log = /var/log/mysql/mysqld.log roundcube_errors_log = /srv/www/roundcubemail/logs/errors solidpop3d_log = %(syslog_mail)s + +# These services will log to the journal via syslog, so use the journal by +# default. +syslog_backend = systemd +sshd_backend = systemd +dropbear_backend = systemd +proftpd_backend = systemd +pureftpd_backend = systemd +wuftpd_backend = systemd +postfix_backend = systemd +dovecot_backend = systemd +mysql_backend = systemd diff --git a/fail2ban/client/configparserinc.py b/fail2ban/client/configparserinc.py index 7bbc7886..81f5bfb3 100644 --- a/fail2ban/client/configparserinc.py +++ b/fail2ban/client/configparserinc.py @@ -168,7 +168,7 @@ after = 1.conf parser, i = self._getSharedSCPWI(resource) if not i: return [] - except UnicodeDecodeError, e: + except UnicodeDecodeError as e: logSys.error("Error decoding config file '%s': %s" % (resource, e)) return [] diff --git a/fail2ban/client/configreader.py b/fail2ban/client/configreader.py index c6dd1b60..a48a0cc6 100644 --- a/fail2ban/client/configreader.py +++ b/fail2ban/client/configreader.py @@ -221,7 +221,7 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes): if not pOptions is None and option[1] in pOptions: continue values[option[1]] = v - except NoSectionError, e: + except NoSectionError as e: # No "Definition" section or wrong basedir logSys.error(e) values[option[1]] = option[2] diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index c2b2e894..71f50955 100755 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -329,7 +329,7 @@ class Fail2banRegex(object): if ret is not None: found = True regex = self._ignoreregex[ret].inc() - except RegexException, e: + except RegexException as e: output( e ) return False return found @@ -346,7 +346,7 @@ class Fail2banRegex(object): regex = self._failregex[match[0]] regex.inc() regex.appendIP(match) - except RegexException, e: + except RegexException as e: output( e ) return False except IndexError: @@ -510,7 +510,7 @@ class Fail2banRegex(object): output( "Use log file : %s" % cmd_log ) output( "Use encoding : %s" % self.encoding ) test_lines = self.file_lines_gen(hdlr) - except IOError, e: + except IOError as e: output( e ) return False elif cmd_log == "systemd-journal": # pragma: no cover diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py index c86c3153..5725f606 100644 --- a/fail2ban/client/jailreader.py +++ b/fail2ban/client/jailreader.py @@ -171,7 +171,7 @@ class JailReader(ConfigReader): self.__actions.append(action) else: raise AttributeError("Unable to read action") - except Exception, e: + except Exception as e: logSys.error("Error in action definition " + act) logSys.debug("Caught exception: %s" % (e,)) return False @@ -192,7 +192,7 @@ class JailReader(ConfigReader): stream = [] for opt in self.__opts: if opt == "logpath" and \ - self.__opts.get('backend', None) != "systemd": + not self.__opts.get('backend', None).startswith("systemd"): found_files = 0 for path in self.__opts[opt].split("\n"): path = path.rsplit(" ", 1) diff --git a/fail2ban/server/database.py b/fail2ban/server/database.py index 560fbfe5..34a8b39a 100644 --- a/fail2ban/server/database.py +++ b/fail2ban/server/database.py @@ -42,7 +42,7 @@ if sys.version_info >= (3,): try: x = json.dumps(x, ensure_ascii=False).encode( locale.getpreferredencoding(), 'replace') - except Exception, e: # pragma: no cover + except Exception as e: # pragma: no cover logSys.error('json dumps failed: %s', e) x = '{}' return x @@ -51,7 +51,7 @@ if sys.version_info >= (3,): try: x = json.loads(x.decode( locale.getpreferredencoding(), 'replace')) - except Exception, e: # pragma: no cover + except Exception as e: # pragma: no cover logSys.error('json loads failed: %s', e) x = {} return x @@ -70,7 +70,7 @@ else: try: x = json.dumps(_normalize(x), ensure_ascii=False).decode( locale.getpreferredencoding(), 'replace') - except Exception, e: # pragma: no cover + except Exception as e: # pragma: no cover logSys.error('json dumps failed: %s', e) x = '{}' return x @@ -79,7 +79,7 @@ else: try: x = _normalize(json.loads(x.decode( locale.getpreferredencoding(), 'replace'))) - except Exception, e: # pragma: no cover + except Exception as e: # pragma: no cover logSys.error('json loads failed: %s', e) x = {} return x @@ -175,7 +175,7 @@ class Fail2BanDb(object): logSys.info( "Connected to fail2ban persistent database '%s'", filename) - except sqlite3.OperationalError, e: + except sqlite3.OperationalError as e: logSys.error( "Error connecting to fail2ban persistent database '%s': %s", filename, e.args[0]) @@ -293,8 +293,12 @@ class Fail2BanDb(object): Jail to be added to the database. """ cur.execute( - "INSERT OR REPLACE INTO jails(name, enabled) VALUES(?, 1)", + "INSERT OR IGNORE INTO jails(name, enabled) VALUES(?, 1)", (jail.name,)) + if cur.rowcount <= 0: + cur.execute( + "UPDATE jails SET enabled = 1 WHERE name = ? AND enabled != 1", + (jail.name,)) @commitandrollback def delJail(self, cur, jail): @@ -317,7 +321,7 @@ class Fail2BanDb(object): cur.execute("UPDATE jails SET enabled=0") @commitandrollback - def getJailNames(self, cur): + def getJailNames(self, cur, enabled=None): """Get name of jails in database. Currently only used for testing purposes. @@ -327,7 +331,11 @@ class Fail2BanDb(object): set Set of jail names. """ - cur.execute("SELECT name FROM jails") + if enabled is None: + cur.execute("SELECT name FROM jails") + else: + cur.execute("SELECT name FROM jails WHERE enabled=%s" % + (int(enabled),)) return set(row[0] for row in cur.fetchmany()) @commitandrollback diff --git a/fail2ban/server/datetemplate.py b/fail2ban/server/datetemplate.py index 8210dad4..8e602289 100644 --- a/fail2ban/server/datetemplate.py +++ b/fail2ban/server/datetemplate.py @@ -64,7 +64,7 @@ class DateTemplate(object): def getRegex(self): return self._regex - def setRegex(self, regex, wordBegin=True): + def setRegex(self, regex, wordBegin=True, wordEnd=True): """Sets regex to use for searching for date in log line. Parameters @@ -72,8 +72,12 @@ class DateTemplate(object): regex : str The regex the template will use for searching for a date. wordBegin : bool - Defines whether the regex should be modified to search at - beginning of a word, by adding "\\b" to start of regex. + Defines whether the regex should be modified to search at beginning of a + word, by adding special boundary r'(?=^|\b|\W)' to start of regex. + Default True. + wordEnd : bool + Defines whether the regex should be modified to search at end of a word, + by adding special boundary r'(?=\b|\W|$)' to end of regex. Default True. Raises @@ -82,8 +86,10 @@ class DateTemplate(object): If regular expression fails to compile """ regex = regex.strip() - if (wordBegin and not re.search(r'^\^', regex)): - regex = r'\b' + regex + if wordBegin and not re.search(r'^\^', regex): + regex = r'(?=^|\b|\W)' + regex + if wordEnd and not re.search(r'\$$', regex): + regex += r'(?=\b|\W|$)' self._regex = regex self._cRegex = re.compile(regex, re.UNICODE | re.IGNORECASE) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 45ae7704..459a47d0 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -24,6 +24,7 @@ __license__ = "GPL" import codecs import fcntl import locale +import logging import os import re import sys @@ -82,6 +83,8 @@ class Filter(JailThread): self.__lastDate = None ## External command self.__ignoreCommand = False + ## Default or preferred encoding (to decode bytes from file or journal): + self.__encoding = locale.getpreferredencoding() self.dateDetector = DateDetector() self.dateDetector.addDefaultTemplate() @@ -105,7 +108,7 @@ class Filter(JailThread): logSys.warning( "Mutliline regex set for jail '%s' " "but maxlines not greater than 1") - except RegexException, e: + except RegexException as e: logSys.error(e) raise e @@ -138,7 +141,7 @@ class Filter(JailThread): try: regex = Regex(value) self.__ignoreRegex.append(regex) - except RegexException, e: + except RegexException as e: logSys.error(e) raise e @@ -280,6 +283,27 @@ class Filter(JailThread): return self.__lineBufferSize ## + # Set the log file encoding + # + # @param encoding the encoding used with log files + + def setLogEncoding(self, encoding): + if encoding.lower() == "auto": + encoding = locale.getpreferredencoding() + codecs.lookup(encoding) # Raise LookupError if invalid codec + self.__encoding = encoding + logSys.info("Set jail log file encoding to %s" % encoding) + return encoding + + ## + # Get the log file encoding + # + # @return log encoding value + + def getLogEncoding(self): + return self.__encoding + + ## # Main loop. # # This function is the main loop of the thread. It checks if the @@ -394,8 +418,31 @@ class Filter(JailThread): return False + if sys.version_info >= (3,): + @staticmethod + def uni_decode(x, enc, errors='strict'): + try: + if isinstance(x, bytes): + return x.decode(enc, errors) + return x + except (UnicodeDecodeError, UnicodeEncodeError): # pragma: no cover - unsure if reachable + if errors != 'strict': + raise + return uni_decode(x, enc, 'replace') + else: + @staticmethod + def uni_decode(x, enc, errors='strict'): + try: + if isinstance(x, unicode): + return x.encode(enc, errors) + return x + except (UnicodeDecodeError, UnicodeEncodeError): # pragma: no cover - unsure if reachable + if errors != 'strict': + raise + return uni_decode(x, enc, 'replace') + def processLine(self, line, date=None, returnRawHost=False, - checkAllRegex=False): + checkAllRegex=False, checkFindTime=False): """Split the time portion from log msg and return findFailures on them """ if date: @@ -414,21 +461,17 @@ class Filter(JailThread): tupleLine = (l, "", "") return "".join(tupleLine[::2]), self.findFailure( - tupleLine, date, returnRawHost, checkAllRegex) + tupleLine, date, returnRawHost, checkAllRegex, checkFindTime) def processLineAndAdd(self, line, date=None): """Processes the line for failures and populates failManager """ - for element in self.processLine(line, date)[1]: + for element in self.processLine(line, date, checkFindTime=True)[1]: ip = element[1] unixTime = element[2] lines = element[3] logSys.debug("Processing line with time:%s and ip:%s" % (unixTime, ip)) - if unixTime < MyTime.time() - self.getFindTime(): - logSys.debug("Ignore line since time %s < %s - %s" - % (unixTime, MyTime.time(), self.getFindTime())) - break if self.inIgnoreIPList(ip, log_ignore=True): continue logSys.info("[%s] Found %s" % (self.jail.name, ip)) @@ -457,7 +500,7 @@ class Filter(JailThread): # @return a dict with IP and timestamp. def findFailure(self, tupleLine, date=None, returnRawHost=False, - checkAllRegex=False): + checkAllRegex=False, checkFindTime=False): failList = list() # Checks if we must ignore this line. @@ -489,6 +532,11 @@ class Filter(JailThread): timeText = self.__lastTimeText or "".join(tupleLine[::2]) date = self.__lastDate + if checkFindTime and date is not None and date < MyTime.time() - self.getFindTime(): + logSys.log(5, "Ignore line since time %s < %s - %s", + date, MyTime.time(), self.getFindTime()) + return failList + self.__lineBuffer = ( self.__lineBuffer + [tupleLine])[-self.__lineBufferSize:] logSys.log(5, "Looking for failregex match of %r" % self.__lineBuffer) @@ -536,7 +584,7 @@ class Filter(JailThread): failRegex.getMatchedLines()]) if not checkAllRegex: break - except RegexException, e: # pragma: no cover - unsure if reachable + except RegexException as e: # pragma: no cover - unsure if reachable logSys.error(e) return failList @@ -554,7 +602,6 @@ class FileFilter(Filter): Filter.__init__(self, jail, **kwargs) ## The log file path. self.__logs = dict() - self.setLogEncoding("auto") ## # Add a log file path @@ -625,21 +672,9 @@ class FileFilter(Filter): # @param encoding the encoding used with log files def setLogEncoding(self, encoding): - if encoding.lower() == "auto": - encoding = locale.getpreferredencoding() - codecs.lookup(encoding) # Raise LookupError if invalid codec + encoding = super(FileFilter, self).setLogEncoding(encoding) for log in self.__logs.itervalues(): log.setEncoding(encoding) - self.__encoding = encoding - logSys.info("Set jail log file encoding to %s" % encoding) - - ## - # Get the log file encoding - # - # @return log encoding value - - def getLogEncoding(self): - return self.__encoding def getLog(self, path): return self.__logs.get(path, None) @@ -660,15 +695,15 @@ class FileFilter(Filter): try: has_content = log.open() # see http://python.org/dev/peps/pep-3151/ - except IOError, e: + except IOError as e: logSys.error("Unable to open %s" % filename) logSys.exception(e) return False - except OSError, e: # pragma: no cover - requires race condition to tigger this + except OSError as e: # pragma: no cover - requires race condition to tigger this logSys.error("Error opening %s" % filename) logSys.exception(e) return False - except Exception, e: # pragma: no cover - Requires implemention error in FileContainer to generate + except Exception as e: # pragma: no cover - Requires implemention error in FileContainer to generate logSys.error("Internal errror in FileContainer open method - please report as a bug to https://github.com/fail2ban/fail2ban/issues") logSys.exception(e) return False @@ -707,7 +742,12 @@ class FileFilter(Filter): try: import hashlib - md5sum = hashlib.md5 + try: + md5sum = hashlib.md5 + # try to use it (several standards like FIPS forbid it): + md5sum(' ').hexdigest() + except: # pragma: no cover + md5sum = hashlib.sha1 except ImportError: # pragma: no cover # hashlib was introduced in Python 2.5. For compatibility with those # elderly Pythons, import from md5 @@ -791,14 +831,19 @@ class FileContainer: @staticmethod def decode_line(filename, enc, line): try: - line = line.decode(enc, 'strict') - except UnicodeDecodeError: - logSys.warning( + return line.decode(enc, 'strict') + except (UnicodeDecodeError, UnicodeEncodeError) as e: + global _decode_line_warn + lev = logging.DEBUG + if _decode_line_warn.get(filename, 0) <= MyTime.time(): + lev = logging.WARNING + _decode_line_warn[filename] = MyTime.time() + 24*60*60 + logSys.log(lev, "Error decoding line from '%s' with '%s'." " Consider setting logencoding=utf-8 (or another appropriate" " encoding) for this jail. Continuing" - " to process line ignoring invalid characters: %r" % - (filename, enc, line)) + " to process line ignoring invalid characters: %r", + filename, enc, line) # decode with replacing error chars: line = line.decode(enc, 'replace') return line @@ -819,6 +864,8 @@ class FileContainer: ## print "D: Closed %s with pos %d" % (handler, self.__pos) ## sys.stdout.flush() +_decode_line_warn = {} + ## # JournalFilter class. @@ -858,11 +905,11 @@ class DNSUtils: # retrieve ip (todo: use AF_INET6 for IPv6) try: return set([i[4][0] for i in socket.getaddrinfo(dns, None, socket.AF_INET, 0, socket.IPPROTO_TCP)]) - except socket.error, e: + except socket.error as e: logSys.warning("Unable to find a corresponding IP address for %s: %s" % (dns, e)) return list() - except socket.error, e: + except socket.error as e: logSys.warning("Socket error raised trying to resolve hostname %s: %s" % (dns, e)) return list() @@ -871,7 +918,7 @@ class DNSUtils: def ipToName(ip): try: return socket.gethostbyaddr(ip)[0] - except socket.error, e: + except socket.error as e: logSys.debug("Unable to find a name for the IP %s: %s" % (ip, e)) return None diff --git a/fail2ban/server/filterpoll.py b/fail2ban/server/filterpoll.py index d0b37775..6d9f84b3 100644 --- a/fail2ban/server/filterpoll.py +++ b/fail2ban/server/filterpoll.py @@ -138,7 +138,7 @@ class FilterPoll(FileFilter): logSys.debug("%s has been modified", filename) self.__prevStats[filename] = stats return True - except OSError, e: + except OSError as e: logSys.error("Unable to get stat on %s because of: %s" % (filename, e)) self.__file404Cnt[filename] += 1 diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py index 100ad233..d551e0f4 100644 --- a/fail2ban/server/filterpyinotify.py +++ b/fail2ban/server/filterpyinotify.py @@ -44,7 +44,7 @@ if not hasattr(pyinotify, '__version__') \ try: manager = pyinotify.WatchManager() del manager -except Exception, e: +except Exception as e: raise ImportError("Pyinotify is probably not functional on this system: %s" % str(e)) diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index d0ebec95..3023155c 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -31,9 +31,9 @@ if LooseVersion(getattr(journal, '__version__', "0")) < '204': raise ImportError("Fail2Ban requires systemd >= 204") from .failmanager import FailManagerEmpty -from .filter import JournalFilter +from .filter import JournalFilter, Filter from .mytime import MyTime -from ..helpers import getLogger +from ..helpers import getLogger, logging, splitwords # Gets the instance of the logger. logSys = getLogger(__name__) @@ -54,14 +54,45 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover # @param jail the jail object def __init__(self, jail, **kwargs): + jrnlargs = FilterSystemd._getJournalArgs(kwargs) JournalFilter.__init__(self, jail, **kwargs) - self.__modified = False + self.__modified = 0 # Initialise systemd-journal connection - self.__journal = journal.Reader(converters={'__CURSOR': lambda x: x}) + self.__journal = journal.Reader(**jrnlargs) self.__matches = [] self.setDatePattern(None) + self.ticks = 0 logSys.debug("Created FilterSystemd") + @staticmethod + def _getJournalArgs(kwargs): + args = {'converters':{'__CURSOR': lambda x: x}} + try: + args['path'] = kwargs.pop('journalpath') + except KeyError: + pass + + try: + args['files'] = kwargs.pop('journalfiles') + except KeyError: + pass + else: + import glob + p = args['files'] + if not isinstance(p, (list, set, tuple)): + p = splitwords(p) + files = [] + for p in p: + files.extend(glob.glob(p)) + args['files'] = list(set(files)) + + try: + args['flags'] = kwargs.pop('journalflags') + except KeyError: + pass + + return args + ## # Add a journal match filters from list structure # @@ -139,21 +170,9 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover def getJournalMatch(self): return self.__matches - ## - # Join group of log elements which may be a mix of bytes and strings - # - # @param elements list of strings and bytes - # @return elements joined as string - - @staticmethod - def _joinStrAndBytes(elements): - strElements = [] - for element in elements: - if isinstance(element, str): - strElements.append(element) - else: - strElements.append(str(element, errors='ignore')) - return " ".join(strElements) + def uni_decode(self, x): + v = Filter.uni_decode(x, self.getLogEncoding()) + return v ## # Format journal log entry into syslog style @@ -161,52 +180,51 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover # @param entry systemd journal entry dict # @return format log line - @classmethod - def formatJournalEntry(cls, logentry): - logelements = [""] - if logentry.get('_HOSTNAME'): - logelements.append(logentry['_HOSTNAME']) - if logentry.get('SYSLOG_IDENTIFIER'): - logelements.append(logentry['SYSLOG_IDENTIFIER']) - if logentry.get('SYSLOG_PID'): - logelements[-1] += ("[%i]" % logentry['SYSLOG_PID']) - elif logentry.get('_PID'): - logelements[-1] += ("[%i]" % logentry['_PID']) - logelements[-1] += ":" - elif logentry.get('_COMM'): - logelements.append(logentry['_COMM']) - if logentry.get('_PID'): - logelements[-1] += ("[%i]" % logentry['_PID']) + def formatJournalEntry(self, logentry): + # Be sure, all argument of line tuple should have the same type: + uni_decode = self.uni_decode + logelements = [] + v = logentry.get('_HOSTNAME') + if v: + logelements.append(uni_decode(v)) + v = logentry.get('SYSLOG_IDENTIFIER') + if not v: + v = logentry.get('_COMM') + if v: + logelements.append(uni_decode(v)) + v = logentry.get('SYSLOG_PID') + if not v: + v = logentry.get('_PID') + if v: + logelements[-1] += ("[%i]" % v) logelements[-1] += ":" - if logelements[-1] == "kernel:": - if '_SOURCE_MONOTONIC_TIMESTAMP' in logentry: - monotonic = logentry.get('_SOURCE_MONOTONIC_TIMESTAMP') - else: - monotonic = logentry.get('__MONOTONIC_TIMESTAMP')[0] - logelements.append("[%12.6f]" % monotonic.total_seconds()) - if isinstance(logentry.get('MESSAGE',''), list): - logelements.append(" ".join(logentry['MESSAGE'])) + if logelements[-1] == "kernel:": + if '_SOURCE_MONOTONIC_TIMESTAMP' in logentry: + monotonic = logentry.get('_SOURCE_MONOTONIC_TIMESTAMP') + else: + monotonic = logentry.get('__MONOTONIC_TIMESTAMP')[0] + logelements.append("[%12.6f]" % monotonic.total_seconds()) + msg = logentry.get('MESSAGE','') + if isinstance(msg, list): + logelements.append(" ".join(uni_decode(v) for v in msg)) else: - logelements.append(logentry.get('MESSAGE', '')) + logelements.append(uni_decode(msg)) - try: - logline = u" ".join(logelements) - except UnicodeDecodeError: - # Python 2, so treat as string - logline = " ".join([str(logline) for logline in logelements]) - except TypeError: - # Python 3, one or more elements bytes - logSys.warning("Error decoding log elements from journal: %s" % - repr(logelements)) - logline = cls._joinStrAndBytes(logelements) + logline = " ".join(logelements) date = logentry.get('_SOURCE_REALTIME_TIMESTAMP', logentry.get('__REALTIME_TIMESTAMP')) logSys.debug("Read systemd journal entry: %r" % "".join([date.isoformat(), logline])) - return (('', date.isoformat(), logline), + ## use the same type for 1st argument: + return ((logline[:0], date.isoformat(), logline), time.mktime(date.timetuple()) + date.microsecond/1.0E6) + def seekToTime(self, date): + if not isinstance(date, datetime.datetime): + date = datetime.datetime.fromtimestamp(date) + self.__journal.seek_realtime(date) + ## # Main loop. # @@ -224,7 +242,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover # Seek to now - findtime in journal start_time = datetime.datetime.now() - \ datetime.timedelta(seconds=int(self.getFindTime())) - self.__journal.seek_realtime(start_time) + self.seekToTime(start_time) # Move back one entry to ensure do not end up in dead space # if start time beyond end of journal try: @@ -233,29 +251,38 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover pass # Reading failure, so safe to ignore while self.active: - if not self.idle: - while self.active: - try: - logentry = self.__journal.get_next() - except OSError: - logSys.warning( - "Error reading line from systemd journal") - continue - if logentry: - self.processLineAndAdd( - *self.formatJournalEntry(logentry)) - self.__modified = True - else: - break - if self.__modified: - try: - while True: - ticket = self.failManager.toBan() - self.jail.putFailTicket(ticket) - except FailManagerEmpty: - self.failManager.cleanup(MyTime.time()) - self.__modified = False + # wait for records (or for timeout in sleeptime seconds): self.__journal.wait(self.sleeptime) + if self.idle: + # because journal.wait will returns immediatelly if we have records in journal, + # just wait a little bit here for not idle, to prevent hi-load: + time.sleep(self.sleeptime) + continue + self.__modified = 0 + while self.active: + logentry = None + try: + logentry = self.__journal.get_next() + except OSError as e: + logSys.error("Error reading line from systemd journal: %s", + e, exc_info=logSys.getEffectiveLevel() <= logging.DEBUG) + self.ticks += 1 + if logentry: + self.processLineAndAdd( + *self.formatJournalEntry(logentry)) + self.__modified += 1 + if self.__modified >= 100: # todo: should be configurable + break + else: + break + if self.__modified: + try: + while True: + ticket = self.failManager.toBan() + self.jail.putFailTicket(ticket) + except FailManagerEmpty: + self.failManager.cleanup(MyTime.time()) + logSys.debug((self.jail is not None and self.jail.name or "jailless") +" filter terminated") return True diff --git a/fail2ban/server/jail.py b/fail2ban/server/jail.py index a866cb51..5cbd17be 100644 --- a/fail2ban/server/jail.py +++ b/fail2ban/server/jail.py @@ -27,6 +27,7 @@ import logging import Queue from .actions import Actions +from ..client.jailreader import JailReader from ..helpers import getLogger # Gets the instance of the logger. @@ -82,6 +83,7 @@ class Jail: return "%s(%r)" % (self.__class__.__name__, self.name) def _setBackend(self, backend): + backend, beArgs = JailReader.extractOptions(backend) backend = backend.lower() # to assure consistent matching backends = self._BACKENDS @@ -98,7 +100,7 @@ class Jail: for b in backends: initmethod = getattr(self, '_init%s' % b.capitalize()) try: - initmethod() + initmethod(**beArgs) if backend != 'auto' and b != backend: logSys.warning("Could only initiated %r backend whenever " "%r was requested" % (b, backend)) @@ -106,7 +108,7 @@ class Jail: logSys.info("Initiated %r backend" % b) self.__actions = Actions(self) return # we are done - except ImportError, e: + except ImportError as e: # Log debug if auto, but error if specific logSys.log( logging.DEBUG if backend == "auto" else logging.ERROR, @@ -117,28 +119,28 @@ class Jail: raise RuntimeError( "Failed to initialize any backend for Jail %r" % self.name) - def _initPolling(self): + def _initPolling(self, **kwargs): from filterpoll import FilterPoll - logSys.info("Jail '%s' uses poller" % self.name) - self.__filter = FilterPoll(self) + logSys.info("Jail '%s' uses poller %r" % (self.name, kwargs)) + self.__filter = FilterPoll(self, **kwargs) - def _initGamin(self): + def _initGamin(self, **kwargs): # Try to import gamin from filtergamin import FilterGamin - logSys.info("Jail '%s' uses Gamin" % self.name) - self.__filter = FilterGamin(self) + logSys.info("Jail '%s' uses Gamin %r" % (self.name, kwargs)) + self.__filter = FilterGamin(self, **kwargs) - def _initPyinotify(self): + def _initPyinotify(self, **kwargs): # Try to import pyinotify from filterpyinotify import FilterPyinotify - logSys.info("Jail '%s' uses pyinotify" % self.name) - self.__filter = FilterPyinotify(self) + logSys.info("Jail '%s' uses pyinotify %r" % (self.name, kwargs)) + self.__filter = FilterPyinotify(self, **kwargs) - def _initSystemd(self): # pragma: systemd no cover + def _initSystemd(self, **kwargs): # pragma: systemd no cover # Try to import systemd from filtersystemd import FilterSystemd - logSys.info("Jail '%s' uses systemd" % self.name) - self.__filter = FilterSystemd(self) + logSys.info("Jail '%s' uses systemd %r" % (self.name, kwargs)) + self.__filter = FilterSystemd(self, **kwargs) @property def name(self): diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index 3e371945..abaf4dfd 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -108,20 +108,20 @@ class Server: pidFile = open(pidfile, 'w') pidFile.write("%s\n" % os.getpid()) pidFile.close() - except IOError, e: + except IOError as e: logSys.error("Unable to create PID file: %s" % e) # Start the communication logSys.debug("Starting communication") try: self.__asyncServer.start(sock, force) - except AsyncServerException, e: + except AsyncServerException as e: logSys.error("Could not start server: %s", e) # Removes the PID file. try: logSys.debug("Remove PID file %s" % pidfile) os.remove(pidfile) - except OSError, e: + except OSError as e: logSys.error("Unable to remove PID file: %s" % e) logSys.info("Exiting Fail2ban") @@ -237,13 +237,11 @@ class Server: def setLogEncoding(self, name, encoding): filter_ = self.__jails[name].filter - if isinstance(filter_, FileFilter): - filter_.setLogEncoding(encoding) + filter_.setLogEncoding(encoding) def getLogEncoding(self, name): filter_ = self.__jails[name].filter - if isinstance(filter_, FileFilter): - return filter_.getLogEncoding() + return filter_.getLogEncoding() def setFindTime(self, name, value): self.__jails[name].filter.setFindTime(value) @@ -543,7 +541,7 @@ class Server: # the child gets a new PID, making it impossible for its PID to equal its # PGID. pid = os.fork() - except OSError, e: + except OSError as e: return((e.errno, e.strerror)) # ERROR (return a tuple) if pid == 0: # The first child. @@ -564,7 +562,7 @@ class Server: # fork guarantees that the child is no longer a session leader, thus # preventing the daemon from ever acquiring a controlling terminal. pid = os.fork() # Fork a second child. - except OSError, e: + except OSError as e: return((e.errno, e.strerror)) # ERROR (return a tuple) if (pid == 0): # The second child. diff --git a/fail2ban/server/transmitter.py b/fail2ban/server/transmitter.py index 0d9f0fe4..636b90da 100644 --- a/fail2ban/server/transmitter.py +++ b/fail2ban/server/transmitter.py @@ -56,7 +56,7 @@ class Transmitter: try: ret = self.__commandHandler(command) ack = 0, ret - except Exception, e: + except Exception as e: logSys.warning("Command %r has failed. Received %r" % (command, e)) ack = 1, e diff --git a/fail2ban/setup.py b/fail2ban/setup.py new file mode 100644 index 00000000..87d50e66 --- /dev/null +++ b/fail2ban/setup.py @@ -0,0 +1,42 @@ +# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
+# vi: set ft=python sts=4 ts=4 sw=4 noet :
+
+# This file is part of Fail2Ban.
+#
+# Fail2Ban is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Fail2Ban is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Fail2Ban; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+__author__ = "Serg G. Brester"
+__license__ = "GPL"
+
+import os
+import sys
+
+
+def updatePyExec(bindir, executable=None):
+ """Update fail2ban-python link to current python version (where f2b-modules located/installed)
+ """
+ bindir = os.path.realpath(bindir)
+ if executable is None:
+ executable = sys.executable
+ pypath = os.path.join(bindir, 'fail2ban-python')
+ # if not exists or point to another version - update link:
+ isfile = os.path.isfile(pypath)
+ if not isfile or os.path.realpath(pypath) != os.path.realpath(executable):
+ if isfile:
+ os.unlink(pypath)
+ os.symlink(executable, pypath)
+ # extend current environment path (e.g. if fail2ban not yet installed):
+ if bindir not in os.environ["PATH"].split(os.pathsep):
+ os.environ["PATH"] = os.environ["PATH"] + os.pathsep + bindir;
diff --git a/fail2ban/tests/action_d/test_badips.py b/fail2ban/tests/action_d/test_badips.py index 3f71b7a3..b0f8b3c3 100644 --- a/fail2ban/tests/action_d/test_badips.py +++ b/fail2ban/tests/action_d/test_badips.py @@ -49,7 +49,7 @@ if sys.version_info >= (2,7): def testCategory(self): categories = self.action.getCategories() - self.assertTrue("ssh" in categories) + self.assertIn("ssh", categories) self.assertTrue(len(categories) >= 10) self.assertRaises( diff --git a/fail2ban/tests/action_d/test_smtp.py b/fail2ban/tests/action_d/test_smtp.py index 35ac2393..b8328743 100644 --- a/fail2ban/tests/action_d/test_smtp.py +++ b/fail2ban/tests/action_d/test_smtp.py @@ -101,21 +101,21 @@ class SMTPActionTest(unittest.TestCase): self.assertEqual(self.smtpd.rcpttos, ["root"]) subject = "Subject: [Fail2Ban] %s: banned %s" % ( self.jail.name, aInfo['ip']) - self.assertTrue(subject in self.smtpd.data.replace("\n", "")) + self.assertIn(subject, self.smtpd.data.replace("\n", "")) self.assertTrue( "%i attempts" % aInfo['failures'] in self.smtpd.data) self.action.matches = "matches" self.action.ban(aInfo) - self.assertTrue(aInfo['matches'] in self.smtpd.data) + self.assertIn(aInfo['matches'], self.smtpd.data) self.action.matches = "ipjailmatches" self.action.ban(aInfo) - self.assertTrue(aInfo['ipjailmatches'] in self.smtpd.data) + self.assertIn(aInfo['ipjailmatches'], self.smtpd.data) self.action.matches = "ipmatches" self.action.ban(aInfo) - self.assertTrue(aInfo['ipmatches'] in self.smtpd.data) + self.assertIn(aInfo['ipmatches'], self.smtpd.data) def testOptions(self): self.action.start() diff --git a/fail2ban/tests/actionstestcase.py b/fail2ban/tests/actionstestcase.py index 0ceb35d5..3b9d2d01 100644 --- a/fail2ban/tests/actionstestcase.py +++ b/fail2ban/tests/actionstestcase.py @@ -65,12 +65,12 @@ class ExecuteActions(LogCaptureTestCase): def testActionsManipulation(self): self.__actions.add('test') self.assertTrue(self.__actions['test']) - self.assertTrue('test' in self.__actions) - self.assertFalse('nonexistant action' in self.__actions) + self.assertIn('test', self.__actions) + self.assertNotIn('nonexistant action', self.__actions) self.__actions.add('test1') del self.__actions['test'] del self.__actions['test1'] - self.assertFalse('test' in self.__actions) + self.assertNotIn('test', self.__actions) self.assertEqual(len(self.__actions), 0) self.__actions.setBanTime(127) diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 0a3734e5..420d4b29 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -36,7 +36,7 @@ from ..client.jailsreader import JailsReader from ..client.actionreader import ActionReader from ..client.configurator import Configurator from ..version import version -from .utils import LogCaptureTestCase +from .utils import LogCaptureTestCase, with_tmpdir TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files") @@ -94,9 +94,8 @@ option = %s if not os.access(f, os.R_OK): self.assertFalse(self.c.read('d')) # should not be readable BUT present else: - # SkipTest introduced only in 2.7 thus can't yet use generally - # raise unittest.SkipTest("Skipping on %s -- access rights are not enforced" % platform) - pass + import platform + raise unittest.SkipTest("Skipping on %s -- access rights are not enforced" % platform.platform()) def testOptionalDotDDir(self): self.assertFalse(self.c.read('c')) # nothing is there yet @@ -281,8 +280,8 @@ class JailReaderTest(LogCaptureTestCase): self.assertEqual(eval(act[2][5]).get('agent', '<wrong>'), useragent) self.assertEqual(act[3], ['set', 'blocklisttest', 'action', 'mynetwatchman', 'agent', useragent]) - def testGlob(self): - d = tempfile.mkdtemp(prefix="f2b-temp") + @with_tmpdir + def testGlob(self, d): # Generate few files # regular file f1 = os.path.join(d, 'f1') @@ -297,9 +296,6 @@ class JailReaderTest(LogCaptureTestCase): self.assertEqual(JailReader._glob(f2), []) self.assertLogged('File %s is a dangling link, thus cannot be monitored' % f2) self.assertEqual(JailReader._glob(os.path.join(d, 'nonexisting')), []) - os.remove(f1) - os.remove(f2) - os.rmdir(d) class FilterReaderTest(unittest.TestCase): @@ -410,7 +406,7 @@ class FilterReaderTest(unittest.TestCase): # from testcase01 filterReader.get('Definition', 'failregex') filterReader.get('Definition', 'ignoreregex') - except Exception, e: # pragma: no cover - failed if reachable + except Exception as e: # pragma: no cover - failed if reachable self.fail('unexpected options after readexplicit: %s' % (e)) @@ -433,10 +429,10 @@ class JailsReaderTestCache(LogCaptureTestCase): cnt += 1 return cnt - def testTestJailConfCache(self): + @with_tmpdir + def testTestJailConfCache(self, basedir): saved_ll = configparserinc.logLevel configparserinc.logLevel = logging.DEBUG - basedir = tempfile.mkdtemp("fail2ban_conf") try: shutil.rmtree(basedir) shutil.copytree(CONFIG_DIR, basedir) @@ -468,7 +464,6 @@ class JailsReaderTestCache(LogCaptureTestCase): cnt = self._getLoggedReadCount(r'action\.d/iptables-common\.conf') self.assertTrue(cnt == 1, "Unexpected count by reading of action files, cnt = %s" % cnt) finally: - shutil.rmtree(basedir) configparserinc.logLevel = saved_ll @@ -525,12 +520,12 @@ class JailsReaderTest(LogCaptureTestCase): self.assertTrue(actionReader.read()) actionReader.getOptions({}) # populate _opts if not actionName.endswith('-common'): - self.assertTrue('Definition' in actionReader.sections(), + self.assertIn('Definition', actionReader.sections(), msg="Action file %r is lacking [Definition] section" % actionConfig) # all must have some actionban defined self.assertTrue(actionReader._opts.get('actionban', '').strip(), msg="Action file %r is lacking actionban" % actionConfig) - self.assertTrue('Init' in actionReader.sections(), + self.assertIn('Init', actionReader.sections(), msg="Action file %r is lacking [Init] section" % actionConfig) def testReadStockJailConf(self): @@ -582,7 +577,7 @@ class JailsReaderTest(LogCaptureTestCase): self.assertTrue(len(actName)) self.assertTrue(isinstance(actOpt, dict)) if actName == 'iptables-multiport': - self.assertTrue('port' in actOpt) + self.assertIn('port', actOpt) actionReader = ActionReader( actName, jail, {}, basedir=CONFIG_DIR) @@ -632,11 +627,13 @@ class JailsReaderTest(LogCaptureTestCase): # and we know even some of them by heart for j in ['sshd', 'recidive']: - # by default we have 'auto' backend ATM - self.assertTrue(['add', j, 'auto'] in comm_commands) + # by default we have 'auto' backend ATM, but some distributions can overwrite it, + # (e.g. fedora default is 'systemd') therefore let check it without backend... + self.assertIn(['add', j], + (cmd[:2] for cmd in comm_commands if len(cmd) == 3 and cmd[0] == 'add')) # and warn on useDNS - self.assertTrue(['set', j, 'usedns', 'warn'] in comm_commands) - self.assertTrue(['start', j] in comm_commands) + self.assertIn(['set', j, 'usedns', 'warn'], comm_commands) + self.assertIn(['start', j], comm_commands) # last commands should be the 'start' commands self.assertEqual(comm_commands[-1][0], 'start') @@ -655,7 +652,7 @@ class JailsReaderTest(LogCaptureTestCase): action_name = action.getName() if '<blocktype>' in str(commands): # Verify that it is among cInfo - self.assertTrue('blocktype' in action._initOpts) + self.assertIn('blocktype', action._initOpts) # Verify that we have a call to set it up blocktype_present = False target_command = ['set', jail_name, 'action', action_name, 'blocktype'] @@ -716,8 +713,8 @@ class JailsReaderTest(LogCaptureTestCase): self.assertEqual(configurator._Configurator__jails.getBaseDir(), '/tmp') self.assertEqual(configurator.getBaseDir(), CONFIG_DIR) - def testMultipleSameAction(self): - basedir = tempfile.mkdtemp("fail2ban_conf") + @with_tmpdir + def testMultipleSameAction(self, basedir): os.mkdir(os.path.join(basedir, "filter.d")) os.mkdir(os.path.join(basedir, "action.d")) open(os.path.join(basedir, "action.d", "testaction1.conf"), 'w').close() @@ -746,4 +743,33 @@ filter = testfilter1 # Python actions should not be passed `actname` self.assertEqual(add_actions[-1][-1], "{}") - shutil.rmtree(basedir) + def testLogPathFileFilterBackend(self): + self.assertRaisesRegexp(ValueError, r"Have not found any log file for .* jail", + self._testLogPath, backend='polling') + + def testLogPathSystemdBackend(self): + try: # pragma: systemd no cover + from ..server.filtersystemd import FilterSystemd + except Exception, e: # pragma: no cover + raise unittest.SkipTest("systemd python interface not available") + self._testLogPath(backend='systemd') + self._testLogPath(backend='systemd[journalflags=2]') + + @with_tmpdir + def _testLogPath(self, basedir, backend): + jailfd = open(os.path.join(basedir, "jail.conf"), 'w') + jailfd.write(""" +[testjail1] +enabled = true +backend = %s +logpath = %s/not/exist.log + /this/path/should/not/exist.log +action = +filter = +failregex = test <HOST> +""" % (backend, basedir)) + jailfd.close() + jails = JailsReader(basedir=basedir) + self.assertTrue(jails.read()) + self.assertTrue(jails.getOptions()) + jails.convert() diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py index 3d156eda..2b599f61 100644 --- a/fail2ban/tests/databasetestcase.py +++ b/fail2ban/tests/databasetestcase.py @@ -48,12 +48,10 @@ class DatabaseTest(LogCaptureTestCase): def setUp(self): """Call before every test case.""" super(DatabaseTest, self).setUp() - if Fail2BanDb is None and sys.version_info >= (2,7): # pragma: no cover + if Fail2BanDb is None: # pragma: no cover raise unittest.SkipTest( "Unable to import fail2ban database module as sqlite is not " "available.") - elif Fail2BanDb is None: - return _, self.dbFilename = tempfile.mkstemp(".db", "fail2ban_") self.db = Fail2BanDb(self.dbFilename) @@ -123,7 +121,7 @@ class DatabaseTest(LogCaptureTestCase): self.db.addLog(self.jail, self.fileContainer) - self.assertTrue(filename in self.db.getLogPaths(self.jail)) + self.assertIn(filename, self.db.getLogPaths(self.jail)) os.remove(filename) def testUpdateLog(self): @@ -318,6 +316,25 @@ class DatabaseTest(LogCaptureTestCase): actions._Actions__checkBan() self.assertLogged("ban ainfo %s, %s, %s, %s" % (True, True, True, True)) + def testDelAndAddJail(self): + self.testAddJail() # Add jail + # Delete jail (just disabled it): + self.db.delJail(self.jail) + jails = self.db.getJailNames() + self.assertIn(len(jails) == 1 and self.jail.name, jails) + jails = self.db.getJailNames(enabled=False) + self.assertIn(len(jails) == 1 and self.jail.name, jails) + jails = self.db.getJailNames(enabled=True) + self.assertTrue(len(jails) == 0) + # Add it again - should just enable it: + self.db.addJail(self.jail) + jails = self.db.getJailNames() + self.assertIn(len(jails) == 1 and self.jail.name, jails) + jails = self.db.getJailNames(enabled=True) + self.assertIn(len(jails) == 1 and self.jail.name, jails) + jails = self.db.getJailNames(enabled=False) + self.assertTrue(len(jails) == 0) + def testPurge(self): if Fail2BanDb is None: # pragma: no cover return diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 3321ffd8..1119efdb 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -39,7 +39,7 @@ except ImportError: from ..client import fail2banregex from ..client.fail2banregex import Fail2banRegex, get_opt_parser, output -from .utils import LogCaptureTestCase, logSys +from .utils import setUpMyTime, tearDownMyTime, LogCaptureTestCase, logSys from .utils import CONFIG_DIR @@ -70,10 +70,12 @@ class Fail2banRegexTest(LogCaptureTestCase): def setUp(self): """Call before every test case.""" LogCaptureTestCase.setUp(self) + setUpMyTime() def tearDown(self): """Call after every test case.""" LogCaptureTestCase.tearDown(self) + tearDownMyTime() def testWrongRE(self): (opts, args, fail2banRegex) = _Fail2banRegex( @@ -159,8 +161,8 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertTrue(fail2banRegex.start(opts, args)) self.assertLogged('Lines: 13 lines, 0 ignored, 5 matched, 8 missed') - self.assertLogged('141.3.81.106 Fri Aug 14 11:53:59 2015') - self.assertLogged('141.3.81.106 Fri Aug 14 11:54:59 2015') + self.assertLogged('141.3.81.106 Sun Aug 14 11:53:59 2005') + self.assertLogged('141.3.81.106 Sun Aug 14 11:54:59 2005') def testWronChar(self): (opts, args, fail2banRegex) = _Fail2banRegex( @@ -169,9 +171,8 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertTrue(fail2banRegex.start(opts, args)) self.assertLogged('Lines: 4 lines, 0 ignored, 2 matched, 2 missed') - self.assertLogged('Error decoding line'); - self.assertLogged('Continuing to process line ignoring invalid characters:', '2015-01-14 20:00:58 user '); - self.assertLogged('Continuing to process line ignoring invalid characters:', '2015-01-14 20:00:59 user '); + self.assertLogged('Error decoding line') + self.assertLogged('Continuing to process line ignoring invalid characters:') self.assertLogged('Nov 8 00:16:12 main sshd[32548]: input_userauth_request: invalid user llinco') self.assertLogged('Nov 8 00:16:12 main sshd[32547]: pam_succeed_if(sshd:auth): error retrieving information about user llinco') diff --git a/fail2ban/tests/files/config/apache-auth/digest.py b/fail2ban/tests/files/config/apache-auth/digest.py index 875ebffe..03588594 100755 --- a/fail2ban/tests/files/config/apache-auth/digest.py +++ b/fail2ban/tests/files/config/apache-auth/digest.py @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env fail2ban-python import requests try: diff --git a/fail2ban/tests/files/ignorecommand.py b/fail2ban/tests/files/ignorecommand.py index dd6b5aab..7011b51b 100755 --- a/fail2ban/tests/files/ignorecommand.py +++ b/fail2ban/tests/files/ignorecommand.py @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env fail2ban-python import sys if sys.argv[1] == "10.0.0.1": exit(0) diff --git a/fail2ban/tests/files/logs/apache-modsecurity b/fail2ban/tests/files/logs/apache-modsecurity index d46d8ab4..3ca2e074 100644 --- a/fail2ban/tests/files/logs/apache-modsecurity +++ b/fail2ban/tests/files/logs/apache-modsecurity @@ -1,5 +1,5 @@ # failJSON: { "time": "2013-12-23T13:12:31", "match": true , "host": "173.255.225.101" } [Mon Dec 23 13:12:31 2013] [error] [client 173.255.225.101] ModSecurity: [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"][tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [hostname "www.mysite.net"] [uri "/"] [unique_id "Urf@f12qgHIAACrFOlgAAABA"] -# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69" } -[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] +# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69", "desc": "additional entry (and exact one space)" } +[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] diff --git a/fail2ban/tests/files/logs/assp b/fail2ban/tests/files/logs/assp index 2c658eb9..21b01f9f 100644 --- a/fail2ban/tests/files/logs/assp +++ b/fail2ban/tests/files/logs/assp @@ -22,4 +22,23 @@ Apr-27-13 02:25:10 [SSL-out] 217.194.197.97 max sender authentication errors (5) Apr-27-13 02:25:10 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6; # failJSON: { "time": "2013-04-27T02:25:11", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:11 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6; - +# failJSON: { "time": "2016-07-29T16:49:52", "match": true , "host": "0.0.0.0" } +Jul-29-16 16:49:52 m1-25391-06124 [Worker_1] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user@example.org relay attempt blocked for: someone@example.org +# failJSON: { "time": "2016-07-30T17:07:25", "match": true , "host": "0.0.0.0" } +Jul-30-16 17:07:25 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6 +# failJSON: { "time": "2016-07-30T17:11:05", "match": true , "host": "0.0.0.0" } +Jul-30-16 17:11:05 m1-13060-05386 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6 +# failJSON: { "time": "2016-07-31T06:45:59", "match": true , "host": "0.0.0.0" } +Jul-31-16 06:45:59 [Worker_1] [TLS-in] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: +# failJSON: { "time": "2016-01-05T08:38:49", "match": true , "host": "0.0.0.0" } +Jan-05-16 08:38:49 m1-01129-09140 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> relay attempt blocked for (parsing): <user2@example> +# failJSON: { "time": "2016-06-12T16:43:37", "match": true , "host": "0.0.0.0" } +Jun-12-16 16:43:37 m1-64217-12013 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user2@example.com relay attempt blocked for (parsing): <a.notheruser69@example.c> +# failJSON: { "time": "2016-01-22T22:25:51", "match": true , "host": "0.0.0.0" } +Jan-22-16 22:25:51 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid authentication mechanism +# failJSON: { "time": "2016-03-19T13:42:20", "match": true , "host": "0.0.0.0" } +Mar-19-16 13:42:20 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid base64 data in continued response +# failJSON: { "time": "2016-07-18T16:54:21", "match": true , "host": "0.0.0.0" } +Jul-18-16 16:54:21 [Worker_2] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server +# failJSON: { "time": "2016-07-18T17:14:23", "match": true , "host": "0.0.0.0" } +Jul-18-16 17:14:23 m1-76453-02949 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server diff --git a/fail2ban/tests/files/logs/asterisk b/fail2ban/tests/files/logs/asterisk index d17d93a1..13808592 100644 --- a/fail2ban/tests/files/logs/asterisk +++ b/fail2ban/tests/files/logs/asterisk @@ -43,6 +43,8 @@ # failJSON: { "time": "2004-11-04T18:30:40", "match": true , "host": "192.168.200.100" } Nov 4 18:30:40 localhost asterisk[32229]: NOTICE[32257]: chan_sip.c:23417 in handle_request_register: Registration from '<sip:301@example.com>' failed for '192.168.200.100:36998' - Wrong password +# failJSON: { "time": "2016-08-19T11:11:26", "match": true , "host": "192.0.2.1", "desc": "Another log_prefix used (` in` should be optional)" } +[2016-08-19 11:11:26] NOTICE[12931]: chan_sip.c:28468 handle_request_register: Registration from 'sip:bob@192.0.2.1' failed for '192.0.2.1:42406' - Wrong password # failed authentication attempt on INVITE using PJSIP # failJSON: { "time": "2015-05-24T08:42:16", "match": true, "host": "10.250.251.252" } diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index 627b8dc8..987f89f1 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -73,3 +73,8 @@ Jul 02 13:49:32 hostname dovecot[442]: pop3-login: Disconnected (no auth attempt # failJSON: { "time": "2005-03-23T06:10:52", "match": true , "host": "52.37.139.121" } Mar 23 06:10:52 auth: Info: ldap(dog,52.37.139.121,): invalid credentials + +# failJSON: { "time": "2005-07-26T11:11:21", "match": true , "host": "192.0.2.1" } +Jul 26 11:11:21 hostname dovecot: imap-login: Disconnected: Too many invalid commands (tried to use disallowed plaintext auth): user=<test>, rip=192.0.2.1, lip=192.168.1.1, session=<S5dIdTFCDKUWWMbU> +# failJSON: { "time": "2005-07-26T11:12:19", "match": true , "host": "192.0.2.2" } +Jul 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid commands (auth failed, 1 attempts in 17 secs): user=<test>, method=PLAIN, rip=192.0.2.2, lip=192.168.1.1, TLS, session=<g3ZKeDECFqlWWMbU> diff --git a/fail2ban/tests/files/logs/mongodb-auth b/fail2ban/tests/files/logs/mongodb-auth new file mode 100644 index 00000000..8a308892 --- /dev/null +++ b/fail2ban/tests/files/logs/mongodb-auth @@ -0,0 +1,30 @@ +# failJSON: { "match": false } +2016-11-20T00:04:00.110+0100 [conn1] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin +# failJSON: { "time": "2016-11-20T00:04:00", "match": true , "host": "192.0.2.35" } +2016-11-20T00:04:00.111+0100 [conn1] end connection 192.0.2.35:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn5] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.171" } +2016-11-20T00:24:00.111+0100 [conn5] end connection 192.0.2.171:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn334] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.176" } +2016-11-20T00:24:00.111+0100 [conn334] end connection 192.0.2.176:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn56] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.1" } +2016-11-20T00:24:00.111+0100 [conn56] end connection 192.0.2.1:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T12:54:02.370+0100 [initandlisten] connection accepted from 127.0.0.1:58774 #2261 (1 connection now open) +# failJSON: { "match": false } +2016-11-20T12:54:02.370+0100 [conn2261] end connection 127.0.0.1:58774 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T13:07:49.781+0100 [conn2271] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# failJSON: { "time": "2016-11-20T13:07:49", "match": false , "host": "192.0.2.178" } +2016-11-20T13:07:49.834+0100 [conn2271] end connection 192.0.2.178:60268 (3 connections now open) + diff --git a/fail2ban/tests/files/logs/postfix-sasl b/fail2ban/tests/files/logs/postfix-sasl index 9fcb0f49..cdcb5121 100644 --- a/fail2ban/tests/files/logs/postfix-sasl +++ b/fail2ban/tests/files/logs/postfix-sasl @@ -26,3 +26,7 @@ Jan 29 08:11:45 mail postfix-incoming/smtpd[10752]: warning: unknown[1.1.1.1]: S # failJSON: { "time": "2005-04-12T02:24:11", "match": true , "host": "62.138.2.143" } Apr 12 02:24:11 xxx postfix/smtps/smtpd[42]: warning: astra4139.startdedicated.de[62.138.2.143]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 + +# failJSON: { "time": "2005-08-03T15:30:49", "match": true , "host": "98.191.84.74" } +Aug 3 15:30:49 ksusha postfix/smtpd[17041]: warning: mail.foldsandwalker.com[98.191.84.74]: SASL Plain authentication failed: + diff --git a/fail2ban/tests/files/logs/sendmail-reject b/fail2ban/tests/files/logs/sendmail-reject index b326cf43..70d4dde6 100644 --- a/fail2ban/tests/files/logs/sendmail-reject +++ b/fail2ban/tests/files/logs/sendmail-reject @@ -40,6 +40,8 @@ Feb 19 18:01:50 batman sm-mta[78152]: ruleset=check_relay, arg1=[196.213.73.146] # failJSON: { "time": "2005-02-27T10:53:06", "match": true , "host": "209.15.212.253" } Feb 27 10:53:06 batman sm-mta[44307]: s1R9r60D044307: rejecting commands from [209.15.212.253] due to pre-greeting traffic after 0 seconds +# failJSON: { "time": "2005-02-27T10:53:07", "match": true , "host": "1.2.3.4" } +Feb 27 10:53:07 strange sm-mta[18001]: u9A0GtpL018001: rejecting commands from example.com [1.2.3.4] due to pre-greeting traffic after 6 seconds # failJSON: { "time": "2005-02-27T15:44:18", "match": true , "host": "41.204.78.137" } Feb 27 15:44:18 batman sm-mta[87838]: s1REiHdq087838: ruleset=check_rcpt, arg1=<gert-jan@t-online.ch>, relay=[41.204.78.137], reject=550 5.7.1 <gert-jan@t-online.ch>... Relaying denied. IP name lookup failed [41.204.78.137] diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index 7baf4be7..0800f86b 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -17,8 +17,10 @@ Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM 1.2.3.4 Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM ::ffff:1.2.3.4 #4 -# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "211.114.51.213" } -Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 211.114.51.213 +# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "192.0.2.1", "desc": "Invalid user" } +Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 192.0.2.1 +# failJSON: { "time": "2005-07-20T14:42:12", "match": true , "host": "192.0.2.2", "desc": "Invalid user with port" } +Jul 20 14:42:12 localhost sshd[22708]: Invalid user ftp from 192.0.2.2 port 37220 #5 new filter introduced after looking at 44087D8C.9090407@bluewin.ch # yoh: added ':' after [sshd] since the case without is not really common any more @@ -117,7 +119,13 @@ Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 po # failJSON: { "time": "2004-11-11T08:04:51", "match": true , "host": "127.0.0.1", "desc": "Injecting on username ssh 'from 10.10.1.1'@localhost" } Nov 11 08:04:51 redbamboo sshd[2737]: Failed password for invalid user from 10.10.1.1 from 127.0.0.1 port 58946 ssh2 +# failJSON: { "time": "2004-11-11T08:04:52", "match": true , "host": "127.0.0.1", "desc": "More complex injecting on username ssh 'test from 10.10.1.2 port 55555 ssh2'@localhost" } +Nov 11 08:04:52 redbamboo sshd[2737]: Failed password for invalid user test from 10.10.1.2 port 55555 ssh2 from 127.0.0.1 port 58946 ssh2 +# failJSON: { "time": "2004-11-11T08:04:52", "match": true , "host": "127.0.0.1", "desc": "More complex injecting on auth-info ssh test@localhost, auth-info: ' from 10.10.1.2 port 55555 ssh2'" } +Nov 11 08:04:52 redbamboo sshd[2737]: Failed password for invalid user test from 127.0.0.1 port 58946 ssh2: from 10.10.1.2 port 55555 ssh2 +# failJSON: { "time": "2005-07-05T18:22:44", "match": true , "host": "127.0.0.1", "desc": "Failed publickey for ..." } +Jul 05 18:22:44 mercury sshd[4669]: Failed publickey for graysky from 127.0.0.1 port 37954 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI # failJSON: { "match": false } Nov 23 21:50:19 sshd[8148]: Disconnecting: Too many authentication failures for root [preauth] @@ -161,4 +169,3 @@ Apr 27 13:02:04 host sshd[29116]: Received disconnect from 1.2.3.4: 11: Normal S # Match sshd auth errors on OpenSUSE systems # failJSON: { "time": "2015-04-16T20:02:50", "match": true , "host": "222.186.21.217", "desc": "Authentication for user failed" } 2015-04-16T18:02:50.321974+00:00 host sshd[2716]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.217 user=root - diff --git a/fail2ban/tests/files/logs/vsftpd b/fail2ban/tests/files/logs/vsftpd index bcd7f611..3205fac3 100644 --- a/fail2ban/tests/files/logs/vsftpd +++ b/fail2ban/tests/files/logs/vsftpd @@ -12,3 +12,6 @@ Fri Jan 19 12:20:33 2007 [pid 27202] [anonymous] FAIL LOGIN: Client "64.106.46.9 # failJSON: { "time": "2004-10-23T21:15:42", "match": true , "host": "58.254.172.161" } Oct 23 21:15:42 vps vsftpd: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=test rhost=58.254.172.161 + +# failJSON: { "time": "2016-09-08T00:39:49", "match": true , "host": "192.0.2.1" } +Thu Sep 8 00:39:49 2016 [pid 15019] [guest] FAIL LOGIN: Client "::ffff:192.0.2.1", "User is not in the allow user list." diff --git a/fail2ban/tests/files/testcase-journal.log b/fail2ban/tests/files/testcase-journal.log index 720a3130..b6fa427b 100644 --- a/fail2ban/tests/files/testcase-journal.log +++ b/fail2ban/tests/files/testcase-journal.log @@ -13,7 +13,7 @@ error: PAM: Authentication failure for kevin from 193.168.0.128 error: PAM: Authentication failure for kevin from 193.168.0.128 error: PAM: Authentication failure for kevin from 193.168.0.128 error: PAM: Authentication failure for kevin from 193.168.0.128 -error: PAM: Authentication failure for kevin from 87.142.124.10 -error: PAM: Authentication failure for kevin from 87.142.124.10 -error: PAM: Authentication failure for kevin from 87.142.124.10 -error: PAM: Authentication failure for kevin from 87.142.124.10 +error: PAM: Authentication failure for göran from 87.142.124.10 +error: PAM: Authentication failure for göran from 87.142.124.10 +error: PAM: Authentication failure for göran from 87.142.124.10 +error: PAM: Authentication failure for göran from 87.142.124.10 diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 40879b66..e9971a06 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -38,7 +38,7 @@ except ImportError: from ..server.jail import Jail from ..server.filterpoll import FilterPoll -from ..server.filter import Filter, FileFilter, DNSUtils +from ..server.filter import Filter, FileFilter, FileContainer, locale, DNSUtils from ..server.failmanager import FailManagerEmpty from ..server.mytime import MyTime from .utils import setUpMyTime, tearDownMyTime, mtimesleep, LogCaptureTestCase @@ -166,6 +166,10 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line return fout +TEST_JOURNAL_FIELDS = { + "SYSLOG_IDENTIFIER": "fail2ban-testcases", + "PRIORITY": "7", +} def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # pragma: systemd no cover """Copy lines from one file to systemd journal @@ -176,9 +180,7 @@ def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # p else: fin = in_ # Required for filtering - fields.update({"SYSLOG_IDENTIFIER": "fail2ban-testcases", - "PRIORITY": "7", - }) + fields.update(TEST_JOURNAL_FIELDS) # Skip for i in xrange(skip): fin.readline() @@ -228,6 +230,19 @@ class BasicFilter(unittest.TestCase): 1) ) + def testWrongCharInTupleLine(self): + ## line tuple has different types (ascii after ascii / unicode): + for a1 in ('', u'', b''): + for a2 in ('2016-09-05T20:18:56', u'2016-09-05T20:18:56', b'2016-09-05T20:18:56'): + for a3 in ( + 'Fail for "g\xc3\xb6ran" from 192.0.2.1', + u'Fail for "g\xc3\xb6ran" from 192.0.2.1', + b'Fail for "g\xc3\xb6ran" from 192.0.2.1' + ): + # join should work if all arguments have the same type: + enc = locale.getpreferredencoding() + "".join([Filter.uni_decode(v, enc) for v in (a1, a2, a3)]) + class IgnoreIP(LogCaptureTestCase): @@ -707,11 +722,16 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover """Call before every test case.""" self.test_file = os.path.join(TEST_FILES_DIR, "testcase-journal.log") self.jail = DummyJail() - self.filter = Filter_(self.jail) + self.filter = None # UUID used to ensure that only meeages generated # as part of this test are picked up by the filter self.test_uuid = str(uuid.uuid4()) self.name = "monitorjournalfailures-%s" % self.test_uuid + self.journal_fields = { + 'TEST_FIELD': "1", 'TEST_UUID': self.test_uuid} + + def _initFilter(self, **kwargs): + self.filter = Filter_(self.jail, **kwargs) self.filter.addJournalMatch([ "SYSLOG_IDENTIFIER=fail2ban-testcases", "TEST_FIELD=1", @@ -720,16 +740,16 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover "SYSLOG_IDENTIFIER=fail2ban-testcases", "TEST_FIELD=2", "TEST_UUID=%s" % self.test_uuid]) - self.journal_fields = { - 'TEST_FIELD': "1", 'TEST_UUID': self.test_uuid} - self.filter.active = True self.filter.addFailRegex("(?:(?:Authentication failure|Failed [-/\w+]+) for(?: [iI](?:llegal|nvalid) user)?|[Ii](?:llegal|nvalid) user|ROOT LOGIN REFUSED) .*(?: from|FROM) <HOST>") - self.filter.start() def tearDown(self): - self.filter.stop() - self.filter.join() # wait for the thread to terminate - pass + if self.filter and self.filter.active: + self.filter.stop() + self.filter.join() # wait for the thread to terminate + pass + + def testJournalFlagsArg(self): + self._initFilter(journalflags=2) # journal.RUNTIME_ONLY def __str__(self): return "MonitorJournalFailures%s(%s)" \ @@ -761,6 +781,8 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.assertEqual(attempts, test_attempts) def test_grow_file(self): + self._initFilter() + self.filter.start() self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) # Now let's feed it with entries from the file @@ -790,6 +812,8 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.assert_correct_ban("193.168.0.128", 3) def test_delJournalMatch(self): + self._initFilter() + self.filter.start() # Smoke test for removing of match # basic full test @@ -819,6 +843,33 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover # we should detect the failures self.assertTrue(self.isFilled(6)) + def test_WrongChar(self): + self._initFilter() + self.filter.start() + # Now let's feed it with entries from the file + _copy_lines_to_journal( + self.test_file, self.journal_fields, skip=15, n=4) + self.assertTrue(self.isFilled(10)) + self.assert_correct_ban("87.142.124.10", 4) + # Add direct utf, unicode, blob: + for l in ( + "error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1", + u"error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1", + b"error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1".decode('utf-8', 'replace'), + "error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2", + u"error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2", + b"error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2".decode('utf-8', 'replace') + ): + fields = self.journal_fields + fields.update(TEST_JOURNAL_FIELDS) + journal.send(MESSAGE=l, **fields) + self.assertTrue(self.isFilled(10)) + endtm = MyTime.time()+10 + while len(self.jail) != 2 and MyTime.time() < endtm: + time.sleep(0.10) + self.assertEqual(sorted([self.jail.getFailTicket().getIP(), self.jail.getFailTicket().getIP()]), + ["192.0.2.1", "192.0.2.2"]) + return MonitorJournalFailures diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index 48074d53..4dfff2d4 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -23,6 +23,7 @@ __license__ = "GPL" import logging import os +import re import sys import unittest import tempfile @@ -32,8 +33,11 @@ import datetime from glob import glob from StringIO import StringIO +from utils import LogCaptureTestCase, logSys as DefLogSys + from ..helpers import formatExceptionInfo, mbasename, TraceBack, FormatterWithTraceBack, getLogger from ..helpers import splitwords +from ..server.datedetector import DateDetector from ..server.datetemplate import DatePatternRegex @@ -67,6 +71,22 @@ class HelpersTest(unittest.TestCase): self.assertEqual(splitwords(' 1\n 2, 3'), ['1', '2', '3']) +if sys.version_info >= (2,7): + def _sh_call(cmd): + import subprocess, locale + ret = subprocess.check_output(cmd, shell=True) + if sys.version_info >= (3,): + ret = ret.decode(locale.getpreferredencoding(), 'replace') + return str(ret).rstrip() +else: + def _sh_call(cmd): + import subprocess + ret = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read() + return str(ret).rstrip() + +def _getSysPythonVersion(): + return _sh_call("fail2ban-python -c 'import sys; print(tuple(sys.version_info))'") + class SetupTest(unittest.TestCase): def setUp(self): @@ -76,6 +96,12 @@ class SetupTest(unittest.TestCase): raise unittest.SkipTest( "Seems to be running not out of source distribution" " -- cannot locate setup.py") + # compare current version of python installed resp. active one: + sysVer = _getSysPythonVersion() + if sysVer != str(tuple(sys.version_info)): + raise unittest.SkipTest( + "Seems to be running with python distribution %s" + " -- install can be tested only with system distribution %s" % (str(tuple(sys.version_info)), sysVer)) def testSetupInstallRoot(self): if not self.setup: @@ -122,6 +148,14 @@ class SetupTest(unittest.TestCase): 'etc/fail2ban/jail.conf'): self.assertTrue(os.path.exists(os.path.join(tmp, f)), msg="Can't find %s" % f) + # Because the install (test) path in virtual-env differs from some development-env, + # it is not a `tmp + '/usr/local/bin/'`, so search for it: + installedPath = _sh_call('find ' + tmp+ ' -name fail2ban-python').split('\n') + self.assertTrue(len(installedPath) > 0) + for installedPath in installedPath: + self.assertEqual( + os.path.realpath(installedPath), os.path.realpath(sys.executable)) + finally: # clean up shutil.rmtree(tmp) @@ -130,7 +164,7 @@ class SetupTest(unittest.TestCase): % (sys.executable, self.setup)) -class TestsUtilsTest(unittest.TestCase): +class TestsUtilsTest(LogCaptureTestCase): def testmbasename(self): self.assertEqual(mbasename("sample.py"), 'sample') @@ -165,12 +199,88 @@ class TestsUtilsTest(unittest.TestCase): if not ('fail2ban-testcases' in s): # we must be calling it from setup or nosetests but using at least # nose's core etc - self.assertTrue('>' in s, msg="no '>' in %r" % s) + self.assertIn('>', s) elif not ('coverage' in s): # There is only "fail2ban-testcases" in this case, no true traceback - self.assertFalse('>' in s, msg="'>' present in %r" % s) - - self.assertTrue(':' in s, msg="no ':' in %r" % s) + self.assertNotIn('>', s) + + self.assertIn(':', s) + + def _testAssertionErrorRE(self, regexp, fun, *args, **kwargs): + self.assertRaisesRegexp(AssertionError, regexp, fun, *args, **kwargs) + + def testExtendedAssertRaisesRE(self): + ## test _testAssertionErrorRE several fail cases: + def _key_err(msg): + raise KeyError(msg) + self.assertRaises(KeyError, + self._testAssertionErrorRE, r"^failed$", + _key_err, 'failed') + self.assertRaises(AssertionError, + self._testAssertionErrorRE, r"^failed$", + self.fail, '__failed__') + self._testAssertionErrorRE(r'failed.* does not match .*__failed__', + lambda: self._testAssertionErrorRE(r"^failed$", + self.fail, '__failed__') + ) + ## no exception in callable: + self.assertRaises(AssertionError, + self._testAssertionErrorRE, r"", int, 1) + self._testAssertionErrorRE(r'0 AssertionError not raised X.* does not match .*AssertionError not raised', + lambda: self._testAssertionErrorRE(r"^0 AssertionError not raised X$", + lambda: self._testAssertionErrorRE(r"", int, 1)) + ) + + def testExtendedAssertMethods(self): + ## assertIn, assertNotIn positive case: + self.assertIn('a', ['a', 'b', 'c', 'd']) + self.assertIn('a', ('a', 'b', 'c', 'd',)) + self.assertIn('a', 'cba') + self.assertIn('a', (c for c in 'cba' if c != 'b')) + self.assertNotIn('a', ['b', 'c', 'd']) + self.assertNotIn('a', ('b', 'c', 'd',)) + self.assertNotIn('a', 'cbd') + self.assertNotIn('a', (c.upper() for c in 'cba' if c != 'b')) + ## assertIn, assertNotIn negative case: + self._testAssertionErrorRE(r"'a' unexpectedly found in 'cba'", + self.assertNotIn, 'a', 'cba') + self._testAssertionErrorRE(r"1 unexpectedly found in \[0, 1, 2\]", + self.assertNotIn, 1, xrange(3)) + self._testAssertionErrorRE(r"'A' unexpectedly found in \['C', 'A'\]", + self.assertNotIn, 'A', (c.upper() for c in 'cba' if c != 'b')) + self._testAssertionErrorRE(r"'a' was not found in 'xyz'", + self.assertIn, 'a', 'xyz') + self._testAssertionErrorRE(r"5 was not found in \[0, 1, 2\]", + self.assertIn, 5, xrange(3)) + self._testAssertionErrorRE(r"'A' was not found in \['C', 'B'\]", + self.assertIn, 'A', (c.upper() for c in 'cba' if c != 'a')) + ## assertLogged, assertNotLogged positive case: + logSys = DefLogSys + self.pruneLog() + logSys.debug('test "xyz"') + self.assertLogged('test "xyz"') + self.assertLogged('test', 'xyz', all=True) + self.assertNotLogged('test', 'zyx', all=False) + self.assertNotLogged('test_zyx', 'zyx', all=True) + self.assertLogged('test', 'zyx', all=False) + self.pruneLog() + logSys.debug('xxxx "xxx"') + self.assertNotLogged('test "xyz"') + self.assertNotLogged('test', 'xyz', all=False) + self.assertNotLogged('test', 'xyz', 'zyx', all=True) + ## assertLogged, assertNotLogged negative case: + self.pruneLog() + logSys.debug('test "xyz"') + self._testAssertionErrorRE(r"All of the .* were found present in the log", + self.assertNotLogged, 'test "xyz"') + self._testAssertionErrorRE(r"was found in the log", + self.assertNotLogged, 'test', 'xyz', all=True) + self._testAssertionErrorRE(r"was not found in the log", + self.assertLogged, 'test', 'zyx', all=True) + self._testAssertionErrorRE(r"None among .* was found in the log", + self.assertLogged, 'test_zyx', 'zyx', all=False) + self._testAssertionErrorRE(r"All of the .* were found present in the log", + self.assertNotLogged, 'test', 'xyz', all=False) def testFormatterWithTraceBack(self): strout = StringIO() @@ -231,3 +341,47 @@ class CustomDateFormatsTest(unittest.TestCase): self.assertEqual( date, datetime.datetime(2007, 1, 25, 16, 0)) + + def testAmbiguousDatePattern(self): + defDD = DateDetector() + defDD.addDefaultTemplate() + logSys = DefLogSys + for (matched, dp, line) in ( + # positive case: + ('Jan 23 21:59:59', None, 'Test failure Jan 23 21:59:59 for 192.0.2.1'), + # ambiguous "unbound" patterns (missed): + (False, None, 'Test failure TestJan 23 21:59:59.011 2015 for 192.0.2.1'), + (False, None, 'Test failure Jan 23 21:59:59123456789 for 192.0.2.1'), + # ambiguous "no optional year" patterns (matched): + ('Aug 8 11:25:50', None, 'Aug 8 11:25:50 14430f2329b8 Authentication failed from 192.0.2.1'), + ('Aug 8 11:25:50', None, '[Aug 8 11:25:50] 14430f2329b8 Authentication failed from 192.0.2.1'), + ('Aug 8 11:25:50 2014', None, 'Aug 8 11:25:50 2014 14430f2329b8 Authentication failed from 192.0.2.1'), + # direct specified patterns: + ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y$', '192.0.2.1 at 20:00:00 01.02.2003'), + ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]', '192.0.2.1[20:00:00 01.02.2003]'), + ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'), + ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]$', '192.0.2.1[20:00:00 01.02.2003]'), + ('[20:00:00 01.02.2003]', r'^\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'), + ('[17/Jun/2011 17:00:45]', r'^\[%d/%b/%Y %H:%M:%S\]', '[17/Jun/2011 17:00:45] Attempt, IP address 192.0.2.1'), + ('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt [17/Jun/2011 17:00:45] IP address 192.0.2.1'), + ('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt IP address 192.0.2.1, date: [17/Jun/2011 17:00:45]'), + # direct specified patterns (begin/end, missed): + (False, r'%H:%M:%S %d.%m.%Y', '192.0.2.1x20:00:00 01.02.2003'), + (False, r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003x192.0.2.1'), + # direct specified patterns (begin/end, matched): + ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '192.0.2.1 20:00:00 01.02.2003'), + ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003 192.0.2.1'), + ): + logSys.debug('== test: %r', (matched, dp, line)) + if dp is None: + dd = defDD + else: + dp = DatePatternRegex(dp) + dd = DateDetector() + dd.appendTemplate(dp) + date = dd.getTime(line) + if matched: + self.assertTrue(date) + self.assertEqual(matched, date[1].group()) + else: + self.assertEqual(date, None) diff --git a/fail2ban/tests/samplestestcase.py b/fail2ban/tests/samplestestcase.py index a40a11da..327410bc 100644 --- a/fail2ban/tests/samplestestcase.py +++ b/fail2ban/tests/samplestestcase.py @@ -94,7 +94,7 @@ def testSampleRegexsFactory(name, basedir): if jsonREMatch: try: faildata = json.loads(jsonREMatch.group(1)) - except ValueError, e: + except ValueError as e: raise ValueError("%s: %s:%i" % (e, logFile.filename(), logFile.filelineno())) line = next(logFile) diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 07e10c7d..99502794 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -228,7 +228,7 @@ class Transmitter(TransmitterBase): time.sleep(1) self.assertEqual( self.transm.proceed(["stop", self.jailName]), (0, None)) - self.assertTrue(self.jailName not in self.server._Server__jails) + self.assertNotIn(self.jailName, self.server._Server__jails) def testStartStopAllJail(self): self.server.addJail("TestJail2", "auto") @@ -242,8 +242,8 @@ class Transmitter(TransmitterBase): time.sleep(0.1) self.assertEqual(self.transm.proceed(["stop", "all"]), (0, None)) time.sleep(1) - self.assertTrue(self.jailName not in self.server._Server__jails) - self.assertTrue("TestJail2" not in self.server._Server__jails) + self.assertNotIn(self.jailName, self.server._Server__jails) + self.assertNotIn("TestJail2", self.server._Server__jails) def testJailIdle(self): self.assertEqual( @@ -688,10 +688,7 @@ class Transmitter(TransmitterBase): def testJournalMatch(self): if not filtersystemd: # pragma: no cover - if sys.version_info >= (2, 7): - raise unittest.SkipTest( - "systemd python interface not available") - return + raise unittest.SkipTest("systemd python interface not available") jailName = "TestJail2" self.server.addJail(jailName, "systemd") values = [ @@ -791,10 +788,8 @@ class TransmitterLogging(TransmitterBase): self.setGetTest("logtarget", "STDERR") def testLogTargetSYSLOG(self): - if not os.path.exists("/dev/log") and sys.version_info >= (2, 7): + if not os.path.exists("/dev/log"): raise unittest.SkipTest("'/dev/log' not present") - elif not os.path.exists("/dev/log"): - return self.assertTrue(self.server.getSyslogSocket(), "auto") self.setGetTest("logtarget", "SYSLOG") self.assertTrue(self.server.getSyslogSocket(), "/dev/log") diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index 8fc78683..9155dc37 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -22,13 +22,17 @@ __author__ = "Yaroslav Halchenko" __copyright__ = "Copyright (c) 2013 Yaroslav Halchenko" __license__ = "GPL" +import itertools import logging import os import re +import tempfile +import shutil import sys import time import unittest from StringIO import StringIO +from functools import wraps from ..server.mytime import MyTime from ..helpers import getLogger @@ -45,6 +49,40 @@ if not CONFIG_DIR: CONFIG_DIR = '/etc/fail2ban' +def with_tmpdir(f): + """Helper decorator to create a temporary directory + + Directory gets removed after function returns, regardless + if exception was thrown of not + """ + @wraps(f) + def wrapper(self, *args, **kwargs): + tmp = tempfile.mkdtemp(prefix="f2b-temp") + try: + return f(self, tmp, *args, **kwargs) + finally: + # clean up + shutil.rmtree(tmp) + return wrapper + + +# backwards compatibility to python 2.6: +if not hasattr(unittest, 'SkipTest'): # pragma: no cover + class SkipTest(Exception): + pass + unittest.SkipTest = SkipTest + _org_AddError = unittest._TextTestResult.addError + def addError(self, test, err): + if err[0] is SkipTest: + if self.showAll: + self.stream.writeln(str(err[1])) + elif self.dots: + self.stream.write('s') + self.stream.flush() + return + _org_AddError(self, test, err) + unittest._TextTestResult.addError = addError + def mtimesleep(): # no sleep now should be necessary since polling tracks now not only # mtime but also ino and size @@ -183,13 +221,13 @@ def gatherTests(regexps=None, no_network=False): try: from ..server.filtergamin import FilterGamin filters.append(FilterGamin) - except Exception, e: # pragma: no cover + except Exception as e: # pragma: no cover logSys.warning("Skipping gamin backend testing. Got exception '%s'" % e) try: from ..server.filterpyinotify import FilterPyinotify filters.append(FilterPyinotify) - except Exception, e: # pragma: no cover + except Exception as e: # pragma: no cover logSys.warning("I: Skipping pyinotify backend testing. Got exception '%s'" % e) for Filter_ in filters: @@ -198,7 +236,7 @@ def gatherTests(regexps=None, no_network=False): try: # pragma: systemd no cover from ..server.filtersystemd import FilterSystemd tests.addTest(unittest.makeSuite(filtertestcase.get_monitor_failures_journal_testcase(FilterSystemd))) - except Exception, e: # pragma: no cover + except Exception as e: # pragma: no cover logSys.warning("I: Skipping systemd backend testing. Got exception '%s'" % e) # Server test for logging elements which break logging used to support @@ -208,16 +246,45 @@ def gatherTests(regexps=None, no_network=False): return tests -# forwards compatibility of unittest.TestCase for some early python versions -if not hasattr(unittest.TestCase, 'assertIn'): - def __assertIn(self, a, b, msg=None): - if a not in b: # pragma: no cover - self.fail(msg or "%r was not found in %r" % (a, b)) - unittest.TestCase.assertIn = __assertIn - def __assertNotIn(self, a, b, msg=None): - if a in b: # pragma: no cover - self.fail(msg or "%r was found in %r" % (a, b)) - unittest.TestCase.assertNotIn = __assertNotIn +# +# Forwards compatibility of unittest.TestCase for some early python versions +# + +if not hasattr(unittest.TestCase, 'assertRaisesRegexp'): + def assertRaisesRegexp(self, exccls, regexp, fun, *args, **kwargs): + try: + fun(*args, **kwargs) + except exccls as e: + if re.search(regexp, str(e)) is None: + self.fail('\"%s\" does not match \"%s\"' % (regexp, e)) + else: + self.fail('%s not raised' % getattr(exccls, '__name__')) + unittest.TestCase.assertRaisesRegexp = assertRaisesRegexp + +# always custom following methods, because we use atm better version of both (support generators) +if True: ## if not hasattr(unittest.TestCase, 'assertIn'): + def assertIn(self, a, b, msg=None): + bb = b + wrap = False + if msg is None and hasattr(b, '__iter__') and not isinstance(b, basestring): + b, bb = itertools.tee(b) + wrap = True + if a not in b: + if wrap: bb = list(bb) + msg = msg or "%r was not found in %r" % (a, bb) + self.fail(msg) + unittest.TestCase.assertIn = assertIn + def assertNotIn(self, a, b, msg=None): + bb = b + wrap = False + if msg is None and hasattr(b, '__iter__') and not isinstance(b, basestring): + b, bb = itertools.tee(b) + wrap = True + if a in b: + if wrap: bb = list(bb) + msg = msg or "%r unexpectedly found in %r" % (a, bb) + self.fail(msg) + unittest.TestCase.assertNotIn = assertNotIn class LogCaptureTestCase(unittest.TestCase): @@ -241,6 +308,7 @@ class LogCaptureTestCase(unittest.TestCase): def tearDown(self): """Call after every test case.""" # print "O: >>%s<<" % self._log.getvalue() + self.pruneLog() logSys = getLogger("fail2ban") logSys.handlers = self._old_handlers logSys.level = self._old_level @@ -248,7 +316,7 @@ class LogCaptureTestCase(unittest.TestCase): def _is_logged(self, s): return s in self._log.getvalue() - def assertLogged(self, *s): + def assertLogged(self, *s, **kwargs): """Assert that one of the strings was logged Preferable to assertTrue(self._is_logged(..))) @@ -258,14 +326,23 @@ class LogCaptureTestCase(unittest.TestCase): ---------- s : string or list/set/tuple of strings Test should succeed if string (or any of the listed) is present in the log + all : boolean (default False) if True should fail if any of s not logged """ logged = self._log.getvalue() - for s_ in s: - if s_ in logged: - return - raise AssertionError("None among %r was found in the log: %r" % (s, logged)) + if not kwargs.get('all', False): + # at least one entry should be found: + for s_ in s: + if s_ in logged: + return + if True: # pragma: no cover + self.fail("None among %r was found in the log: ===\n%s===" % (s, logged)) + else: + # each entry should be found: + for s_ in s: + if s_ not in logged: # pragma: no cover + self.fail("%r was not found in the log: ===\n%s===" % (s_, logged)) - def assertNotLogged(self, *s): + def assertNotLogged(self, *s, **kwargs): """Assert that strings were not logged Parameters @@ -273,13 +350,22 @@ class LogCaptureTestCase(unittest.TestCase): s : string or list/set/tuple of strings Test should succeed if the string (or at least one of the listed) is not present in the log + all : boolean (default False) if True should fail if any of s logged """ logged = self._log.getvalue() - for s_ in s: - if s_ not in logged: - return - raise AssertionError("All of the %r were found present in the log: %r" % (s, logged)) + if not kwargs.get('all', False): + for s_ in s: + if s_ not in logged: + return + if True: # pragma: no cover + self.fail("All of the %r were found present in the log: ===\n%s===" % (s, logged)) + else: + for s_ in s: + if s_ in logged: # pragma: no cover + self.fail("%r was found in the log: ===\n%s===" % (s_, logged)) + def pruneLog(self): + self._log.truncate(0) def getLog(self): return self._log.getvalue() diff --git a/fail2ban/version.py b/fail2ban/version.py index 57aa3c28..194918d7 100644 --- a/fail2ban/version.py +++ b/fail2ban/version.py @@ -24,4 +24,4 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko, Steven Hiscocks, Daniel Black" __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2005-2016 Yaroslav Halchenko, 2013-2014 Steven Hiscocks, Daniel Black" __license__ = "GPL-v2+" -version = "0.9.5" +version = "0.9.6" diff --git a/files/monit/fail2ban b/files/monit/fail2ban index 8e6c9419..7873dbe0 100644 --- a/files/monit/fail2ban +++ b/files/monit/fail2ban @@ -1,7 +1,7 @@ check process fail2ban with pidfile /var/run/fail2ban/fail2ban.pid group services start program = "/etc/init.d/fail2ban force-start" - stop program = "/etc/init.d/fail2ban stop || :" + stop program = "/etc/init.d/fail2ban stop" if failed unixsocket /var/run/fail2ban/fail2ban.sock then restart if 5 restarts within 5 cycles then timeout diff --git a/man/fail2ban-client.1 b/man/fail2ban-client.1 index f67e71ea..ff3e5f77 100644 --- a/man/fail2ban-client.1 +++ b/man/fail2ban-client.1 @@ -1,12 +1,12 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-CLIENT "1" "July 2016" "fail2ban-client v0.9.5" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-CLIENT "1" "December 2016" "fail2ban-client v0.9.6" "User Commands" .SH NAME fail2ban-client \- configure and control the server .SH SYNOPSIS .B fail2ban-client [\fI\,OPTIONS\/\fR] \fI\,<COMMAND>\/\fR .SH DESCRIPTION -Fail2Ban v0.9.5 reads log file that contains password failure report +Fail2Ban v0.9.6 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .SH OPTIONS .TP diff --git a/man/fail2ban-regex.1 b/man/fail2ban-regex.1 index 1db372cc..1ed2c327 100644 --- a/man/fail2ban-regex.1 +++ b/man/fail2ban-regex.1 @@ -1,5 +1,5 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-REGEX "1" "July 2016" "fail2ban-regex 0.9.5" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-REGEX "1" "December 2016" "fail2ban-regex 0.9.6" "User Commands" .SH NAME fail2ban-regex \- test Fail2ban "failregex" option .SH SYNOPSIS diff --git a/man/fail2ban-server.1 b/man/fail2ban-server.1 index 96f0c3e5..5278302c 100644 --- a/man/fail2ban-server.1 +++ b/man/fail2ban-server.1 @@ -1,12 +1,12 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-SERVER "1" "July 2016" "fail2ban-server v0.9.5" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-SERVER "1" "December 2016" "fail2ban-server v0.9.6" "User Commands" .SH NAME fail2ban-server \- start the server .SH SYNOPSIS .B fail2ban-server [\fI\,OPTIONS\/\fR] .SH DESCRIPTION -Fail2Ban v0.9.5 reads log file that contains password failure report +Fail2Ban v0.9.6 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .PP Only use this command for debugging purpose. Start the server with diff --git a/man/fail2ban-testcases.1 b/man/fail2ban-testcases.1 index 1c2f1a8e..658555ac 100644 --- a/man/fail2ban-testcases.1 +++ b/man/fail2ban-testcases.1 @@ -1,5 +1,5 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-TESTCASES "1" "July 2016" "fail2ban-testcases 0.9.5" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-TESTCASES "1" "December 2016" "fail2ban-testcases 0.9.6" "User Commands" .SH NAME fail2ban-testcases \- run Fail2Ban unit-tests .SH SYNOPSIS @@ -19,9 +19,11 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. __author__ = "Cyril Jaquier, Steven Hiscocks, Yaroslav Halchenko" -__copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2008-2013 Fail2Ban Contributors" +__copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2008-2016 Fail2Ban Contributors" __license__ = "GPL" +import platform + try: import setuptools from setuptools import setup @@ -38,12 +40,40 @@ except ImportError: # python 2.x from distutils.command.build_py import build_py from distutils.command.build_scripts import build_scripts +# all versions +from distutils.command.install_scripts import install_scripts + import os from os.path import isfile, join, isdir, realpath import sys import warnings from glob import glob +from fail2ban.setup import updatePyExec + + +# Wrapper to install python binding (to current python version): +class install_scripts_f2b(install_scripts): + + def get_outputs(self): + outputs = install_scripts.get_outputs(self) + fn = None + for fn in outputs: + if os.path.basename(fn) == 'fail2ban-server': + break + bindir = os.path.dirname(fn) + print('creating fail2ban-python binding -> %s' % (bindir,)) + updatePyExec(bindir) + return outputs + + +# Update fail2ban-python env to current python version (where f2b-modules located/installed) +rootdir = os.path.realpath(os.path.dirname( + # __file__ seems to be overwritten sometimes on some python versions (e.g. bug of 2.6 by running under cProfile, etc.): + sys.argv[0] if os.path.basename(sys.argv[0]) == 'setup.py' else __file__ +)) +updatePyExec(os.path.join(rootdir, 'bin')) + if setuptools and "test" in sys.argv: import logging logSys = logging.getLogger("fail2ban") @@ -85,6 +115,18 @@ if os.path.exists('/var/run'): # realpath is used to possibly resolve /var/run -> /run symlink data_files_extra += [(realpath('/var/run/fail2ban'), '')] +# Installing documentation files only under Linux or other GNU/ systems +# (e.g. GNU/kFreeBSD), since others might have protective mechanisms forbidding +# installation there (see e.g. #1233) +platform_system = platform.system().lower() +doc_files = ['README.md', 'DEVELOP', 'FILTERS', 'doc/run-rootless.txt'] +if platform_system in ('solaris', 'sunos'): + doc_files.append('README.Solaris') +if platform_system in ('linux', 'solaris', 'sunos') or platform_system.startswith('gnu'): + data_files_extra.append( + ('/usr/share/doc/fail2ban', doc_files) + ) + # Get version number, avoiding importing fail2ban. # This is due to tests not functioning for python3 as 2to3 takes place later exec(open(join("fail2ban", "version.py")).read()) @@ -99,12 +141,16 @@ setup( url = "http://www.fail2ban.org", license = "GPL", platforms = "Posix", - cmdclass = {'build_py': build_py, 'build_scripts': build_scripts}, + cmdclass = { + 'build_py': build_py, 'build_scripts': build_scripts, + 'install_scripts': install_scripts_f2b + }, scripts = [ 'bin/fail2ban-client', 'bin/fail2ban-server', 'bin/fail2ban-regex', 'bin/fail2ban-testcases', + # 'bin/fail2ban-python', -- link (binary), will be installed via install_scripts_f2b wrapper ], packages = [ 'fail2ban', @@ -148,10 +194,6 @@ setup( ('/var/lib/fail2ban', '' ), - ('/usr/share/doc/fail2ban', - ['README.md', 'README.Solaris', 'DEVELOP', 'FILTERS', - 'doc/run-rootless.txt'] - ) ] + data_files_extra, **setup_extra ) |